Robust Reinforcement Learning with Wasserstein Constraint
Authors: Linfang Hou
Liang Pang
Xin Hong
Yanyan Lan
Zhiming Ma
Dawei Yin
Venue:  (2020)

Abstract: Robust Reinforcement Learning aims to find the optimal policy with some degree of robustness to environmental dynamics. Existing learning algorithms usually enable the robustness though disturbing the current state or simulated environmental parameters in a heuristic way, which lack quantified robustness to the system dynamics (i.e.  transition probability).  To overcome this issue, we leverage Wasserstein distance to measure the disturbance to the reference transition probability. With Wasserstein distance, we are able to connect transition probability disturbance to the state disturbance, and reduces an infinite-dimensional optimization problem to a finite-dimensional risk-aware problem. Through the derived risk-aware optimal Bellman equation,  we first show the existence of optimal robust policies,  provide  a  sensitivity  analysis  for  the  perturbations,  and  then  design  a novel robust learning algorithm—WassersteinRobustAdvantageActor-Critic algorithm (WRA2C). The effectiveness of the proposed algorithm is verified in theCart-Pole environment.

Under review as a conference paper at ICLR 2020
ROBUST REINFORCEMENT LEARNING WITH WASSER- STEIN CONSTRAINT
Anonymous authors Paper under double-blind review
ABSTRACT
Robust Reinforcement Learning aims to find the optimal policy with some ex- tent of robustness to environmental dynamics. Existing learning algorithms usu- ally enable the robustness though disturbing the current state or simulated en- vironmental parameters in heuristic way, which lack quantified robustness the system dynamics (i.e. transition probability). To overcome this issue, we leverage Wasserstein distance to measure the disturbance the reference transi- tion kernel. With Wasserstein distance, we are able to connect transition kernel disturbance to the state disturbance, i.c. reduce an dimenii-dimeiiia optimiza- tion problem to finie-dimenniionll risk-aware problem. Through the derived risk-aware optimal Bellman equation, we show the existence of optimal robust policies, provide a sensitivity analysis for the perturbations, and then design a novel robust learning Wassestmn Robust Advantage Actor-Crite al- gorithm (WRAAC). The effectiveness of the proposed algorithm is verified in the Cart-Pole environment.
INTRODUCTION
Robustness environmental dynamics is an important topic in safe Reinforcement Learning. Take autonomous vehicle as an example. Autonomous vehicles have to adapt the complex real-world sit- uations, but usually To handle this issue, typically, a simulated environment are employed to help build a driving agent, however, the gap between the training and target environments makes the strategies trained with simulated environments sub-optimal to the real-world scenarios (Mannor et al. [2044] (007). Learn- ing robust policies from simulated environments is a challenging problem for safe Reinforcement Learning.
unlikely to cover all scenarios during training in real-world environments.
For robust Reinforcement Learning algorithms, existing methods lie on two branches: One type of methods, borrowed from game theory. introduces extra agent to disturb the simulated envi- ronmental parameters during training (Atkeson & Morimoto [2003] Morimoto & Doyal 2005 Pinto et al. 2017 Rajeswaran et al. (016)). This method has to rely the environmental characterization. The other type of methods disturbs the current state through Adversarial Examples (Huang et al.) 2017 Kos & Song [017 Lin al. [2017] Mandlekar et al. 2017 Pattanaik et al. 2018), which is more heuristic. Unfortunately, both methods are lack of theoretical guarantee to the robustness extent transition dynamics.
To address these issues, we design a Wasserstern constraint, which restricts the admissible transition probabilities within Wasserstein ball centered some reference transition dynamics. By apply- ing the strong duality of Wasserstein distance (Santambrogio) 2015 Blanchet & Murthy 2019). we are able to connect the disturbance on transition dynamics with the disturbance on the current robust optimal problem is reduced to some finite-
state. As result, the original dimensional ordinary risk-aware RL problem. Through the moderated optimal Bellman equation, prove the existence of robust optimal policies, provide the theoretical analyse on the performance of optimal policies, and design a corresponding Wasserstein Robust Advantage Actor Crioic algo- rithm (WRAAC), which does not depend the environmental characterization. In the experiments, we verified the robustness and effectiveness the proposed algorithms in the Cart-Pole environ- ment.

---

Under review as a conference paper at ICLR 2020
The remainder of this paper is organized as follows. In Section 2 briefty introduce some related work in Markov Decision Processes. Section we mainly describe the framework of Wasserstein robust Reinforcement Learning. In Section 4 propose robust Advantage Actor-Critte algorithms according to the moderated robust Bellman equation. In Section 5 we perform experiments on the Cart-Pole environment to verify the effectiveness of our method. Finally, Section 6 concludes study and provide possible future works.
RELATED WORK
In this section, we introduce some related work in the fields of MDPs. In robust MDP, the set of all possible transition kernels is called uncertainty set, which can be defined in various ways: one choice could be likelihood regions or entropy bounds of the environment parameters (White III & Eldeib, 1994 Nilim & El Chaoui 2005 Iyengar 2005 Wiesemann al. 2013); another choice is con- strain the deviation from reference environment through some statistical distance. For example, Osogami (2012) discussed such robust problem where the uncertainty set are defined via Kullback- Leibler divergence, and also uncover the relations between robust MDPs using f-divergence con-
straint and risk-aware MDPs.
Indeed, it was observed that since the robust MDP framework ignores probabilistic information of the uncertainty set, it can provide conservative solutions (Delage & Mannor [0T0] Xu & Mannor 2007). Some papers consider bringing prior knowledge of dynamics to robust MDPs, and name such problem distributionally robust MDPs. Xu & Mannor (2010) discuss robust MDPs with prior infor- mation estimate the confidence region of parameters abound, which is moment- based constraint, and they also show that such distributionally robust problems can be reduced to standard robust MDP problems. Yang 2017| 2018) Wasserstein distance to evaluate the difference among the prior distributions of transition probabilities. However, Yang algorithms are not appropriate for complex situations, because they need to estimate enough transition kernels approximate prior distribution at each step.
WASSERSTEIN ROBUST REINFORCEMENT LEARNING
In this section, we specify the problem of interest, which is actually minimax problem constrained by some Wassserstein-basee uncertainty We start with introducing general theoretical frame- work, i.c.. robust Markov Decision Process, and then briefly recall the definition of Wasserstein distance between probability measures. Inspired by the strong duality brought by Wasserstein-based uncertainty set, the robust MDP is reformulated to some rask-aware MDP, making connections clear between robustness dynamics and robustness to states.
3.1 ROBUST MARKOV DECISION PROCESS
Unlike ordinary Markov Decision Processes (MDPs), in robust MDP, environmental dynamics, in- cluding transition probabilities and rewards, might change over time (Nilim & El Ghaoui 2004] 2005). Theoretically, such dynamics can be treated as stochastic changes within an uncertainty set. The objective of robust MDP is to find the optimal policy under the worst dynamics.
Given discrete-time robust MDPs with continuous state and action spaces, without loss of general- ization, we only consider the robustness to transition probabilities. Basic elements of robust MDPs include (X' .A. Q, c), where
V: state space, which is Borel measurable metric space.
A: action space, which is a Borel measurable space. Let A(x) A represent all the admissible actions at state x E X, and K,, denote all the possible state-action pairs, i.e.,
KA (z,, a): X, E A(x)).
Q: the uncertainty set that contains all possible transition kernels.
c: KA + R. the immediate cost function. Generally we assume it is continuous and [0, c] for some non-negative constant
2

---

Under review as a conference paper at ICLR 2020
The robust system evolves in the following way. Let N denote the current time and In E X' the current state. Agent chooses an action an E A(r,) and environment selects a transition kernel In. from the uncertainty set Q respectively. Then at the next time n + 1, an agent observes an immediate cost c( an) and a new state In+1 E X which follows the dis- each stage and produces trajectories in form of (Io, do; 9o, Co 21, @1, C1,....... Let Ω = (X x A x Q x [0, 20)0 denote all the trajecto- ries. Let On {wn, (2o, ao go, Co2 21, d1, Q1, C1 c1, ... In)} denote all trajectories up to time n
tribution qu (-]arn; an). The process repeats
and Q. fwh, = (zo: ao. 9a, co. IT, a, q1 C1 .... Iny a,)) denote all trajectories up to time n with
action (nn-
Correspondingly, randomized policy is a series of stochastic kernels: = (Ko, 11, T2, where Tn (+14x) is a probability measure over A(rn). We name и primal policy and use II to represent all such randomized policies. If TT, (-(l,)) = Tn (-(rx) for n 2 O, we say the policy is Markov. Tn = To for any 2 0, this policy is stationary. If there exists measurable functions fn Sn A such that nn (fn (wnn (w,) = 1, n 2 0. this policy is called deterministic. We denote the set of all such deterministic, stationary, Markov policies by F.
The selection of transition kernels can be treated as a deterministic policy deployed by a secondary adversarial agent. Let g = (gu, I1 92. ...) with In : át,, + Q denote the adversarial policy. We use G to represent all such deterministic policies. Similarly, if gn (-)@;,) = gn (-1ar an) for all n O, the policy is Markov, and if 9n = 9o for any n 2 0, the policy is stationary.
Given the initial state Xo E X' primal policy πT II and adversarial policy 9 G. applying the lonescu-Tulled theorem (Hernández-Leemm & Lasserre 2012a Bertsekas & Shreve, 2004). there exist a probability measure T on trajectory space. Let 12 denote the corresponding expectation operation.
As for the performance criterion, we consider the infinite-horizon discounted cost. Let 7 E (0,1) be the discounting factor. The discounted cost contributed by trajectory w E Ω is (.(w) ===0>"((z a,). Given the initial state zo = x. policies and g. the expected infinite-horizo horizon discounted cost is
(z)
T p "eexn an)].
Robust MDPs aim find the optimal policy x² for the agent under the worst realization of g E G, which means that x reaches
inf sup C7
#
This minimax problem can be seen as zero-sum game of two agents.
3.2 WASSERSTEIN DISTANCE
The popular Wasserstein distance is a special case of optimal transport costs, which measures the discrepancy between two probabilities in terms of minimum total costs associated with some trans- port function. For any two probability measures Q and P over the measurable space (X, B(2)), let E(Q, P) denote the set of all joint distributions on X' x X with Q and P are respective marginals. Each element in E(Q, P) is called a coupling between Q and P. Let K X' x X > [0, xc) be the transport cost function between two positions, which is non-negative, lower semi-continuous and satisfy s(2, y) = if and only if 2 y. Intuitively, the quantity s(2, y) specifies the cost of transporting unit mass from & in X' to another element y of X. Then the optimal transport total cost associated with i is defined as follows:
Ur n(z ((E
P)( inf SEE(Q.P)
Therefore, the optimal transport cost D₂rQQ. P) corresponds to the lowest transport cost that can be obtained among all couplings between Q and P. Let the transport cost function be some distance metric d on X, and then it is actually the Wasserstein distance of first order. Wasserstein distance of order p is defined as:
W, W,(QP)) inf
d(z, d(2,8)) .P21. 1.
3

---

Under review as a conference paper at ICLR 2020
Unlike Kullback-Liebie divergence other likelihood-basee divergence measures, Wasserstein distance is a proper metric on the space probabilities. More importantly, Wasserstein distance does not restrict probabilities to share the same support (Villani 2008; Santambrogio 2015). Let d(2, y) =II z 2. k(2, y) = ll 2 12 and 6 = ce, the e- Wasserstein ball of order and the ball are identical:
{Q: (Q, P) D)
Due to its superior statistical properties, Wasserstein-based uncertainty set has recently received great deal of attention in DRSO problem (Gao & Kleywegt 2016 Esfahani & Kuhn 2018 Blanchet & Murthy 2019), adversarial example (Sinha et al. 2017 and so We will apply it to robust RL.
3.3
MAIN RESULT
Let the uncertainty set Q be e e-Wasserstein ball of order p centered at some reference transition kernel P:
@ (Q: W a), P(l,,, a)) & V(z, a) E kA} ={Q Dx((((I(,, P(+r, 6, V(z a) E KA),
(3)
The radius or ò reflects the extent of adversarial perturbation to the reference transition kernel P. The difference between our theoretical framework and Yang (017:: 2018) is that our framework is trying to find the optimal solution for the worst transition kernel within the Wasserstein ball, while theirs trying to find the optimal solution for the worst distribution over transition kernels.
Recall the state value function ( at state zo given primal policy π and adversarial policy we can rewrite the state value function as follows,
C7 (2o))
=E [> c(In, an)
= -B [c( ao) + EE (1 11
xs (-100,0)) [ 7 clann an)l
=Ba -000mm..00 [c(20, ao) + x1EX
go(dx ao) )o 2xxx x (x1)).
where az) (T,, ..... and (1) (g1, (91.92.... ..... are the shift policies. Since is continuous and bounded, the value function is actually continuous in X and belongs to [0, 1-
Let X R be measurable, upper semi-continuous function with E 10, اافر and let U denote the of all such functions. For state E X' and action A(x). Consider the following operator 4d defined on U::
QEQ JyEX (H) c(z, a) + sup y / Q(dy)z,
(5)
Applying Lagrangian method and the strong duality property brought by Wasserstein dis- tance (Blanchet & Murthy 2019), we reformulate (5) to the following form:
of P(dyla, ( ()(sup(u(=) As(2, =ex
(H) inf c(x,,)) a) 7A8 + +7 X
(6)
yEX
The significance of this strong dual representation lies on the fact that the operator supg in eq. (5) is replaced by supsex in eq. (6). which leads much easier optimization algorithm. The right- hand side of eq. (6) is a normal iterated-risk function. That is, reduces the infinite-dimensional rroaabiltt--sarccing problem (5) into ordinary fiite-dimennional optimization problem (6).
is easy to verify that /!" maps U to U. Thus, given a state x E and agent policy æ, we have the following expected Bellman-form operator:
(H)))(()) = [
((dalxx) [(da())(I)uxx)
JaEA(a)
= inf 7A0+ + A20
[(dalx)(e(e,,)) a) +7 P(dyla, a) ()sup(u(2)) - (x(2,8))) JaEA(z)
zEx
4

---

Under review as a conference paper at ICLR 2020
Similarly, H maps U to U as well. Under the following Assumption are able to define the optimal iteration operator and show its contraction property.
Assumption 1. X' is compact metric space. For any x E X, A(æ) is compact and H is lower semi-continuous on a E A(x).
Then, given an initial state E X, the following optimal operator over U is well-defined. inf aEA(a)
(Hu)(z)
H²v(x)
inf dx,a) + 7āo + / P(dylæ, a) [sup(u(=) - Ax(z, QEA(a)A20
(8)
=E
It is simple to verify that H maps U U. The contraction property of H is shown in Lemma put the proof in the appendix.
We
Lemma 1. H a contraction operator in U under Lee norm. There exists an unique element in U, denoted as w', satisfying Hu² = u²
For any un U. Un Hun-1 H ug. Due the contraction, we have
lim Un lim H = u., n-+00 2-+30
which indicates an iterative procedure of finding the optimal value function. Based on this optimal value function, policy who deterministic, Markov and stationary, as shown in Theorem We put the proof the appendix.
demonstrate the existence of optimal policies, and single out optimal
Theorem 1. There exists deterministic Markov stationary policy f E F that satisfies
H'u Hu" =u"
We now obtain the existence of an unique robust optimal value function, well robust optimal policies, which is deterministic, Markov and stationary. Through an iterative procedure as ( we can design corresponding algorithms for robust Reinforcement Learning.
3.4 SENSITIVITY ANALYSIS
Before going to the algorithm design, we present a sensitivity analysis for the optimal value function w.r.t. the radius and the Wasserstein order p. Let X" and 2 2y A*) arg m
X *=x(2,)) be solution of equation (8), and X is non-negative.
X* which means the worst transition kernel within our fixed easseerstein ball, equation g can be reduced to an ordinary problem:
(Hu)(z) inf c(x,a) + u()) aεA(z)
=EX
Thus u² has nothing to do with 8 or p.
If X > O, via the envelop theorem, the gradient of optimal value function w.r.t. can be calculated follows.
Ou'
7X 0.
(10)
05
This gradient remains positive. That is, the optimal value function increases as the volume of Wasserstein ball increases (remember that 8 Le and the value function represents the discounted cost). Similarly, via the envelop theorem, the gradient w.r.t. p can be calculated follows.
Ou' (x) = - S Op
1 112 -
2 (v, - - 1
(y( 2
(1I)
Since X > 0, the worst transition kernel Q* satisties WP(QPP)) = i.. Da(Q" P) = 8 Notice that calculating 2" (y, X*) for y is actually trying to find optimal transport map T x X,
Derived from the fact that if W (Q'Q", P) e, there must be X 0.

---

Under review as a conference paper at ICLR 2020
which substantially perturbs P to Q". Recall that u² is upper semi-continuous and its domain compact, and then we can actually regard v" as the Kantorovich potential (Villani 2008 for a transport cost function λ* in the transport from P to Q". Forp > 1, X* is strictly convex. Through theorem 1.17 in Santambrogio (2015), we can write the optimal transport map in an explicit way, as well as the gradient over p.
2" 2" X-) = = - || Vg (v)
V
(v), 1.
Ou' (n²))) Op
p(p - 1) / P(dylæ, alog
7x*
V V yyu"(u) (V)
- p
P-
V V (v)
X 1|2
pp>1. 1.
x
Thus when 1 1 log || V24"(0) 22 for all X' the gradient over p is non oon-negaiiv. Larger p X makes non-negativity more likely to happen. Remember that X actually reflect the extent of ro- bustness, i.e., larger A coincides with smaller radius e. Intuitively, when the volume of Wasserstein ball is very small, the extent of perturbation at each point is small with high probability, making the gradient 11 positive. Thus in such situation, smaller p is preferred.
A-
4 WASSERSTEIN ROBUST ADVANTANGE ACTOR-CRITI ALGORITHMS
reinforcement learning, the agent does not know the precise environment dynamics, i.e., the tran- sition kernel and immediate cost function unknown. Some researchers leverage adversarial agent to inject perturbations into environmental parameters during training procedures (Pinto et al. 2017). However, such methods have to work with pre-defined environmental parameters, and lack of quantified robustness toward transition kernels. Other researchers borrow the idea of adver- sarial examples and disturb observed states in heuristic way (Nguyen al. 2013). They also lose the explanation of robustness towards system dynamics.
we develop robust Advantage Actor-Critic algorithm: critic neural network with parameters w, denoted by thr, is employed to estimate value function; and an actor neural network with parameters denoted by Te, designed as the primal policy. Rewrite equation (8):
Following the analysis Section 3
To(da)))(e(z,a) + 720 + / P(dylæ, ( s)(p (2) Ax((, 8))]
(H (Huw)() inf 8,220 JaEA(z)
ZEX'
Update for and A: Let fu (2; == uz) - Ax(2, v) where x(2,)) p Given y X' and [0, oo). denote
y 1p 1.
yy
arg max (2; *EX
Initially, Zy, can be treated as the maximum perturbation to state E X, given the penalty A.
The gradient of fw over 2 is: Vefu = (2))(2) - A| yll--2( - y). Let Gw = A8 + fx ((dyla,a)(sup.ee ( (2) )x(2,,))) and we get
Az == arg min (A;
Combining the envelope theorem, we obtain the gradient of G w.r.t. X: VAG 5 Ix P(dy|æ, a)x(=y.x,)) The expectation in the gradient can be approximated by Monte Carlo: take action a state x for times; under the reference transition kernel P. observe the next states y and quadruples (x,, ar,,,,,,)), c,y"), 8 1 E R(Zy),As8))
1,2, n, and then we can approximate V Ro
Critic Update Rule: Given state X' and policy no, let
((0,w = ff SEEALE) To(da|=) + (W,(ar,
To calculate J, similarly, leverage Monte Carlo, take actions a, n0(-12), == 2, at the same state x for m times, observe m "state-action" pairs (r, az), i i= 1, ...... m. and then approximate J(0, w. [o(o, az) + G (ur,ai a;)].

---

Under review as a conference paper at ICLR 2020
Let e(x,aa) c(z, a,) yGu Ar, a.) - (2), and e(z) denote the difference between the observed cost and the critic network:
1 (x) x m
1 Σ Jc(a, a,) (Aaaaa twe (2)]
e(z) == J(O,w, w. x)-
e(a, a;)
Through the envelope theorem, we can obtain the following gradient of e(x) w.r.t. wt
V Vie((x== =
Σ 1=1
P(dy xr ai) uw (Zvidaa)))))) V the (x)
(12)
71
72
wil (Z (o)).
(13)
17871
Notice that we should actually update the critic network via minimizing {e(x)?, and the gradient is
10 exx)² e(x)) vee(x).
In practice, we usually can let m = n = 1 to obtain faster convergence.
Actor Update Rule: In classical AC algorithms, directly minimizing "state-action" value function JO, w. may cause large variance and slow convergence, and optimizing the advantage function better choice instead. The advantage function is
A(x, a) = c(x, a) GG (Az,aiz, (x) = e(xx a).
Thus we can find the optimal 0 via minimizing the expected advantage function A(x, = SaEA(z) To mo(da(r)e(z; a). Similarly, we can approximate the gradient of A w.r.t. 0 as follows:
VeA(z,0) 1/c JAEAGH)
m
To(da x V log (o
Volog (2, az)e(z, (14)
m
Finally, we obtain a corresponding Robust Advantage Actor-Critic algorithms. We name Wasserstein Robust Advantage Actor-Critic algorithm with order p, described in Algorithm Algorithm Algorithm [ is actually an inner loop that certifies the extent of perturbations, while Algorithm finds the optimal policy in normal way. Let the learning rates satisfy the Robbins- Monro condition (Robbins & Monro 1951). and 31 ((32), 32 o(33). 8r = o(B3,), and via the multi-time-scaal scales theory (Borkar 2008). the convergence a local minimum can be guaranteed.
I and
EXPERIMENTS
In this section, will verify WRAAC algorithm in Cart-Pole environment State space has four dimensions, including cart position, cart velocity, pole angle and pole velocity at tip. There are only two admissible actions: left right. The target is to prevent the pole from falling over.
Our baseline includes the ordinary Advantage Actor-Critie algorithm. Policies are learnt under the default environment for WRAAC and the baseline. Then, we test the performances of these two policies under different environmental dynamics. We change the simulated environmental parame- ters such as gravity or pole-length emulate different test dynamics. Note that the unit change on gravity and pole-length will result in different extents of the dynamic's robustness.
We apply WRAAC algorithm of order 2, and fix the degree of dynamical robustness at 8 10. For each quadruple (I, a, r. y). if is not the last state of the trajectory, we set initial À be 0 and initial 2 y x (0, /26 0, /26 (designed according to the simulated dynamics of Cart-Pole). If is the last state, we set X = 0 and = = y- The baseline policy and WRAAC are tested in environments with different gravity or different pole-length, shown in Figure and Figure
Remember that different parameters in the Cart-Pole environment have different effects to the dy- namic's robustness. We can see that our robust algorithm changes smoothly as parameter changes,

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Calculating Perturbations.
Input: x E X, w, E A(x), 2 0. 0, ge = 0, discount factor a, order 2 1, K = 0, learning rates 31, 32. for j = 1,2. ... 12 do
collect roll-out (x, cl 1²). 2d + y²
2 update:
9: uuwl V2uw(()
X(122 1/1l (1--)) - y²);
2² + 2 + 31 9z:
- e + a² a 16 + [u 221130 yy [|T]] (x)
9e - 9e + aa ww
ww
K K || yd
end for
À update:
gx 8 LK,
A - A + 32 9x:
9e 9
Output: e, 9e. A.
Algorithm Wasserstein Robust Advantage Actor-Critic Algorithm with Order p.
Input: E X, 0, w, 2 0, discount factor % order 2 1, learning rates B3, B4
for each step do
E = 0, gø = 0.
for = 1,2,
m do sample 0, (=r-)=);; use Algorithm and obtain e, Ic.
C, + F. +e
DE DE Ie
end for
update:
B3 E1 g
w f- w
update:
1= Volog a.)e:
< 34 9o
state update:
choose ~ Ioe (+1x). and collect roll-out (a, a, c, y)
end for
Output: 0,
while the baseline plunges. When the perturbation of parameter reaches some level (related with the fixed 10), our robust policy keeps the pole from falling over for a longer time, which indicates that our algorithm does learn some level of robustness, compared with baseline. If the perturbation of parameter is small, the baseline performs better, due to the fact that the perturbed environment is close to the default environment.
CONCLUSIONS
In this paper, investigate the robust Reinforcement Learning with Wasserstein constraint. The de- rived theoretical framework can reformulated into a tractable iterated-rin -isk aware problem and the theoretical guarantee is then obtained by building connection between robustness transition prob- abilities and robustness to states. Subsequently, we demonstrate the existence optimal policies, provide sensitivity analysis to reveal the effects of uncertainty set, and design a proper two-stage
8

---

Under review as a conference paper at ICLR 2020
Figure 1: Robustness to gravity.
Figure 2: Robustness to length.
learning algorithm WRAAC. The experimental results on the Cart-Polc environment verified the effectiveness and robustness of our proposed approaches.
Future works may favor a complete study for the effects of the radius of Wasserstein ball in our WRAAC algorithm. We are also interested in studying robust policy improvement in data-driven situation where we only have access to the set of collected trajectories.
REFERENCES
Christopher G Atkeson and Jun Morimoto. Nonparametrie representation of policies and value functions: A trajectory-bast approach. Advances in neural information processing systems, pp. 1643-1650, 2003.
Dimitir Bertsekas and Steven Shreve. Stochastic optimal control: the discrete-tine case. 2004.
Jose Blanchet and Karthyek Murthy, Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 2019.
Vivek Borkar. Stochastic approximation: dynamical systems viewpoint. Baptism's 91 Witnesses, 2008.
Erick Delage and Shie Mannor. Percentile optimization for markov decision processes with param- eter uncertainty. Operations research, 58(1):203-211 2010.
Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization
using the wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming, 171(1-2):1 15--11666 2018.
Rui Gao and Anton J Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. arXiv preprint arXiv:1604.022 021999, 2016.
Onésimo Hermandez-Lerma and Jean B Lasserre. Discrete-time Markov control processes: basic optimality criteria, volume 30. Springer Science & Business Media, 2012a.
Onésimo Hernández-Lerma and Jean B Lasserre. Further topics on discrete-time Markov control processes, volume 42. Springer Science & Business Media, 2012b.
Sandy Huang, Nicolas Papernot Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint 1702:0170,,2284, 2017.
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint 1oX.v:770.006455 2017.
Yen-Chen Lin, Zhang- Wei Hong, Yuan Hong Liao, Meng-Li Li Shih, Ming- Yu Liu, and Min Sun. Tac- tics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXiv:1703.06743 2017.

---

Under review as a conference paper at ICLR 2020
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of poysically-plausibe perturbations. In 2017 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3032-3992299 IEEE, 2017.
Shie Mannor, Duncan Simester, Peng Sun, and John Tsitsiklis. Bias and variance in value function estimation. In Proceedings of the twenty-firs, international conference Machine learning, pp- 72. ACM, 2004.
Shie Mannor, Duncan Simester, Peng Sun, and John N Tsitsiklis. Bias and variance approximation in value function estimates. Management Science, 53(2):308-322, 2007.
Jun Morimoto and Kenji Doya. Robust Robust reinforcement learning. Neural computation, 17(2):335-359, 2005.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi-
dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Arnab Nilim and Laurent El Ghaoui. Robustness in markov decision problems with uncertain tran- sition matrices. In Advances in Neural Information Processing Systems, pp. 839-846, 2004.
Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.
Takayuki Osogami. Robustness and risk-sensitivity in markov decision processes. In Advances in Neural Information Processing Systems, pp. 233-241, 2012.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, pp. 2040-2042. International Foun- dation for Autonomous Agents and Multiagent Systems, 2018
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2817-2826 JMLR. org. 2017.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint 16Xiv:1610.012 2016.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkduser, NY, pp. 99-102, 2015.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprini arXiv: 10571, 2017.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Chelsea C White III and Hany K Eldeib. Markov decision processes with imprecise transition probabilities. Operations Research, 42(4):739-749, 1994.
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathe- matics of Operations Research, 38(1):153-183, 2013.
Huan Xu and Shie Mannor. The crauustesss-perffmmaae tradeoff in markov decision processes. In Advances in Neural Information Processing Systems, pp. 1537-1544, 2007.
Huan Xu and Shie Mannor. Distributionally robust markov decision processes. In Advances in Neural Information Processing Systems, pp. 2505 2013,251,,22 2010.
Insoon Yang. A convex optimization approach to distributionally robust markov decision processes with wasserstein distance. IEEE control systems letters, 161) 2017.
Insoon Yang. Wasserstein distributionally robust stochastic control: A data-driven approach. arXiv preprint arXiv: 09808, 2018
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
The trajectory space F), where F is the o-algebra of Ω satisfies
(Xo x) 1,
P Pr (,)) Tn (dalww))
[T9 (w)) E dq).
(X E dar One (n = Un (Xa+1 E dalwn, an).
Proof of Lemma
Proof. (1) First, for {ui, U2) C U, uf 2 u2, it's easy have Hu, 2 Huz, i.., the operator H is monotone about
(2) For any real constant C and U, we can verify that H u + C) = Hu + 7C.
(3) For any u1 U, u2 E U, there is I U2 1| U2 loo Combining (1) and (2), have
Huj & Huz + 7/|uu - u2lloo. i... Huj Huz S 711u1 - uglloo. Thus || Huzl xo S
y||u1 walloe. Furthermore, since (0, 1), the operator H has the contract property in U under
Los norm.
(4) Via Banach fixed-point theorem, there exist an unique x" U satisfying Hu" u²
Proof of Theorem 1
Proof. Due to Assumption for any u E U, it is a measurable function on K, an (H) is lower semi-continuous w.r.t. Based the measurable selection theorem (see Lemma 8.3.8 in [Hernández-Lerma & Lasserre 2012b)), there is deterministic Markov stationary policy / E F. satisfying H' Hu u.


Reviews:
[{"review_id": "rkxqmGa3KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this paper, the authors study the problem of robust MDP (RMDP), where the feasibility set is defined as a Wasserstein ball around the reference transition probability. After formulating the problem, they first prove the contraction of the resulting operator and the existence of a deterministic stationary Markov optimal policy (Sec. 2.3). Then, they provide a sensitivity analysis of the optimal value function w.r.t. the radius and order of the Wasserstein ball (Sec. 2.4). Finally, they propose an actor-critic algorithm to solve the Wasserstein RMDP problem (Sec. 3) and evaluate its performance using simple experiments (Sec. 4). \n\nComments:\n- Section 2.3, which the authors call it Main Result, is not that novel. The Wasserstein RMDP is just a state-action-rectangular RMDP with convex ambiguity set, and Lemma 1 and Theorem 1 are known to be true for such RMDPs. \n- The sensitivity analysis section (Sec. 2.4) is interesting and useful, although it could have been written much better. \n- The algorithm is new, although its section (Sec. 3) is not well-written. There is no analysis for the convergence of the algorithm either. It is a multi-time-scale stochastic approximation algorithm. Similar policy gradient and actor-critic algorithms have been derived in risk-sensitive MDPs for mean-variance, mean-VaR, and mean-CVaR optimization (see the references below), but such algorithms for RMDPs is relatively new. \n- The experimental results are very simple and not very convincing. \n- I would suggest that the authors put their emphasis on the algorithm and try to explain it much better. And support it better with more comprehensive and convincing experimental results. This would definitely improve the quality of the paper.\n\n\nReferences on risk-sensitive MDPs: \n1) A. Tamar, D. Di Castro, and S. Mannor. “Policy Gradients with Variance Related Risk Criteria”. ICML-2012.\n2) Prashanth L.A. and M. Ghavamzadeh. “Actor-Critic Algorithms for Risk-Sensitive MDPs”. NIPS-2013.\n3) Y. Chow and M. Ghavamzadeh. “Algorithms for CVaR Optimization in MDPs”. NIPS-2014.\n4) A. Tamar, Y. Glassner, and S. Mannor. “Optimizing the CVaR via Sampling”. AAAI-2015.\n5) A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor. “Policy Gradient for Coherent Risk Measures”. NIPS-2015.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgYlcYJqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis work aims to produce reinforcement learning methods that are ‘distributionally robust’. They approach this by assuming the transition function may vary between elements of the domain distribution and extend some recent results (Blanchet & Murthy, 2019) to give some theoretical results (contraction and optimal deterministic policy) and an actor-critic algorithm. This is an important research area and the work takes a very nice approach. However, the clarity of the work and the empirical results could both use some work.\n\nThere are a great many grammatical errors throughout the paper. The nature of the errors suggests a non-native speaker, which I really do not want to discourage, but this needs a proofread and edit. One exception, there is an actual typo just after assumption 1 “dfnied”, but actually better grammatically to just remove the word entirely.\n\nIn the intro, the citation of Mannor et al. 2004; 2007 for what is essentially a description of the gap between simulation to real transfer is a very strange choice. Is it possible this was a mistake?\n\nThe description leading up to the main result could be made much clearer. I went through this section, the main results, and the proofs in the appendix. Things look good, although I do feel like (as just said) it could be made much better/clearer. One snag I hit was in the proof of Lemma 1, everything is good except I don’t know why “u_1 \\le u_2 + \\gamma \\| u_1 - u_2 \\|_\\infty” must hold, but I might have simply missed something obvious so please help me out.\n\nThe experimental results are very minimal and not very convincing. It is not immediately clear to me that Figures 1&2 actually show that the robustac algorithm is more robust than ac. The performance is worse than ac, which is completely understandable, but for most of the environments ac is still significantly better.\n\nLastly, a small point to notice that this paper bleeds into the 9th page, and considering the amount of contributions (largely down to the two theoretical results and a very small experiment) I do not think the extra space is warranted. This could be compressed and actually come out stronger as a result of being forced to be clearer and more concise.\n\nUpdate:\n\nThank you for your response, especially the fix to the proof. I will keep my score, but do believe that further experiments and small adjustments to the writing will see a future version of this accepted.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lN7ZnAtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work the authors studied the robust reinforcement learning problem in which the constraint on model uncertainty is captured by the Wasserstein distance. Inspired by the analysis in the distribution-ally robust setting, they derived several sensitivity conditions to study how the radius of the wasserstein ball and the order of the wasserstein distance affect the conservativeness in model uncertainty.  Similar to the robust MDP work, they also show that this robust RL problem has a robust optimal Bellman operator, and the optimal value function can be computed using  robust policy iteration, which can be extended to model-free actor critic algorithm when state/action spaces are large/continuous.\n\nThis work extends the wasserstein uncertainty modeling  to the robust MDP setting. However, I found the contribution rather limited/unclear. First, the proposed robust Bellman optimality result is standard and can be found in many robust MDP work when the uncertainty set is state-action rectangular. The major novel part here is therefore the sensitivity analysis that is specifically tied to wasserstein distance, which has limited novelty. Second, the robust actor critic algorithm is based on a multi-time scale minimax gradient approach, which is also quite standard in this field, and besides asymptotic convergence it is unclear how efficient this algorithm is. Third, the experiment in this paper is very limited (only based on cartpole, and is only compared with the non-robust AC algorithm) and its illustration on the effectiveness of this algorithm is rather limited. More comparisons with other robust MDP methods would be useful to understand how the value of the proposed wasserstein robust RL formulation.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the robust reinforcement learning problem in which the constraint on model uncertainty is captured by the Wasserstein distance. The reviewers expressed concerns regarding novelty with respect to prior work, the presentation or the results, and unconvincing experiments. In its current form the paper is not ready for acceptance to ICLR-2020.
Decision: Reject