Title: ROBUST REINFORCEMENT LEARNING WITH WASSER

Abstract: Robust Reinforcement Learning aims to find the optimal policy with some extent of robustness to environmental dynamics. Existing learning algorithms usually enable the robustness though disturbing the current state or simulated environmental parameters in a heuristic way, which lack quantified robustness to the system dynamics (i.e. transition probability). To overcome this issue, we leverage Wasserstein distance to measure the disturbance to the reference transition kernel. With Wasserstein distance, we are able to connect transition kernel disturbance to the state disturbance, i.e. reduce an infinite-dimensional optimization problem to a finite-dimensional risk-aware problem. Through the derived risk-aware optimal Bellman equation, we show the existence of optimal robust policies, provide a sensitivity analysis for the perturbations, and then design a novel robust learning algorithm-Wasserstein Robust Advantage Actor-Critic algorithm (WRAAC). The effectiveness of the proposed algorithm is verified in the Cart-Pole environment.

1. INTRODUCTION:

Robustness to environmental dynamics is an important topic in safe Reinforcement Learning. Take autonomous vehicle as an example. Autonomous vehicles have to adapt the complex real-world situations, but usually it is unlikely to cover all scenarios during training in real-world environments. To handle this issue, typically, a simulated environment are employed to help build a driving agent, however, the gap between the training and target environments makes the strategies trained with simulated environments sub-optimal to the real-world scenarios (Mannor et al., 2004; 2007) . Learning robust policies from simulated environments is a challenging problem for safe Reinforcement Learning.

For robust Reinforcement Learning algorithms, existing methods lie on two branches: One type of methods, borrowed from game theory, introduces an extra agent to disturb the simulated environmental parameters during training (Atkeson & Morimoto, 2003; Morimoto & Doya, 2005; Pinto et al., 2017; Rajeswaran et al., 2016) . This method has to rely on the environmental characterization. The other type of methods disturbs the current state through Adversarial Examples (Huang et al., 2017; Kos & Song, 2017; Lin et al., 2017; Mandlekar et al., 2017; Pattanaik et al., 2018) , which is more heuristic. Unfortunately, both methods are lack of theoretical guarantee to the robustness extent of transition dynamics.

To address these issues, we design a Wasserstern constraint, which restricts the admissible transition probabilities within a Wasserstein ball centered at some reference transition dynamics. By applying the strong duality of Wasserstein distance (Santambrogio, 2015; Blanchet & Murthy, 2019) , we are able to connect the disturbance on transition dynamics with the disturbance on the current state. As a result, the original infinite-dimensional robust optimal problem is reduced to some finitedimensional ordinary risk-aware RL problem. Through the moderated optimal Bellman equation, we prove the existence of robust optimal policies, provide the theoretical analyse on the performance of optimal policies, and design a corresponding -Wasserstein Robust Advantage Actor-Critic algorithm (WRAAC), which does not depend on the environmental characterization. In the experiments, we verified the robustness and effectiveness of the proposed algorithms in the Cart-Pole environment.

The remainder of this paper is organized as follows. In Section 2, we briefly introduce some related work in Markov Decision Processes. In Section 3, we mainly describe the framework of Wasserstein robust Reinforcement Learning. In Section 4, we propose robust Advantage Actor-Critic algorithms according to the moderated robust Bellman equation. In Section 5, we perform experiments on the Cart-Pole environment to verify the effectiveness of our method. Finally, Section 6 concludes our study and provide possible future works.

2. RELATED WORK:

In this section, we introduce some related work in the fields of MDPs. In robust MDP, the set of all possible transition kernels is called uncertainty set, which can be defined in various ways: one choice could be likelihood regions or entropy bounds of the environment parameters (White III & Eldeib, 1994; Nilim & El Ghaoui, 2005; Iyengar, 2005; Wiesemann et al., 2013) ; another choice is to constrain the deviation from a reference environment through some statistical distance. For example, Osogami (2012) discussed such robust problem where the uncertainty set are defined via Kullback-Leibler divergence, and also uncover the relations between robust MDPs using f -divergence constraint and risk-aware MDPs. Indeed, it was observed that since the robust MDP framework ignores probabilistic information of the uncertainty set, it can provide conservative solutions (Delage & Mannor, 2010; Xu & Mannor, 2007) . Some papers consider bringing prior knowledge of dynamics to robust MDPs, and name such problem distributionally robust MDPs. Xu & Mannor (2010) discuss robust MDPs with prior information to estimate the confidence region of parameters abound, which is a moment-based constraint, and they also show that such distributionally robust problems can be reduced to standard robust MDP problems. Yang (2017; 2018) use Wasserstein distance to evaluate the difference among the prior distributions of transition probabilities. However, Yang's algorithms are not appropriate for complex situations, because they need to estimate enough transition kernels to approximate prior distribution at each step.

3. WASSERSTEIN ROBUST REINFORCEMENT LEARNING:

In this section, we specify the problem of interest, which is actually a minimax problem constrained by some Wassserstein-based uncertainty set. We start with introducing a general theoretical framework, i.e., robust Markov Decision Process, and then briefly recall the definition of Wasserstein distance between probability measures. Inspired by the strong duality brought by Wasserstein-based uncertainty set, the robust MDP is reformulated to some risk-aware MDP, making connections clear between robustness to dynamics and robustness to states.

3.1. ROBUST MARKOV DECISION PROCESS:

Unlike ordinary Markov Decision Processes (MDPs), in robust MDP, environmental dynamics, including transition probabilities and rewards, might change over time (Nilim & El Ghaoui, 2004; 2005) . Theoretically, such dynamics can be treated as stochastic changes within an uncertainty set. The objective of robust MDP is to find the optimal policy under the worst dynamics.

Given discrete-time robust MDPs with continuous state and action spaces, without loss of generalization, we only consider the robustness to transition probabilities. Basic elements of robust MDPs include (X , A, Q, c), where • X : state space, which is a Borel measurable metric space.

• A: action space, which is a Borel measurable space. Let A(x) ∈ A represent all the admissible actions at state x ∈ X, and K A denote all the possible state-action pairs, i.e., K A = {(x, a) : x ∈ X , a ∈ A(x)}. • Q: the uncertainty set that contains all possible transition kernels.

• c: K A → R, the immediate cost function. Generally we assume it is continuous and c ∈ [0, c] for some non-negative constant c.

The robust system evolves in the following way. Let n ∈ N denote the current time and

x n ∈ X the current state. Agent chooses an action a n ∈ A(x n ) and environment selects a transition kernel q n from the uncertainty set Q , respectively. Then at the next time n + 1, an agent observes an immediate cost c(x n , a n ) and a new state x n+1 ∈ X which follows the distribution q n (•|x n , a n ). The process repeats at each stage and produces trajectories in a form of ω = (x 0 , a 0 , q 0 , c 0 , x 1 , a 1 , q 1 , c 1 , ...).

Let Ω = (X × A × Q × [0, c])

∞ denote all the trajectories. Let Ω n = {ω n = (x 0 , a 0 , q 0 , c 0 , x 1 , a 1 , q 1 , c 1 , ..., x n )} denote all trajectories up to time n and Ωn = {ω n = (x 0 , a 0 , q 0 , c 0 , x 1 , a 1 , q 1 , c 1 , ..., x n , a n )} denote all trajectories up to time n with action a n .

Correspondingly, a randomized policy is a series of stochastic kernels: π = (π 0 , π 1 , π 2 , ...) where

π n (•|ω n ) is a probability measure over A(x n ).

We name π primal policy and use Π to represent all such randomized policies. If

π n (•|ω n ) = π n (•|x n ) for n ≥ 0,

we say the policy is Markov. If π n ≡ π 0 for any n ≥ 0, this policy is stationary. If there exists measurable functions

f n : Ω n → A such that π n (f n (ω n )|ω n ) ≡ 1, n ≥ 0,

this policy is called deterministic. We denote the set of all such deterministic, stationary, Markov policies by F.

The selection of transition kernels can be treated as a deterministic policy deployed by a secondary adversarial agent. Let g = (g 0 , g 1 , g 2 , ...) with g n : Ωn → Q denote the adversarial policy. We use G to represent all such deterministic policies. Similarly, if

g n (•|ω n ) = g n (•|x n , a n ) for all n ≥ 0,

the policy is Markov, and if g n ≡ g 0 for any n ≥ 0, the policy is stationary.

Given the initial state X 0 = x ∈ X , primal policy π ∈ Π and adversarial policy g ∈ G, applying the Ionescu-Tulcea theorem (Hernández-Lerma & Lasserre, 2012a; Bertsekas & Shreve, 2004) , there exist a probability measure P π,g x on trajectory space. Let E π,g x denote the corresponding expectation operation.

As for the performance criterion, we consider the infinite-horizon discounted cost. Let γ ∈ (0, 1) be the discounting factor. The discounted cost contributed by trajectory

EQUATION

Robust MDPs aim to find the optimal policy π * for the agent under the worst realization of g ∈ G, which means that π * reaches inf

EQUATION

This minimax problem can be seen as a zero-sum game of two agents.

3.2. WASSERSTEIN DISTANCE:

The popular Wasserstein distance is a special case of optimal transport costs, which measures the discrepancy between two probabilities in terms of minimum total costs associated with some transport function. For any two probability measures Q and P over the measurable space (X , B(X )), let Ξ(Q, P ) denote the set of all joint distributions on X × X with Q and P are respective marginals. Each element in Ξ(Q, P ) is called a coupling between Q and P . Let κ : X × X → [0, ∞) be the transport cost function between two positions, which is non-negative, lower semi-continuous and satisfy κ(z, y) = 0 if and only if z = y. Intuitively, the quantity κ(z, y) specifies the cost of transporting unit mass from z in X to another element y of X . Then the optimal transport total cost associated with κ is defined as follows:

D κ (Q, P ) := inf ξ∈Ξ(Q,P ) X ×X κ(z, y)dξ(z, y) .

Therefore, the optimal transport cost D κ (Q, P ) corresponds to the lowest transport cost that can be obtained among all couplings between Q and P . Let the transport cost function κ be some distance metric d on X , and then it is actually the Wasserstein distance of first order. Wasserstein distance of order p is defined as:

W p (Q, P ) := inf ξ∈Ξ(Q,P ) X ×X d(z, y) p dξ(z, y) 1 p , p ≥ 1.

Unlike Kullback-Liebler divergence or other likelihood-based divergence measures, Wasserstein distance is a proper metric on the space of probabilities. More importantly, Wasserstein distance does not restrict probabilities to share the same support (Villani, 2008; Santambrogio, 2015) . Let d(z, y) = z -y 2 , κ(z, y) = 1 p z -y p 2 and δ = 1 p p , the -Wasserstein ball of order p and the δ-optimal-transport ball are identical:

{Q : W p (Q, P ) ≤ } = {Q : D κ (Q, P ) ≤ δ}.

Due to its superior statistical properties, Wasserstein-based uncertainty set has recently received a great deal of attention in DRSO problem (Gao & Kleywegt, 2016; Esfahani & Kuhn, 2018; Blanchet & Murthy, 2019) , adversarial example (Sinha et al., 2017) , and so on. We will apply it to robust RL.

3.3. MAIN RESULT:

Let the uncertainty set Q be a -Wasserstein ball of order p centered at some reference transition kernel P :

EQUATION

The radius or δ reflects the extent of adversarial perturbation to the reference transition kernel P .

The difference between our theoretical framework and Yang (2017; 2018) is that our framework is trying to find the optimal solution for the worst transition kernel within the Wasserstein ball, while theirs is trying to find the optimal solution for the worst distribution over transition kernels.

Recall the state value function (1) at state x 0 given primal policy π and adversarial policy g, we can rewrite the state value function as follows,

EQUATION

[c(x 0 , a 0 ) + γ x1∈X q 0 (dx 1 |x 0 , a 0 )C

(1) π, (1) g γ

(x 1 )],

where (1) π = (π 1 , π 2 , ...) and (1) g = (g 1 , g 2 , ...) are the shift policies. Since c is continuous and bounded, the value function is actually continuous in X and belongs to [0, c 1-γ ]. Let u : X → R be a measurable, upper semi-continuous function with u ∈ [0, c 1-γ ], and let U denote the set of all such functions. For state x ∈ X and action a ∈ A(x). Consider the following operator H a defined on U:

(H a u)(x) := c(x, a) + sup Q∈Q γ y∈X Q(dy|x, a)u(y).

(5)

Applying Lagrangian method and the strong duality property brought by Wasserstein distance (Blanchet & Murthy, 2019) , we reformulate (5) to the following form:

EQUATION

The significance of this strong dual representation lies on the fact that the operator sup Q in eq. ( 5) is replaced by sup z∈X in eq. ( 6), which leads a much easier optimization algorithm. The righthand side of eq. ( 6) is a normal iterated-risk function. That is, it reduces the infinite-dimensional probability-searching problem (5) into an ordinary finite-dimensional optimization problem (6).

It is easy to verify that H a maps U to U. Thus, given a state x ∈ X and agent policy π, we have the following expected Bellman-form operator:

(H π u)(x) := a∈A(x) π(da|x)H a u(x) = inf λ≥0 γλδ + a∈A(x) π(da|x)[c(x, a) + γ X P (dy|x, a)[sup z∈X (u(z) -λκ(z, y))]].

Similarly, H π maps U to U as well. Under the following Assumption 1, we are able to define the optimal iteration operator and show its contraction property. Assumption 1. X is a compact metric space. For any x ∈ X , A(x) is compact and H a is lower semi-continuous on a ∈ A(x).

Then, given an initial state x ∈ X , the following optimal operator over U is well-defined.

(Hu)(x) := inf a∈A(x)

EQUATION

It is simple to verify that H maps U to U. The contraction property of H is shown in Lemma 1. We put the proof in the appendix.

Lemma 1. H is a contraction operator in U under L ∞ norm. There exists an unique element in U, denoted as u * , satisfying Hu * = u * .

For any u 0 ∈ U, u n := Hu n-1 = H n u 0 . Due to the contraction, we have

EQUATION

which indicates an iterative procedure of finding the optimal value function. Based on this optimal value function, we can demonstrate the existence of optimal policies, and single out an optimal policy who is deterministic, Markov and stationary, as shown in Theorem 1. We put the proof in the appendix.

Theorem 1. There exists a deterministic Markov stationary policy f ∈ F that satisfies

H f u * = Hu * = u * .

We now obtain the existence of an unique robust optimal value function, as well as a robust optimal policies, which is deterministic, Markov and stationary. Through an iterative procedure as (9), we can design corresponding algorithms for robust Reinforcement Learning.

3.4. SENSITIVITY ANALYSIS:

Before going to the algorithm design, we present a sensitivity analysis for the optimal value function w.r.t. the radius δ and the Wasserstein order p. Let λ * and z * (y, λ * ) = arg max z∈X (u(z)λ * κ(z, y)) be a solution of equation ( 8), and λ * is non-negative.

If λ * = 0, which means the worst transition kernel is within our fixed -Wasserstein ball, equation ( 8) can be reduced to an ordinary problem:

(Hu)(x) = inf a∈A(x) c(x, a) + γ sup z∈X u(z).

Thus u * has nothing to do with δ or p.

If λ * > 0, via the envelop theorem, the gradient of optimal value function w.r.t. δ can be calculated as follows.

EQUATION

)

This gradient remains positive. That is, the optimal value function increases as the volume of Wasserstein ball increases (remember that δ =1 p p and the value function represents the discounted cost). Similarly, via the envelop theorem, the gradient w.r.t. p can be calculated as follows.

EQUATION

)

Since λ * > 0, the worst transition kernel Q * satisfies W p (Q * , P ) = , i.e. D κ (Q * , P ) = δ. 1 Notice that calculating z * (y, λ * ) for y is actually trying to find an optimal transport map T p : X → X , which substantially perturbs P to Q * . Recall that u * is upper semi-continuous and its domain is compact, and then we can actually regard u * as the Kantorovich potential (Villani, 2008) for a transport cost function λ * κ in the transport from P to Q * . For p > 1, λ * κ is strictly convex. Through theorem 1.17 in Santambrogio (2015) , we can write the optimal transport map in an explicit way, as well as the gradient over p.

z * (y, λ * ) = T p (y) = y -(λ * ) -1 p-1 ∇ y u * (y) -p-2 p-1 ∇ y u * (y), p > 1. ∂u * (x) ∂p = - γλ * p(p -1) X P (dy|x, a)(log ∇ y u * (y) λ * 2 - p -1 p ) • ∇ y u * (y) λ * p p-1 2 , p > 1.

Thus when:

1 p ≤ 1 -log ∇yu * (y) λ *

2 for all y ∈ X , the gradient over p is non-negative. Larger λ * makes non-negativity more likely to happen. Remember that λ * actually reflect the extent of robustness, i.e., larger λ * coincides with smaller radius . Intuitively, when the volume of Wasserstein ball is very small, the extent of perturbation at each point is small with high probability, making the gradient (11) positive. Thus in such situation, smaller p is preferred.

4. WASSERSTEIN ROBUST ADVANTANGE ACTOR-CRITIC ALGORITHMS:

In reinforcement learning, the agent does not know the precise environment dynamics, i.e., the transition kernel and immediate cost function are unknown. Some researchers leverage an adversarial agent to inject perturbations into environmental parameters during training procedures (Pinto et al., 2017) . However, such methods have to work with pre-defined environmental parameters, and are lack of quantified robustness toward transition kernels. Other researchers borrow the idea of adversarial examples and disturb observed states in a heuristic way (Nguyen et al., 2015) . They also lose the explanation of robustness towards system dynamics.

Following the analysis in Section 3, we develop a robust Advantage Actor-Critic algorithm: a critic neural network with parameters w, denoted by u w , is employed to estimate value function; and an actor neural network with parameters θ, denoted by π θ , is designed as the primal policy. Rewrite equation ( 8):

(Hu w )(x) = inf θ,λ≥0 a∈A(x) π θ (da|x)[c(x, a) + γλδ + γ X P (dy|x, a)[sup z∈X (u w (z) -λκ(z, y))]].

Update for z and λ: Let f w (z; y, λ) := u w (z) -λκ(z, y) where κ(z, y)

= 1 p z -y p , p ≥ 1. Given y ∈ X and λ ∈ [0, ∞), denote z y,λ := arg max z∈X f w (z; y, λ).

Initially, z y,λ can be treated as the maximum perturbation to state y ∈ X , given the penalty λ. The gradient of f w over z is: Combining the envelope theorem, we can obtain the gradient of G w w.r.t. λ: ∇ λ G w = δ -X P (dy|x, a)κ(z y,λ , y). The expectation in the gradient can be approximated by Monte Carlo: take action a at state x for n times; under the reference transition kernel P , observe the next states y j and quadruples (x, a, c, y j ), j = 1, 2, • • • , n; and then we can approximate ∇ λ G w ≈ δ -1 n n j=1 κ(z y j ,λ , y j ). Critic Update Rule: Given state x ∈ X and policy π θ , let J(θ, w, x) := a∈A(x) π θ (da|x)[c(x, a) + γG w (λ x,a ; x, a)].

∇ z f w = ∇ z u w (z) -λ||z -y|| p-2 (z -y). Let G w (λ; x, a) := λδ + X P (dy|x, a)[sup z∈X (u w (z) -λκ(z, y))],

To calculate J, similarly, we leverage Monte Carlo, take actions a i ∼ π θ (•|x), i = 1, 2, • • • , m at the same state x for m times, observe m "state-action" pairs (x, a i ), i = 1, 2, • • • , m, and then approximate J(θ, w, x)

≈ 1 m m i=1 [c(x, a i ) + γG w (λ x,a ; x, a i )].

Let e(x, a i ) := c(x, a i ) + γG w (λ x,a ; x, a i ) -u w (x), and e(x) denote the difference between the observed cost and the critic network:

e(x) := J(θ, w, x) -u w (x) ≈ 1 m m i=1 e(x, a i ) = 1 m m i=1 [c(x, a i ) + γG w (λ x,a ; x, a i ) -u w (x)].

Through the envelope theorem, we can obtain the following gradient of e(x) w.r.t. w:

EQUATION

Notice that we should actually update the critic network via minimizing 1 2 e(x)2 , and the gradient is

∇ w 1 2 e(x) 2 = e(x) • ∇ w e(x).

In practice, we usually can let m = n = 1 to obtain faster convergence.

Actor Update Rule: In classical AC algorithms, directly minimizing "state-action" value function J(θ, w, x) may cause large variance and slow convergence, and optimizing the advantage function is a better choice instead. The advantage function is A(x, a) := c(x, a) + γG w (λ x,a ; x, a) -u w (x) = e(x, a).

Thus we can find the optimal θ via minimizing the expected advantage function A(x, θ) = a∈A(x) π θ (da|x)e(x, a). Similarly, we can approximate the gradient of A w.r.t. θ as follows:

EQUATION

Finally, we obtain a corresponding Robust Advantage Actor-Critic algorithms. We name it Wasserstein Robust Advantage Actor-Critic algorithm with order p, described in Algorithm 1 and Algorithm 2. Algorithm 1 is actually an inner loop that certifies the extent of perturbations, while Algorithm 2 finds the optimal policy in a normal way. Let the learning rates satisfy the Robbins-Monro condition (Robbins & Monro, 1951) , and

β 1 = o(β 2 ), β 2 = o(β 3 ), β 3 = o(β 4

), and via the multi-time-scales theory (Borkar, 2008) , the convergence to a local minimum can be guaranteed.

5. EXPERIMENTS:

In this section, we will verify WRAAC algorithm in Cart-Pole environment 2 . State space has four dimensions, including cart position, cart velocity, pole angle and pole velocity at tip. There are only two admissible actions: left or right. The target is to prevent the pole from falling over.

Our baseline includes the ordinary Advantage Actor-Critic algorithm. Policies are learnt under the default environment for WRAAC and the baseline. Then, we test the performances of these two policies under different environmental dynamics. We change the simulated environmental parameters such as gravity or pole-length to emulate different test dynamics. Note that the unit change on gravity and pole-length will result in different extents of the dynamic's robustness.

We apply WRAAC algorithm of order 2, and fix the degree of dynamical robustness at δ = 10. For each quadruple (x, a, r, y), if y is not the last state of the trajectory, we set initial λ be 0 and initial z be y + δ × (0, 1 √ 26 , 0, 5 √ 26 ) (designed according to the simulated dynamics of Cart-Pole). If y is the last state, we set λ ≡ 0 and z ≡ y. The baseline policy and WRAAC are tested in environments with different gravity or different pole-length, shown in Figure 1 and Figure 2 .

Remember that different parameters in the Cart-Pole environment have different effects to the dynamic's robustness. We can see that our robust algorithm changes smoothly as parameter changes, Algorithm 1 Calculating Perturbations.

Input: x ∈ X , w, a ∈ A(x), δ ≥ 0, λ ≥ 0, e = 0, g e = 0, discount factor α, order p ≥ 1, κ = 0, learning rates β 1 , β 2 . for j = 1, 2, • • • , n do collect roll-out (x, a, c j , y j ). z j ← y j . z update:

g z ← ∇ z u w (z) -λ(||z j -y j || p-2 )(z j -y j ), z j ← z j + β 1 • g z , e ← e + c j + α[λδ + [u w (z j ) -λ 1 p ||z j -y j || p ]] -u w (x) g e ← g e + α∇ w u w (z) -∇ w u w (x) κ ← κ + 1 p ||z -y j || p , end for λ update: g λ ← δ -1 n κ, λ ← λ + β 2 • g λ , e = 1 n e g e = 1

n g e Output: e, g e , λ.

Algorithm 2 Wasserstein Robust Advantage Actor-Critic Algorithm with Order p.

Input: x ∈ X , θ, w, δ ≥ 0, discount factor γ, order p ≥ 1, learning rates β 3 , β 4 for each step do E = 0, g E = 0. for i = 1, 2, (•|x) , and collect roll-out (x, a, c, y) .

x ← y end for Output: θ, w.

while the baseline plunges. When the perturbation of parameter reaches some level (related with the fixed δ = 10), our robust policy keeps the pole from falling over for a longer time, which indicates that our algorithm does learn some level of robustness, compared with baseline. If the perturbation of parameter is small, the baseline performs better, due to the fact that the perturbed environment is close to the default environment.

6. CONCLUSIONS:

In this paper, we investigate the robust Reinforcement Learning with Wasserstein constraint. The derived theoretical framework can be reformulated into a tractable iterated-risk aware problem and the theoretical guarantee is then obtained by building connection between robustness to transition probabilities and robustness to states. Subsequently, we demonstrate the existence of optimal policies, provide a sensitivity analysis to reveal the effects of uncertainty set, and design a proper two-stage Future works may favor a complete study for the effects of the radius of Wasserstein ball in our WRAAC algorithm. We are also interested in studying robust policy improvement in a data-driven situation where we only have access to the set of collected trajectories.

:

Derived from the fact that if Wp(Q * , P ) < , there must be λ * = 0.

https://gym.openai.com/envs/CartPole-v0/

Figures and Tables:

and we get λ x,a := arg min λ G w (λ; x, a).

Figure 1: Robustness to gravity.

• • • , m do sample a i ∼ π θ (•|x); use Algorithm 1 and obtain e, g e . e i ← e E ← E + e g E ← g E + g e log π θ (x, a i )e i θ ← θ -β 4 • g θ state update: choose a ∼ π θ

Table content: <table><tr><td>end for</td></tr><tr><td>w update:</td></tr><tr><td>w ← w -β 3 • ( 1 m E) • ( 1 m g E ) θ update: g θ = 1 m m i=1 ∇</td></tr></table>

Peer review (iclr2020):
Summary: In this paper, the authors study the problem of robust MDP (RMDP), where the feasibility set is defined as a Wasserstein ball around the reference transition probability. After formulating the problem, they first prove the contraction of the resulting operator and the existence of a deterministic stationary Markov optimal policy (Sec. 2.3). Then, they provide a sensitivity analysis of the optimal value function w.r.t. the radius and order of the Wasserstein ball (Sec. 2.4). Finally, they propose an actor-critic algorithm to solve the Wasserstein RMDP problem (Sec. 3) and evaluate its performance using simple experiments (Sec. 4). Comments: - Section 2.3, which the authors call it Main Result, is not that novel. The Wasserstein RMDP is just a state-action-rectangular RMDP with convex ambiguity set, and Lemma 1 and Theorem 1 are known to be true for such RMDPs. - The sensitivity analysis section (Sec. 2.4) is interesting and useful, although it could have been written much better. - The algorithm is new, although its section (Sec. 3) is not well-written. There is no analysis for the convergence of the algorithm either. It is a multi-time-scale stochastic approximation algorithm. Similar policy gradient and actor-critic algorithms have been derived in risk-sensitive MDPs for mean-variance, mean-VaR, and mean-CVaR optimization (see the references below), but such algorithms for RMDPs is relatively new. - The experimental results are very simple and not very convincing. - I would suggest that the authors put their emphasis on the algorithm and try to explain it much better. And support it better with more comprehensive and convincing experimental results. This would definitely improve the quality of the paper. References on risk-sensitive MDPs: 1) A. Tamar, D. Di Castro, and S. Mannor. “Policy Gradients with Variance Related Risk Criteria”. ICML-2012. 2) Prashanth L.A. and M. Ghavamzadeh. “Actor-Critic Algorithms for Risk-Sensitive MDPs”. NIPS-2013. 3) Y. Chow and M. Ghavamzadeh. “Algorithms for CVaR Optimization in MDPs”. NIPS-2014. 4) A. Tamar, Y. Glassner, and S. Mannor. “Optimizing the CVaR via Sampling”. AAAI-2015. 5) A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor. “Policy Gradient for Coherent Risk Measures”. NIPS-2015.