Title: A DYNAMIC GAME APPROACH TO TRAINING ROBUST DEEP POLICIES

Abstract: We present a method for evaluating the sensitivity of deep reinforcement learning (RL) policies. We also formulate a zero-sum dynamic game for designing robust deep reinforcement learning policies. Our approach mitigates the brittleness of policies when agents are trained in a simulated environment and are later exposed to the real world where it is hazardous to employ RL policies. This framework for training deep RL policies involve a zero-sum dynamic game against an adversarial agent, where the goal is to drive the system dynamics to a saddle region. Using a variant of the guided policy search algorithm, our agent learns to adopt robust policies that require less samples for learning the dynamics and performs better than the GPS algorithm. Without loss of generality, we demonstrate that deep RL policies trained in this fashion will be maximally robust to a "worst" possible adversarial disturbances.

1. INTRODUCTION:

Deep reinforcement learning (RL) for complex agent behavior in realistic environments usually combines function approximation techniques with learning-based control. A good RL controller should guarantee fulfillment of performance specifications under external disturbances, or modeling errors. Quite often in practice, however, this is not the case -with deep RL policies not often generalizing well to real-world scenarios. This can be attributed to the inherent differences between the training and testing environments. Recently, there have been efforts at integrating function approximation techniques with learning-based control, in an end-to-end fashion, in order to have systems that optimize objectives while guaranteeing generalization to environmental uncertainties. Examples include trajectory-based optimization for known dynamics ( [17, 26] ), or trajectory optimization for unknown dynamics such as guided policy search algorithms [1, 14, 16] .

While these methods produce performance efficiency for agent tasks in the real world, there are sensitivity questions of such policies that need to be addressed such as, how to guarantee maximally robust deep RL policies in the presence of external disturbances, or modeling errors. A typical approach employed in minimizing sample inefficiency is to engineer an agent's policy in a simulated environment, and later transfer such policies to physical environments. However, questions of robustness persist in such scenarios as the agent often has to cope with modeling errors and new sensory inputs from a different environment. For continuous control tasks, learned policies may become brittle in the presence of external perturbations, or a slight change in the system dynamics may significantly affect the performance of the learned controller [21] -defeating the purpose of having a robust policy that is learned through environmental interaction .

The contribution of this paper is two-fold:

• first, we provide a framework that demonstrates the brittleness of a state-of-the-art deep RL policy; specifically, given a trained RL policy, we pose an adversarial agent against the fixed trained policy; the goal is to perturb the parameter space of the learned policy. We demonstrate that the most sophisticated deep policies fail in the presence of adversarial perturbations.

• second, we formulate an iterative dynamic zero-sum, two player game, where each agent executes an opposite reaction to its pair: a concave-convex problem follows explicitly, and our goal is to achieve a saddle point equilibrium, where the state is everywhere defined but possibly infinite-valued).

Noting that lack of generalization of learned reward functions to the real-world can be thought of as external disturbance that perturb the system dynamics, we formulate the learning of robust control policies as a zero-sum two player Markov game -an iterative dynamic game (iDG) -that pits an adversarial agent against a protagonist agent.

The controller aims to minimize a given cost while the second agent, an adversary aims to maximize the given cost in the presence of an additive disturbance. We run the algorithm in finite episodic settings and show a dynamic game approach aimed at generating policies that are maximally robust.

The content of this paper is thus organized: we review relevant literature to our contribution in Sec. 2; we then provide an H ∞ background in Sec. 3 . This H ∞ technical introduction will be used in formulating the design of perturbation signals in Sec. 4. Without loss of generality, we provide a formal treatment of the iDG algorithm within the guided policy search framework in Sec. 5. Experimental evaluation on multiple robots is provided in Sec. 6 followed by conclusions in Sec. 7.

2. RELATED WORK:

Robustness studies in classical control have witnessed the formalization of algorithms and computation necessary to carry out stable feedback control and dynamic game tasks (e.g. [2, 15, 19] ). There now exist closed-form and iterative-based algorithms to quantify the sensitivity of a control system and design robust feedback controllers. These methods are well-studied in classical H ∞ control theory. While questions of robustness of policies have existed for long in connectionist RL settings [25] , only recently have researchers started addressing the question of incorporating robustness guarantees into deep RL controllers.

Heess et. al [9] posit that rich, robust performance will emerge if an agent is simulated in a sufficiently rich and diverse environment. [9] proposed a learning framework for agents in locomotion tasks which involved choosing simple reward functions, but exposing the agent to various levels of difficult environments as a way of achieving ostensibly sophisticated performance objectives.

Incorporating various levels of difficulty in obstacles, height and terrain smoothness to an agent's environment for every episodic task, they achieved robust behaviors for difficult locomotion tasks after many episodes ( ≈ 10 6 ) of training. However, this strategy defeats one of the primary objectives of RL namely, to make an agent discover good policies with finite data based on little interaction with the environment. An ideal robust RL controller must come from data-efficient samples or imitations. Furthermore, this approach takes a qualitative measure at building robust signals into the reward function via means such as locomotion hurdles with variations in height, slopes, and slalom walls. We reckon that building such physical barriers for an agent is expensive in the real-world and learning such emergent locomotion behaviors takes a long training time.

Pinto et. al. [20] , posed the learning of robust RL rewards in a zero-sum, two-player markov decision process (MDP) defined by the standard RL tuple {S, A 1 , A 2 , P, R, γ, s 0 }, where A 1 and A 2 denote the continuous action spaces of the two players. Both players share a joint transition probability P and reward R. Pinto's approach assumed a knowledge of the underlying dynamics so that an adversarial policy, π adv θ (u t |x t ), can exploit the weakness in a protagonist agent's policy, π prot θ (u t |x t ). This relied on a minimax alternating optimization process: optimizing for one set of actions while holding the other fixed, to the end of ensuring robustness of the learned reward function. While it introduced H ∞ control as a robustness measure for classical RL problems, it falls short of adapting H ∞ for complex agent tasks and policy optimizations. Moreover, there are no theoretical analyses of saddle-/pareto-point or Nash equilibrium guarantees and the global optimum that assures maximal robustness at π prot θ (u t |x t ) = π adv θ (u t |x t ) is left unaddressed. Perhaps the closest formulation to this work is [10] 's neural fictitious self-play for large games with imperfect state information whereby players select best responses to their opponents' average strategies. [10] showed that with deep reinforcement learning, self-play in imperfect-information environments approached a Nash equilibrium where other reinforcement learning methods diverged.

From a methodical perspective, we formulate the robustness of RL controllers within an H ∞ framework (see [19] ) for deep robot motor tasks. Similar to a matrix game with two agents, we let both agents play a zero-sum, two person game where each agent's action strategy or security level never falls below that of the other. The ordering according to which the players act so that each player acts optimally in a "min max" fashion does not affect the convergence to saddle point in our formulation. We consider the case where the security levels of both players coincide so that the strategy pair of both agents constitute a saddle-point pure strategy [3, p. 19 ].

3. BACKGROUND AND PRELIMINARIES:

Reinforcement learning in robot control tasks consists of selecting control commands u from the space of a control policy π (often parameterized by θ) that act on a high-dimensional state x. The x typically composed of internal (e.g. joint angles and velocities) and external (e.g. object pose, positional information in the world) components. For a stochastic policy π(u t |x t ) the commands influence the state of the robot based on the transition distribution π θ (u t |x t , t). The state and action pairs constitute a trajectory distribution τ

= (x 1 , u 1 , x 2 , u 2 , • • • , x T , u T ).

The performance of the robot on an episodic motor task is evaluated by an accumulated reward function R(τ ) defined as

R(τ ) = T -1 t=0 r t (x t , u t ) + r t f (x t f )

for an instantaneous reward function, r t , and a final reward, r t f . Many tasks in robot learning domains can be formulated as above, whereby we choose a locally optimal policy π θ that optimizes the expectation of the accumulated reward

π θ = E(R(τ )|π θ ) = R(τ )p π θ (τ )dτ ,

where p π θ (τ ) denotes distribution over trajectories τ and is defined as

p π θ (τ ) = p(x 1 ) T -1 t=0 π θ (u t |x t )p(x t+1 |x t , u t ).

p(x t+1 |x t , u t ) above represents the robot's dynamics and its environment.

Given the inadequacy of value function approximation methods in managing high-dimensional continuous state and action spaces as well as the difficulty of carrying out arbitrary exploration given hardware constraints [7] , we resolve to use policy search (PS) methods as they operate in the parameter space of parameterized policies. However, direct PS are often specialized algorithms that produce optimal policies for a particular task (often using policy gradient methods), and they come with the negative effects of not generalizing well to flexible trajectory optimizations and large representations e.g. using neural network policies.

Guided policy search (GPS) algorithms [1, 14, 16] are able to guide the search for parameterized policies from poor local minima using an alternating block coordinate ascent of the optimization problem, made up of a C-Step and an S-Step. In the C-step, a well-posed cost function is minimized with respect to the trajectory samples, generating guiding distributions p i (u t |x t ); and in the Sstep, the locally learned time-varying control laws, p i (u t |x t ), are parameterized by a nonlinear, neural network policy using supervised learning. The S-step fits policies of the form π θ (u t |x t ) = N (µ π (x t ), Σ π (x t )) to the local controllers p i (u t |x t ), where µ π (x t ), and Σ π (x t ) are functions that are estimated.

In order to ensure the learned policy for a dynamical system is robust to external uncertainties, modeling and transfer learning errors, we propose an iterative dynamic game consisting of an agent within an environment, and an adversarial agent, interacting with the original agent in the closed-loop environment E, over a finite horizon, T (it could also be extended to the infinite horizon case). The adversary could represent a spoofing agent in the world or modeling errors between the plant and dynamics. The states evolve according to the following stochastic dynamics p(x t+1 |x t , u t , v t ), ∀ t = 0, ..., T where x t ∈ X t is a markovian system state, u t ∈ U t is the action taken by the agent (henceforth called the protagonist), v t ∈ V t is the action taken by an adversarial agent. The subscripts denote time steps t ∈ [1, T ], allowing for a simpler structuring of the individual policies per time step [7] . The problems we consider are control tasks with complex dynamics, having continuous state and action spaces, and with trajectories defined as

τ = {x 1 , u 1 , v 1 , x 2 , u 2 , v 2 , • • • , x t f , u t f , v t f }.

At time t, the system's controller visits states with high rewards while the adversarial agent wants to visit states with low rewards. The solution to this zero-sum game follows with an equilibrium at the origin -a saddle-point solution ensues.

The policies that govern the behavior of the agents are defined as π θ (u t |x t ) and π θ (v k |x k ) respectively, and the learned local linear time-varying Gaussian controllers are defined as p(u k |x k ), p(v k |x k ). In the next subsection, we show that learned motor policies, p(u k |x k ), are sensitive to minute additive disturbances; we later on propose how to build robustness to such trained neural network policies. This is important in learning tasks where the robustness margins of a trained controller need to be known in advance before being introduced to a new execution environment.

Our goal is to select a suitable policy parameterization, so as to assure robustness and stability guarantees [4] . In this paper, we specifically use convex variant of the mirror descent version of GPS [16] .

4. CASE FOR ROBUSTNESS IN PS ALGORITHMS:

In this section, we show why guided policy search algorithms are non-robust to even the simplest form of perturbations -additive disturbance. Before we proceed, we note that policy search algorithms are popular among the RL tools available because they have a "modest" level of robustness built into them e.g.

• by requiring the learning controller to start the policy parameterization from multiple initial states, • adding a Kullback-Leibler (KL) constraint term to the reward function e.g. [1, 14] ,

• solving a motor task in multiple ways where more than one solution exist [7] ,

• or by introducing additive noise into the system as white noise e.g. differential dynamic programming (DDP) methods [11] or their iLQG variants [23] .

A fundamental drawback of these robustness mechanisms, however, is that the learned policy can only tolerate disturbance for slightly changing conditions; parametric uncertainty is not suitably modeled as white noise, and treating the error as an extra input might be relative to the size of the inputs (drawn from the environment) -necessitating the need for a formal treatment of robustness in PS algorithms.

A methodical way of solving the robustness problem in deep RL would be to consider techniques formalized in H ∞ control theory, where controller sensitivity and robustness are solved within the framework of a differential game. We conjecture that the lack of robustness of a RL trained policy arises from the difference between a plant model and the real system, or the difference in learning environments. If we can measure the sensitivity of a system, γ, then we can aim for policy robustness by ensuring that γ is sufficiently small to reject disturbance arising from the training environment or modeling errors if the gain of mapping from the error space to the disturbance is less than γ -1 [27] . The figure to the right depicts the standard H ∞ control problem. Suppose G in the left inset is a plant for which we can find an internally stabilizing controller, Σ K , that ensures stable transfer of input u to measurement y, the H ∞ control objective is to find the "worst" possible disturbance, v, which produces an undesired output, z; we want to minimize the effect of z. In the right inset in the figure, we treat unmodeled dynamics, transfer errors and other uncertainty as an additional feedback to which we would like to adapt with respect to the worst possible disturbance in a prescribed range. Σ ∆ in the right inset represents these uncertainties; our goal is to find the closed-loop optimal policy for which the plant, G, will satisfy performance requirements and maintain robustness for a large range of systems Σ ∆ . We focus on conditions under Algorithm 1 Guided policy search: convex linear variant

1: for iteration k ∈ {1, . . . , K} do 2: C-step: p i ← arg min pi E pi(τ ) T t=1 r(x t , u t ) such that D KL (p i (τ ) π θ (τ )) ≤ 3:

S-step: π θ ← arg min θ i D KL (p i (τ ) π θ (τ )) (from supervised learning) 4: end for which we can make the H ∞ norm of the system less than a given prior, γ. Specifically, we want to design a controller Σ K that minimizes the H ∞ norm of the closed-loop transfer function T zv from disturbance v to output z defined as

T zv ∞ = sup v z 2 v 2 .

From the small-gain theorem, the system in the right figure above will be stable for any stable mapping ∆ : z → v for ∆ ∞ < γ -1 [19] . In a differential game setting, we can consider a minmax solution to the H ∞ problem for the plant G with dynamics given by ẋ = f (x, u, w) so that we solve an H ∞ problem that satisfies the constraint

T zv ∞ = sup v z 2 2 v 2 2 ≤ γ 2 , or find a control input u that satisfies the constraint V = T t=0 z T t z t -γ 2 v T t v t dt ≤ 0 for all possible disturbances v for which x 0 = 0.

We consider a differential game for which the best control u that minimizes V , and the worst disturbance v that maximizes V are derived from

EQUATION

The optimal value function is determined from the Hamilton-Jacobi-Isaacs (HJI) equation,

min u max v z T t z t -γ 2 v T t v t + ∂V ∂x f (x, u, v) = 0

from which the optimal u and v can be computed. is adjusted based on the formulation in [16] . The dynamics

p i (x k+1 |x k , u k , v k ) = N (f xk x k + f uk u k + f v k v k , F k ) are fitted to samples {x i k+1 , x i k , u i k , v i k }

using a mixture of Gaussian models to the generated samples {x i k+1 , x i k , u i k , v i k } at iteration i for all time k. Specifically, we incorporate a normal-inverse-Wishart prior on the Gaussian model as described in [14, §A.3].

4.1. SENSITIVITY OF A LEARNED RL POLICY:

This section offers guidance on testing the sensitivity of a deep neural network policy for an agent. We consider additive disturbance to a deep RL policy. Our goal is to study the degradation of performance of a trained neural network policy in the presence of the "worst" possible disturbance in the parameter space of the policy; if this disturbance cannot alter the performance of the trained policy, we have some value for the policy parameters in the prescribed range that the decision strategy is acceptable. We follow the model described above, where Σ ∆ denotes the uncertainty injected by the adversary. We arrive at the nominal system from u to y when the transfer matrix of Σ ∆ is zero. We call Σ ∆ the adversary whose control, v's effect on the output z is to be minimized. We quantify the effect of v on z in closed loop using a suitable cost function as a min-max criteria. This can be seen as an H ∞ norm on the system. Suppose the local actions, p(u k |x k ), of the controller belong to the policy space π = [π 0 , ..., π T ] that maximize the expected sum of rewards

EQUATION

Therefore, the augmented reward for the closed-loop protagonist-adversary system becomes

EQUATION

where

(τ ) = r(x t , u t , t)+r(x t f , u t f , t f ), and ( τ ) = r(x t , u t , v t , t)+r(x t f , u t f , v t f , t f ). Essen- tially, ( τ ) = ( τ ) -γ 2 α(v t )

where α(•) can be chosen as a function of the adversarial disturbance v t 1 . We chose α as the L 2 norm of the disturbance v t in our implementation. γ is a sensitivity parameter that adjusts the strength of the adversary by increasing the penalty incurred by its actions. In (2), we carry out the optimization procedure by first learning the optimal policy for the controller; we then fix this optmal policy and carry out the minimization of the augmented reward function with the adversary in closed-loop as in (3) .

As γ → ∞ in (3), the optimal closed-loop policy is for the agent to do nothing, since any action will incur a large penalty; as γ decreases, however, the adversary's actions have a greater effect on the state of the closed-loop system. The (inverse of the) lowest value of γ for which the adversary's policy causes unacceptable performance provides a measure of robustness of the control policy π θ (u t |x t ). For various values of γ, the state-of-the-art robot learning policies are non-robust to small perturbations as we show in Sec. 6.

4.2. ROBUST ZERO-SUM, TWO-PERSON GAMES:

To learn robust policies, we run an alternating optimization algorithm that maximizes the cost function with respect to the adversarial controller (modeled with the worst possible disturbance) and minimizes the cost function with respect to the protagonist's policy. We consider a two-player, zero-sum Markov game framework for simultaneously learning policies for the protagonist and the adversary. We seek to learn saddle-point equilibrium strategies for the zero-sum game:

EQUATION

where we have overloaded notation such that π(

x t ) = π θ (u t |x t ) and π(x t ) = π θ (v t |x t ); (x t , u t , v t ) = r(x t , u t ) + γα(v t )

is the stage cost. π(x t ) denotes that the adversarial actions are drawn from outside of the action space of the protagonist's policy. Fixing a value of γ is equivalent to an assumption on the capability of the adversary or the magnitude of a worst possible disturbance. To validate this proposal, we develop locally robust controllers for a trajectory optimization problem from multiple initial states using (4) as a guiding cost; a neural network function approximator is then used to parameterize these local controllers using supervised learning. We discuss this procedure in the next section.

4.3. ROBUST GPS:

GPS adds off-policy guiding samples to a sample set: this guides the policy toward spaces of high rewards. If p(τ ) is the trajectory distribution induced by the locally linear Gaussian controller p(u t |x t ) and p(u t |x t ) denotes the previous local controller, GPS algorithms reduce the effect of visiting regions of low entropy by minimizing the KL divergence of the current local policy from the previous one as follows,

EQUATION

where H is the entropy term that favors broad distributions, η is a Lagrange multiplier and the first term forces the actions p to be high in regions of high reward. The trajectory is optimized using optimal control principles under linear quadratic Gaussian assumptions [23] . GPS minimizes the expected cost, E π θ (xt,ut) r(x t , u t ) over the joint distribution of state and action pairs given by the marginals π θ (τ ) = p(x 1 ) T t=1 p(x t+1 |x t , u t ). GPS algorithms optimize the cost J(θ) via a split process of trajectory optimization of local control laws and a standard supervised learning to generalize to high-dimensional policy space settings. A generic GPS algorithm is shown in Algorithm 1. During the C-step, multiple local control laws, p i (u t |x t ), are generated for different initial states x i 1 ∼ p(x 1 ). The supervised learning stage (S-step) regresses the global policy π θ (u t |x t ) to all the local actions computed in the C-step. For unknown dynamics, one can fit p(x t+1 |x t , u t ) to sampled trajectories from the trajectory distribution under p(τ ). To avoid divergence in dynamics, the difference between the current and previous trajectories are constrained by the KL divergence as in step 2 in algorithm 1.

The KL divergence p from p in (5) will not optimize for a robust policy in the presence of modeling errors, changes in environment settings or disturbance as we show in the sensitivity section in subsection 4.1. To make the computed neural network policy robust to these uncertainties, we propose a zero-sum, two-person dynamic game scenario in the next section. Fit linear-Gaussian dynamics pi(x k+1 |x k , u k , v k ) using samples in Di 4:

Fit linearized protagonist policy π θi (u k |x k ) using samples in Di 5:

Regress global policies πθi (u

k |x k ), πθi (v k |x k ) with samples in Di 6: C-step: pi ← arg minp u i maxp v i Ep( τ ) K k=1 l(x k , u k , v k ) s.t. DKL(pu i ( τ ) πθu i ( τ )) ≤ 7: S-step: π θ ← arg min θ max θ k,i,j DKL(π θ (u k |x k,i,j ) pu i (u k |x k,i,j

)) (via supervised learning) 8:

Adjust (see [16, §4.2]) 9: end for

5. TWO-PLAYER ZERO-SUM ITERATIVE DYNAMIC GPS:

To guarantee robust performance during the training of policies of a stochastic system, we introduce the "worst" disturbance in the H ∞ paradigm to the search for a good guiding distribution problem. We begin by augmenting the reward function with a term that allows for withstanding a disturbing input

EQUATION

)

where γ 2 v T v allows us to introduce a quadratic weighting term in the disturbing input; γ denotes the robustness parameter. A zero-sum game follows explicitly: the protagonist is guided toward regions of high reward regions while adversary pulls in its own favorite direction -yielding a saddle-point solution.This framework facilitates learning control decision strategies that are robust in the presence of disturbances and modeling errors -improving upon the generic optimal control policies that GPS and indeed deep RL algorithms guarantee.

5.1. TWO-PLAYER TRAJECTORY OPTIMIZATION:

We propose repeatedly solving an MPC-based finite-horizon trajectory optimization problem within the framework of DDP. Specifically, we generalize a DDP variant -the iLQG algorithm of [22] , to a two-player, zero-sum dynamic game as follows:

• we iteratively approximate the nonlinear dynamics, ẋ = f (x t , u t , v t ), starting with nominal control, ūt ; t ∈ [t 0 , t f ], and nominal adversarial input vt ; t ∈ [t 0 , t f ] which are assumed to be available.

• we run the passive dynamics with ūt and vt to generate a trajectory (x t , ūt , vt )

• discretizing time, we linearize the nonlinear system, ẋt , about (x k , ūk , vk ), so that the new state and action pairs become

δx k = x k -xk , δu k = u k -ūk , δv k = v k -vk

δx k , δu k , and δv k are measured w.r.t the nominal vectors xk , ūk , vk and are not necessarily small. The LQG approximation to the original optimal control problem and reward become

δx k+1 ≈ f xk δx k + f uk δu k + f vk δv k (x k , u k , v k ) ≈ δx T k xk + δu T k uk -γδv T k vk + 1 2 δx T k xxk δx + 1 2 δu T k uuk δu + 1 2 γ 2 δv T k vvk δv + δu u T xk δx -γδv T vxk δx + (x k , ūk , vk ) + E(w t ).

where single and double subscripts in the augmented reward denote first-order and second-order derivatives respectively, and f zk are the respective Jacobians e.g.

f xk = ∂f (•) ∂x | k and f xxk = ∂ ∂x ∂f (•)

∂x | k at time k, E(w t ) is an additive random noise term (folded into v t in our implementation); the value function is the cost-to-go given by the min-max of the control sequence

V (x k ) = min Ui max Vi i,j (x k , U i , V j ). Setting V (x k f ) = k f (x k f ),

where k f is the final time step, the dynamic programming problem transforms the min-max over an entire control sequence to a series of optimizations over a single control, which proceeds backward in time as

V (x k ) = min pu k max pv k [ (x k , u k , v k ) + V (f (x k+1 , u k+1 , v k+1 ))].

The Hamiltonian, (•) + V (•), can be considered as a function of perturbations around the tuple {x k , u k , v k }. Given the intractability of solving the Bellman partial differential equation above, we restrict our attention to the local neighborhood of the nominal trajectory by expanding a power series about the nominal, nonoptimal trajectory similar to [18] . We proceed as follows:

• we maintain a second-order local model of the perturbed Q-coefficients of the LQR problem,

(Q k , Q xk , Q uk , Q vk , Q xxk , Q uxk , Q vxk , Q uuk , Q vvk ) 2 , defined thus Q(δx k , δu k , δv k , k) = (x k + δx k , u k + δu k , v k + δv k ) -(x k , u k v k ) -V (f (x k , u k v k )) +V (f (x k + δx k , u k + δu k , v k + δv k )),

• a second-order Taylor approximation of Q(δx k , δu k , δv k , k) in the preceding equation yields

EQUATION

• the best possible (protagonist) action and the worst possible (adversarial) action can be found by performing the respective arg min and arg max operations

δu k = arg min δu k Q(δx k , δu k , δv k ), and δv k = arg max δv k Q(δx k , δu k , δv k )

so that we have the following linear controllers that minimize and maximize the quadratic Q-function respectively:

δu k = -Q -1 uuk Q T uk + Q uxk δx k + Q uvk δv k , δv k = -Q -1 vvk Q T vk + Q vxk δx k + Q vuk δu k .

For nonlinear systems, the inverse of the 2nd partial derivatives of the Hamiltonian with respect to the controls must be strictly positive definite. When the inverse of the Hessians above are nonpositive-definite, we can circumvent this bottleneck by adding a suitably large positive quantity to Q -1 uuk and Q -1 vvk [12, 5] , by replacing the Hessian with an identity matrix (which gives the steepest descent) [23] , or by multiplying by lowest eigenvalue of the matrix. We find that the protagonist and adversary in the above-equations have a local action containing a state feedback term, G, and an open-loop term, g, given by

EQUATION

respectively. The tuple {g u k , G u k , g v k , G v k } can be computed efficiently as shown in (17) . We can construct linear Gaussian controllers with mean given by the deterministic optimal solutions and the covariance proportional to the curvatures of the respective Q functions:

p(u k |x k ) = N (ū + g uk + G uk δx k , Q -1 uuk ), p(v k |x k ) = N (v + g vk + G vk δx k , Q -1 vvk ).

[13] has shown that these types of distributions optimize an objective function with maximum entropy given by arg min

p( τ )∈N ( τ ) E[ ( τ ) -H(p( τ ))] subject to p(x t+1 |x t , u t ) = N (x t+1 ; f xt x t + f ut u t , F t ) (9) while p(v k |x k ) optimizes arg max p( τ )∈N ( τ ) E[ ( τ ) -H(p( τ ))] subject to p(x t+1 |x t , v t ) = N (x t+1 ; f xt x t + f vt v t , F t ) (10)

where τ = (x i t , u i t , v i t ) is the system's trajectory evolution over all states, i, visited by both local controllers, and H is the differential entropy. Equation ( 9) produces a trajectory that follows the widest, highest-entropy distribution while minimizing the expected cost under linearized dynamics and quadratic cost; (10) produces an opposing trajectory to what p(u k |x k ) does by maximizing the expected cost under locally linear quadratic assumptions about the dynamics.

Note that the open-loop control strategies in (8) depend on the action of the other player. Therefore, equations ( 8) ensure we have a cooperative game in which the protagonist and the adversary alternate between taking best possible and worst possible local actions during the trajectory optimization phase. This helps maintain equilibrium around the system's desired trajectory, while ensuring robustness in local policies. Substituting ( 8) into (7) and equating coefficients of

δx k , δu k , δv k to those of V (x k + δx k , k) = V (x k , k) + V xk δx k + 1 2 δx T k V xxk δx k

, we obtain a quadratic value function at time k, through the backward pass given by ( 19) in the appendix.

Say, the protagonist first implements its strategy, then transmits its information to the adversary, who subsequently chooses its strategy; it follows that the adversary can choose a more favorable outcome since it knows what the protagonist's choice of strategy is. It becomes obvious that the best action for the protagonist is to choose a control strategy that is an optimal response to the choice of the adversary determined from

δv k = min p δv k (δx k , δu k , δv k ) = max p δv k min p δu k (δx k , δu k , δv k ).

Similarly, if the roles of the players are changed, the protagonist response to the adversary's worst choice will be

δu k = max δu k (δx k , δu k , δv k ) = min p δu k max pv k (δx k , δu k , δv k ).

Therefore, it does not matter that the order of play is predetermined. We end up with an iterative dynamic game, where each agent's strategy depends on its previous actions. The update rules for the Q coefficients are determined using a Gauss-Newton approximation and is given in (15) in the appendices.

In the forward pass, we integrate the state equation, ẋ, compute the protagonist's deterministic optimal policy and update the trajectory as follows:

EQUATION

Compared to previous GPS algorithms, the local controllers not only produce locally linear Gaussian controllers that favors the widest and highest entropy, they also have robustness to disturbance and modeling errors built into them in the H ∞ sense.

We arrive at a saddle point in the energy space of the cost function and we posit that the local controllers generated during the trajectory optimization phase become robust to external perturbations, modeling errors e.t.c. We arrive at a saddle point in the energy space of the cost function and we posit that the local controllers generated during the trajectory optimization phase become robust to external perturbations, modeling errors e.t.c. The next section shows how we generate the function V (x k ) that guarantees saddle-point equilibria for our examples.

5.2. ESTIMATING DYNAMICS DISTRIBUTION:

The dynamics of the two player system is given by the tuple {x i t , u i t , v i t , x i t+1 } and we fit the system dynamics using piecewise linear functions in the form of a mixture of N Gaussians as proposed in [1] and [14] over the vectors {x i t , u i t , v i t , x i t+1 } T , where the ith index represents the i-th trajectory rollout on the robot. We build a Gaussian Mixture Model (GMM) to fit piecewise linear dynamics so that within each GMM cluster, k i , we represent a linear Gaussian dynamics model as

k i (x i t+1 |x i t , u i t , v i t )

and the marginal k i (x i t |u i t , v i t ) represents the portion of the state-actions space where our Gaussian model is valid.

In order to avoid the GMM not being a good separator of boundaries of complex modes, we follow [1] , and use the GMM to generate a prior for the regression phase. This enables us to obtain different linear modes at separate time steps based on the observed transitions, even when the states are dissimilar. The correct linear mode is obtained from the empirical covariance of {x t , u t , v t } with x t+1 in the current samples at time t. As in [1] and [14] , we improve sample efficiency by refitting the GMM at each iteration to all of the samples at all time steps from the current iteration and the previous 3 iterations and use this to construct a good prior for the dynamics. We then obtain linear Gaussian dynamics by fitting Gaussian distributions to samples {x i t , u i t , v i t , x i t+1 } which are then conditioned on [x t , u t , v t ] T . The prior allows us to build a normal-inverse Wishart prior on the conditioned Gaussians so that the maximum a posteriori estimates for mean µ and covariance Σ are given by

Σ = Φ + N Σ e + N m N + m (µ e -µ 0 )(µ e -µ 0 ) T N + n 0 , µ = mµ 0 + n 0 µ e m + n 0

where Σ e and µ e are respectively the empirical covariance and mean of the dataset and Φ, µ 0 , m and n 0 are prior parameters so chosen: Φ = n 0 Σ and µ 0 = μ. As in [14] , we set n 0 = m = 1 in order to fit the prior to many samples than what is available at each time step.

5.3. SUPERVISED LEARNING OF GLOBAL NEURAL NETWORK POLICIES:

The trajectories from the previous subsection are used to generate training data for global policies for the controller and adversary. The local policies p τ (u k |x k ) and p τ (v k |x k ) will ideally be generated for all possible initial states x i 1 ∼ p(x k ). Since the iLQG-based linearized dynamics will only be valid within a finite region of the state space; we used the KL-divergence constraint proposed in [16] to ensure the current protagonist policy does not diverge too much from the previous policy.

The learning problem involves imposing KL constraints on the cost function such that the protagonist controller distribution agree with the global policy π θ (u t |x t ) by performing the following alternating optimization between two steps at each iteration i:

EQUATION

Essentially, p(u k |x k ) above generates robust local policies; The first step in (12) solves for a robust local policy p(u k |x k ) via the min-max operation, by constraining p(u k |x k ) against its global policy π i using the given KL divergence constraint; the second step projects the local linear Gaussian controller distribution onto the constraint set Π Θ , with respect to the divergence D(p i , π). The local policy that governs the agent's dynamics is given by

EQUATION

Notice that the state is linearly dependent on the mean of the distribution p(u k |x k ) and the covariance is independent of v k ; we therefore end up with a linear Gaussian controller for the robust guided policy search algorithm. For linear Gaussian dynamics and policies, the iterative KL constraint during the S-step translates to minimizing the KL-divergence between policies i.e. ,

D KL (p i ( τ ) π θ ( τ ) = K k=1 E p(u k |x k ) D KL (p(u k |x k ) π θ (u k |x k )).

For the nonlinear cases that we treat in this work, the KL-divergence term in the S-step above is flipped as proposed in [16] so that where x k,i,j is the j th sample from p i (x k ) obtained by running p i (u k |x k ) on the real system, and D i are the trajectory samples rolled out on the system. Our robust GPS algorithm is thus given in algorithm 2. We follow the prior works in [1, 13, 14, 16] in computing the KL divergence term and we refer readers to these works for a more detailed treatment.

D KL (π θ (u k |x k ) p i (u k |x k ) minimizes the augmented stage cost under p i (u k |x k ) w.r.t π θ (u k |x k ). Therefore, the S-step minimizes, i,k E pi(x k ) [D KL (π θ (u k |x k ) p i (u k |x k ))] ≈ 1 |D i | i,k,j D KL (π θ (u k,i,j |x k ) p i (u k,i,j |x k )),

6. EXPERIMENTAL RESULTS:

In this section, we present experiments to (i) confirm our hypothesis that guided policy search methods, with carefully engineered complex high-dimensional policies, fail when exposed to the simplest of all perturbation signals; and (ii) answer the question of robustness using the trajectory optimization scheme and the robust guided policy framework we have presented. We solve this under unknown dynamics.

We answer both questions in this paper by using physics engines for policies that do not use visual features as feedback. Our validation examples are implemented in the MuJoCo physics engine [24] and the pybox2d game engine [6] , aided by the publicly available GPS codebase [8] . Highdimensional policy experiments are implemented on a PR2 robot, while low-dimensional policy experiments are implemented using a 2-DOF cart-pole swing-up experiment in the pybox2d game engine. The perturbation signal we consider are those that enter additively through the reward functions as described in subsection 4.1.

6.1. SENSITIVITY OF AN RL POLICY:

We conducted simulated experiments demonstrating that guided policy search policies are sensitive to disturbance introduced into the action space of their policies. The 7-DoF robot result presented shortly previously appeared in our abstract that introduced robust GPS [21]. The states x k are the joint angles, joint velocities, pose and velocity of the end effector as 3 points in 3-D. We assume the initial velocity of the 2-link and robot arm are zero.

Experimental tasks. We simulated a 3D peg insertion task by a robot into a hole at the bottom of the slot Fig. 1 . The difficulty of this experiment stems from the discontinuity in dynamics from the contact between the peg and the walls.

The 2-link arm swing-up experiment involves learning to balance the arm vertically about the origin of the cart (see right inset of Fig. 1 ). The diffculty lies in the discontinuity of the dynamics along the vertical axis of the arm when it is upright.

We initialized the linear-Gaussian controllers p i (u k |x k ), p i (v k |x k ) in the neighborhood of the initial state x 1 using a PD control law for both the inverted pendulum task and the peg insertion task.

Peg Insertion: We implement the sensitivity algorithm for the peg insertion task of [8] with a robotic arm that requires dexterous manipulation. The robot has 12 states consisting of joint angles and angular velocities with two controller states. We train the protagonist's policy using the GPS algorithm. We then pit an adversarial disturbance against the trained policy so that the adversary stays in closed-loop with the trained protagonist; The closed-loop cost function is given by

EQUATION

where γ represents the disturbance term, d x k denotes the end effector's (EE) position at state x k and d denotes the EE's position at the slot's base. 12 (ζ) is a term that makes the peg reach the target at the hole's base, precisely given by 1

2 ζ t ζ + (α + ζ 2 ) 1 2

. We set w u and w p to 10 -6 and 1 respectively. For various values of γ, we check the sensitivity of the trained policy and its effect on the task performance by maximizing the cost function above w.r.t v k . We run each sensitivity experiment for a total of 10 iterations. Fig. 2 shows that the adversary causes a sharp degradation in the protagonist's performance for values of γ < 1.5. This corresponds to when the GPS-trained policy gets destabilized and the arm struggles to reach the desired target. As values of γ ≥ 1.5, however, we find that the adversary has a reduced effect on task performance: the adversary's effect decreases as γ gets larger. Video of this result is available at https://goo.gl/YmmdhC. Arm Swing-up: Similar to the peg insertion task, we carry out a sensitivity evaluation procedure as we did for the robot arm with the peg insertion experiment with a 2D arm. The goal is to balance a 2D arm vertically about its origin. This agent has 7 states made up of two joint angles, two joint angle velocities and a 3D end effector point. The action space has two dimensions. Contrary to the example in [8] that uses the Bregman alternating direction method of multipliers algorithm, we implement this experiment using the mirror descent GPS algorithm. We proceed as before: first, we optimize the optimal global policy using GPS on the agent; we then fix the agent's policy and pit various adversarial disturbances, controlled by the γ robustness term in order to evaluate its sensitivity. We use a similar cost function as the one used for the peg insertion task. Fig. 3 shows the evolution of the cost function as we vary the values of γ. We notice that the augmented reward function gets larger as the adversary's torque increases in magnitude and for lower values of γ, the augmented cost is relatively low stays the same. The values of γ < 10 12 in Fig. 3 represent the disturbance band where the protagonist's learned policy becomes unstable and the arm never reaches the vertical position (see videos here: https://goo.gl/52rKnt). This experiment further confirms that the state-of-the-art reinforcement learning algorithms fail in the presence of additive disturbances to their parameter space making them brittle when used in situations that call for robustness. To mitigate these sensitivity errors, we implement the robust two-player, zero-sum game framework provided in 5 in order to develop more robust deep RL controllers and mitigate modeling errors and uncertainty. As proposed in section 5, our goal is to improve the robustness of the controller's policy in the presence of modeling errors and uncertainties and transfer errors. We follow the formulation in section 5 and generate v k from zero-mean, unit variance noise samples in every iteration. We employ various values of γ as a robustness parameter and we run the dynamic game during the trajectory optimization phase of the GPS algorithm. Specifically, for the values of γ that the erstwhile policies in the previous subsection fail, we run the dynamic game algorithm to provide robustness in performnace at test time compared against the GPS algorithm. We run experiments on the peg insertion task to verify the algorithm. Figure 3 shows the cost of running the robust GPS algorithm on the 7-DoF robot. We see that the policies that show achieve optimal performance behavior are now less costly compared to vanilla GPS algorithm. For values of the sensitivity term γ that the algorithm erstwhile fails in, we now see smoother execution of the trajectory in trying to achieve our goal. The modeling phase of the algorithm is also much less data consuming as our GMM algorithm now takes less samples before generalizing to the global model.

7. CONCLUSION AND FUTURE WORK:

We have evaluated the sensitivity of select deep reinforcement learning algorithms and shown that despite the most carefully designed policies, such policies implemented on real-world agents exhibit a potential for disastrous performances when unexpected such as when there exist modeling errors and discrepancy between training environment and real-world roll-outs (as evidenced by the results from the two dynamics the agent faces in our sensitivity experiment). We then test the dynamic trajectory optimization two-player algorithm on a robot motor task using Levine et al's [14] 's guided policy search algorithm. In our implementation of the dynamic game algorithm, we focus on the robustness parameters that cause the robot's policy to fail in the presence of the erstwhile sensysensitivity parameter. We demonstrate that our two-player game framework allows agents operating under nonlinear dynamics to learn the underlying dynamics under significantly more finite samples than vanilla GPS algorithm does -thus improving upon the Gaussian model mixture method used in [1] and [14] .

Having agents that are robust to unmodeled nonlinearities, dynamics, and high frequency modes in a nonlinear dynamical system has long been a fundamental question that control theory strives to achieve. To the best of our knowledge, we are not aware of other works that addresses the robustness of deep policies that are trained end-to-end from a maximal robustness perspective. In future work, we hope to replace the crude Gaussian Mixture Model for the dynamics with a more sophisticated nonlinear model, and evaluate how the agent behaves in the presence of unknown dynamics.

APPENDIX I:

The Q-coefficients are estimated using LQR as follows:

EQUATION

where the subscript terms denote the partial derivatives with respect to the given matrix.

APPENDIX II:

The closed-form equations for the closed loop policy obtained after substituting (8) are given by

EQUATION

)

Solving the system of equations in (16) , we find that

EQUATION

)

δv k = Q uvk I -Q -1 vvk Q vuk Q -1 uuk Q vvk -1 Q vuk Q -1 uuk Q uxk -Q vxk δx k + Q vuk Q -1 uuk Q T uk -Q T vk

Suppose we let

K uk = Q uuk I -Q -1 uuk Q uvk Q -1 vvk Q T uvk -1 , K vk = Q vvk I -Q -1 vvk Q vuk Q -1 uuk Q uvk -1 ,

so that

g uk = K uk (Q uvk Q -1 vvk Q vk -Q T uk ), G uk = K uk Q uvk Q -1 vvk Q vxk -Q uxk and g vk = K vk (Q vuk Q -1 uuk Q T uk -Q T vk ), G vk = K vk Q vuk Q -1 uuk Q uxk -Q vxk

it follows that we can rewrite δu k and δv k as

EQUATION

Plugging δu and δv back into the Q function expansion in (7) , we find that

Q(δx k , δu k , δv k , k) = Q T uk g u k + Q T vk g v k + 1 2 g T uk Q uuk g uk + 1 2 g T vk Q vvk g vk (Q T xk + Q T uk Gu k + Q T vk Gv k + g T uk Q uuk G uk + g T vk Q vvk G vk + g T uk Q uxk + g T vk Q vxk + g T uk Q uvk G vk + g T vk Q T uvk G uk )δx k + 1 2 δx T k (Q xxk + G T u k Q uuk Gu + G T v Q vvk Gv + 2G T u k Q uxk + 2G T v k Q vxk + 2G T u k Q uvk Gv k )δx k

Comparing coefficients, we obtain the following for the value function's coefficients

EQUATION

In practice, the Ku k and Kv k inverse terms above will result in numerical errors when the matrices are not positive definite since the DDP algorithm does not guarantee that the inverse of the Q functions will be positive definite. To guarantee numerical stability and preserve the concave-convex properties, we add to Ku k and Kv k a sufficiently large positive quantity (greater than the lowest eigenvalue) in order to regularize the update equations [12, 5] .

:

This formulation assumes that Vt is a vector space, though one can define nonnegative adversary input penalty functions in other settings, e.g. when Vt is a finite set.

whereQ k = (x k , u k , v k , k) + V (x k+1 , k + 1).Vector subscripts indicate partial derivatives.

Figures and Tables:

Robust guided policy search: unknown nonlinear dynamics 1: for iteration k ∈ {1, . . . , K} do 2: Generate samples Di = {τi,j} by running pi(u k |x k ) and pi(v k |x k ) or π θi (u|x k ) and π θi (v|x k ) 3:

Figure 1: Simulation Experiments

Figure 2: Sensitivity Analysis for Peg Insertion Task

Figure 3: [LEFT]: Cost of running the dynamic game-based robust guided policy search algoruithm for various values of gamma for the robot peg insertion task. Our algorithm uses lesser number for the Gaussian mixture models and requires fewer samples to generalize to the real-world. RIGHT: Sensitivity Analysis for Arm Swing-up Task

Peer review (iclr2018):
 The paper presents a method for evaluating the sensitivity and robustness of deep RL policies, and proposes a dynamic game approach for learning robust policies. The paper oversells the approach in many ways. The authors claim that "experiments confirm that state-of-the-art reinforcement learning algorithms fail in the presence of additive disturbances, making them brittle when used in situations that call for robustness". However, their methods and experiments are only applied to Guided Policy Search (GPS), which seems like a specialized RL algorithm. Conclusions drawn from empirically running GPS on a problem cannot be generalized to all "state-of-the-art RL algorithms". In Fig 3, the authors conclude that "our algorithm uses lesser number for the GMMs and requires fewer samples to generalize to the real-world". I*m not sure how this can be concluded from Fig 3 [LEFT]. The two line graphs for different values of gamma almost overlay each other, and the cost seems to go up and down, even with number of samples on a log scale. If this shows the variance in the procedure, then the authors should run enough repeats of the experient to smooth out the variance and show the true signal (with error bars if possible). All related conclusions with regards to the dynamic game achieving higher sample efficiency for GMM dynamics fitting need to be backed up with better experimental data (or perhaps clearer presentation, if such data already exists). Figures 2 and 3 talk about optimal adversarial costs. The precise mathematical definition of this term should be clarified somewhere, since there are several cost functions described in the paper, and it*s unclear which terms are actually being plotted here. The structure of the global policies used in the experiments should be mentioned somewhere. Note about anonymity: Citation [21] breaks anonymity, since it*s referred to in the text as "our abstract". The link to the YouTube video breaks author anonymity. Further, the link to a shared dropbox folder breaks reviewer anonymity, hence I have not watched those videos.