Title: TOWARD EVALUATING ROBUSTNESS OF DEEP REIN-FORCEMENT LEARNING WITH CONTINUOUS CONTROL

Abstract: Deep reinforcement learning has achieved great success in many previously difficult reinforcement learning tasks, yet recent studies show that deep RL agents are also unavoidably susceptible to adversarial perturbations, similar to deep neural networks in classification tasks. Prior works mostly focus on model-free adversarial attacks and agents with discrete actions. In this work, we study the problem of continuous control agents in deep RL with adversarial attacks and propose the first two-step algorithm based on learned model dynamics. Extensive experiments on various MuJoCo domains (Cartpole, Fish, Walker, Humanoid) demonstrate that our proposed framework is much more effective and efficient than model-free attacks baselines in degrading agent performance as well as driving agents to unsafe states.

1. INTRODUCTION:

Deep reinforcement learning (RL) has revolutionized the fields of AI and machine learning over the last decade. The introduction of deep learning has achieved unprecedented success in solving many problems that were intractable in the field of RL, such as playing Atari games from pixels and performing robotic control tasks (Mnih et al., 2015; Lillicrap et al., 2015; Tassa et al., 2018) . Unfortunately, similar to the case of deep neural network classifiers with adversarial examples, recent studies show that deep RL agents are also vulnerable to adversarial attacks.

A commonly-used threat model allows the adversary to manipulate the agent's observations at every time step, where the goal of the adversary is to decrease the agent's total accumulated reward. As a pioneering work in this field, (Huang et al., 2017) show that by leveraging the FGSM attack on each time frame, an agent's average reward can be significantly decreased with small input adversarial perturbations in five Atari games. (Lin et al., 2017) further improve the efficiency of the attack in (Huang et al., 2017) by leveraging heuristics of detecting a good time to attack and luring agents to bad states with sample-based Monte-Carlo planning on a trained generative video prediction model.

Since the agents have discrete actions in Atari games (Huang et al., 2017; Lin et al., 2017) , the problem of attacking Atari agents often reduces to the problem of finding adversarial examples on image classifiers, also pointed out in (Huang et al., 2017) , where the adversaries intend to craft the input perturbations that would drive agent's new action to deviate from its nominal action. However, for agents with continuous actions, the above strategies can not be directly applied. Recently, (Uesato et al., 2018) studied the problem of adversarial testing for continuous control domains in a similar but slightly different setting. Their goal was to efficiently and effectively find catastrophic failure given a trained agent and to predict its failure probability. The key to success in (Uesato et al., 2018) is the availability of agent training history. However, such information may not always be accessible to the users, analysts, and adversaries.

Besides, although it may not be surprising that adversarial attacks exist for the deep RL agents as adversarial attacks have been shown to be possible for neural network models in various supervised learning tasks. However, the vulnerability of RL agents can not be easily discovered by existing baselines which are model-free and build upon random searches and heuristics -this is also verified by our extensive experiments on various domains (e.g. walker, humanoid, cartpole, and fish), where the agents still achieve close to their original best rewards even with baseline attacks at every time step. Hence it is important and necessary to have a systematic methodology to design non-trivial adversarial attacks, which can efficiently and effectively discover the vulnerabilities of deep RL agents -this is indeed the motivation of this work. This paper takes a first step toward this direction by proposing the first sample-efficient model-based adversarial attack. Specifically, we study the robustness of deep RL agents in a more challenging setting where the agent has continuous actions and its training history is not available. We consider the threat models where the adversary is allowed to manipulate an agent's observations or actions with small perturbations, and we propose a two-step algorithmic framework to find efficient adversarial attacks based on learned dynamics models. Experimental results show that our proposed modelbased attack can successfully degrade agent performance and is also more effective and efficient than model-free attacks baselines.

The contributions of this paper are the following:

• To the best of our knowledge, we propose the first model-based attack on deep RL agents with continuous actions. Our proposed attack algorithm is a general two-step algorithm and can be directly applied to the two commonly-used threat models (observation manipulation and action manipulation).

• We study the efficiency and effectiveness of our proposed model-based attack with modelfree attack baselines based on random searches and heuristics. We show that our modelbased attack can degrade agent performance in numerous MuJoCo domains by up to 4× in terms of total reward and up to 4.6× in terms of distance to unsafe states (smaller means stronger attacks) compared to the model-free baselines.

• Our proposed model-based attack also outperform all the baselines by a large margin in a weaker adversary setting where the adversary cannot attack at every time step. In addition, ablation study on the effect of planning length in our proposed technique suggests that our method can still be effective even when the learned dynamics model is not very accurate.

2. BACKGROUND:

Adversarial attacks in reinforcement learning. Compared to the rich literature of adversarial examples in image classifications (Szegedy et al., 2013) and other applications (including natural language processing (Jia & Liang, 2017) , speech (Carlini & Wagner, 2018) , etc), there is relatively little prior work studying adversarial examples in deep RL. One of the first several works in this field are (Huang et al., 2017) and (Lin et al., 2017) , where both works focus on deep RL agent in Atari games with pixels-based inputs and discrete actions. In addition, both works assume the agent to be attacked has accurate policy and the problem of finding adversarial perturbation of visual input reduces to the same problem of finding adversarial examples on image classifiers. Hence, (Huang et al., 2017) applied FGSM (Goodfellow et al., 2015) to find adversarial perturbations and (Lin et al., 2017) further improved the efficiency of the attack by heuristics of observing a good timing to attack -when there is a large gap in agents action preference between most-likely and leastlikely action. In a similar direction, (Uesato et al., 2018) study the problem of adversarial testing by leveraging rejection sampling and the agent training histories. With the availability of training histories, (Uesato et al., 2018) successfully uncover bad initial states with much fewer samples compared to conventional Monte-Carlo sampling techniques. Recent work by (Gleave et al., 2019) consider an alternative setting where the agent is attacked by another agent (known as adversarial policy), which is different from the two threat models considered in this paper. Finally, besides adversarial attacks in deep RL, a recent work (Wang et al., 2019) study verification of deep RL agent under attacks, which is beyond the scope of this paper.

Learning dynamics models. Model-based RL methods first acquire a predictive model of the environment dynamics, and then use that model to make decisions (Atkeson & Santamaria, 1997) . These model-based methods tend to be more sample efficient than their model-free counterparts, and the learned dynamics models can be useful across different tasks. Various works have focused on the most effective ways to learn and utilize dynamics models for planning in RL (Kurutach et al., 2018; Chua et al., 2018; Chiappa et al., 2017; Fu et al., 2016) .

3. PROPOSED FRAMEWORK:

In this section, we first describe the problem setup and the two threat models considered in this paper. Next, we present an algorithmic framework to rigorously design adversarial attacks on deep RL agents with continuous actions.

3.1. PROBLEM SETUP AND FORMULATION:

Let s i ∈ R N and a i ∈ R M be the observation vector and action vector at time step i, and let π : R N → R M be the deterministic policy (agent). Let f : R N × R M → R N be the dynamics model of the system (environment) which takes current state-action pair (s i , a i ) as inputs and outputs the next state s i+1 . We are now in the role of an adversary, and as an adversary, our goal is to drive the agent to the (un-safe) target states s target within the budget constraints.

We can formulate this goal into two optimization problems, as we will illustrate shortly below. Within this formalism, we can consider two threat models:

Threat model (i): Observation manipulation. For the threat model of observation manipulation, an adversary is allowed to manipulate the observation s i that the agent perceived within an budget:

EQUATION

where ∆s i ∈ R N is the crafted perturbation and U s ∈ R N , L s ∈ R N are the observation limits.

Threat model (ii): Action manipulation. For the threat model of action manipulation, an adversary can craft ∆a i ∈ R M such that

EQUATION

where U a ∈ R M , L a ∈ R M are the limits of agent's actions.

Our formulations. Given an initial state s 0 and a pre-trained policy π, our (adversary) objective is to minimize the total distance of each state s i to the pre-defined target state s target up to the unrolled (planning) steps T . This can be written as the following optimization problems in Equations 3 and 4 for the Threat model (i) and (ii) respectively:

EQUATION

EQUATION

A common choice of d(x, y) is the squared 2 distance x -y 2 2 and f is the learned dynamics model of the system, and T is the unrolled (planning) length using the dynamics models.

3.2. OUR ALGORITHM:

In this section, we propose a two-step algorithm to solve Equations 3 and 4. The core of our proposal consists of two important steps: learn a dynamics model f of the environment and deploy optimization technique to solve Equations 3 and 4. We first discuss the details of each factor, and then present the full algorithm by the end of this section.

Step 1: learn a good dynamics model f . Ideally, if f is the exact (perfect) dynamics model of the environment and assuming we have an optimization oracle to solve Equations 3 and 4, then the solutions are indeed the optimal adversarial perturbations that give the minimal total loss with -budget constraints. Thus, learning a good dynamics model can conceptually help on developing a strong attack. Depending on the environment, different forms of f can be applied. For example, if the environment of concerned is close to a linear system, then we could let f (s, a) = As+Ba, where A and B are unknown matrices to be learned from the sample trajectories (s i , a i , s i+1 ) pairs. For a more complex environment, we could decide if we still want to use a simple linear model (the next state prediction may be far deviate from the true next state and thus the learned dynamical model is less useful) or instead switch to a non-linear model, e.g. neural networks, which usually has better prediction power but may require more training samples. For either case, the model parameters A, B or neural network parameters can be learned via standard supervised learning with the sample trajectories pairs (s i , a i , s i+1 ).

Step 2: solve Equations 3 and 4. Once we learned a dynamical model f , the next immediate task is to solve Equation 3 and 4 to compute the adversarial perturbations of observations/actions. When the planning (unrolled) length T > 1, Equation 3 usually can not be directly solved by off-theshelf convex optimization toolbox since the deel RL policy π is usually a non-linear and non-convex neural network. Fortunately, we can incorporate the two equality constraints of Equation 3 into the objective and with the remaining -budget constraint (Equation 1), Equation 3 can be solved via projected gradient descent (PGD)1 . Similarly, Equation 4 can be solved via PGD to get ∆a i . We note that, similar to the n-step model predictive control, our algorithm could use a much larger planning (unrolled) length T when solving Equations 3 and 4 and then only apply the first n (≤ T ) adversarial perturbations on the agent over n time steps. Besides, with the PGD framework, f is not limited to feed-forward neural networks. Our proposed attack is summarized in Algorithm 2 for

Step 1, and Algorithm 3 for Step 2.

Algorithm 1 Collect trajectories 1: Input: pre-trained policy π, MaxSampleSize n s , environment env 2: Output: a set of trajectory pairs S 3: k ← 0, S ← φ 4: s 0 ← env.reset() 5: while k < n s do 6:

a k ← π(s k ) 7: s k+1 ← env.step(a k ) 8: S ∪ {(s k , a k , s k+1 )} 9: k ← k + 1 10: end while 11: Return S

4. EXPERIMENTS:

In this section, we conduct experiments on standard reinforcement learning environment for continuous control (Tassa et al., 2018) Solve Eq. 3 with parameters (π, f, , T ) via PGD to get δ 1 , . . . , δ T 5: else if threat model is action manipulation (Eq. 2) then 6:

Solve Eq. 4 with parameters (π, f, , T ) via PGD to get δ 1 , . . . , δ T 7: end if 8: Return δ 1 , . . . , δ n JoCo Tassa et al. (2018) and corresponding tasks: Cartpole-balance/swingup, Fish-upright, Walkerstand/walk and Humanoid-stand/walk. For the deep RL agent, we train a state-of-the-art D4PG agent (Barth-Maron et al., 2018) with default Gaussian noise N (0, 0.3I) on the action and the score of the agents without attacks is summarized in Appendix A.3. The organization is as follows: we first evaluate the effectiveness of our proposed model-based attack and three model-free baselines in terms of both loss and reward. Next, we conduct ablation study on the key parameter of our algorithm the planning length T , evaluate our algorithm on a weaker attack setting and also discuss the efficiency of our proposed attack in terms of sample complexity.

Evaluations. We conduct experiments for 10 different runs, where the environment is reset to different initial states in different runs. For each run, we attack the agent for one episode with 1000 time steps (the default time intervals is usually 10 ms) and we compute the total loss and total return reward. The total loss calculates the total distance of current state to the unsafe states and the total return reward measures the true accumulative reward from the environment based on agent's action. Hence, the attack algorithm is stronger if the total return reward and the total loss are smaller.

Baselines. We compare our algorithm with the following model-free attack baselines with random searches and heuristics:

• rand-U: generate m randomly perturbed trajectories from Uniform distribution with interval [-, ] and return the trajectory with the smallest loss (or reward), • rand-B: generate m randomly perturbed trajectories from Bernoulli distribution with probability 1/2 and interval [-, ], and return the trajectory with the smallest loss (or reward), • flip: generate perturbations by flipping agent's observations/actions within the budget in ∞ norm.

For rand-U and rand-B, they are similar to Monte-Carlo sampling methods, where we generate m sample trajectories from random noises and report the loss/reward of the best trajectory (with minimum loss or reward among all the trajectories). We set m = 1000 throughout the experiments. More details see Appendix A.2.

Our algorithm. A 4-layer feed-forward neural network with 1000 hidden neurons per layer is trained as the dynamics model f respectively for the domains of Cartpole, Fish, Walker and Humanoid. We use standard 2 loss (without regularization) to learn a dynamics model f . Instead of using recurrent neural network to represent f , we found that the 1-step prediction for dynamics with the 4-layer feed-forward network is already good for the MuJoCo domains we are studying. Specifically, for the Cartpole and Fish, we found that 1000 episodes (1e6 training points) are sufficient Only our proposed attack can constantly make the Walker fall down (since we are minimizing its head height to be zero).

to train a good dynamics model (the mean square error for both training and test losses are at the order of 10 -5 for Cartpole and 10 -2 for Fish), while for the more complicated domain like Walker and Humanoid, more training points (5e6) are required to achieve a low test MSE error (at the order of 10 -1 and 10 0 for Walker and Humanoid respectively). Consequently, we use larger planning (unrolled) length for Cartpole and Fish (e.g. T = 10, 20), while a smaller T (e.g. 3 or 5) is used for Walker and Humanoid. Meanwhile, we focus on applying projected gradient descent (PGD) to solve Equation 3 and 4. We use Adam as the optimizer with optimization steps equal to 30 and we report the best result for each run from a combination of 6 learning rates, 2 unroll length {T 1 , T 2 } and n steps of applying PGD solution with n ≤ T i .

4.1. RESULTS:

For observation manipulation, we report the results on Walker, Humanoid and Cartpole domains with tasks (stand, walk, balance, swingup) respectively. The unsafe states s target for Walker and Humanoid are set to be zero head height, targeting the situation of falling down. For Cartpole, the unsafe states are set to have 180 • pole angle, corresponding to the cartpole not swinging up and nor balanced. For the Fish domain, the unsafe states for the upright task target the pose of swimming fish to be not upright, e.g. zero projection on the z-axis.

The full results of both two threat models on observation manipulation and action manipulation are shown in Table 1a , b and c, d respectively. Since the loss is defined as the distance to the target (unsafe) state, the lower the loss, the stronger the attack. It is clear that our proposed attack achieves much lower loss in Table 1a & c than the other three model-free baselines, and the averaged ratio is also listed in 1b & d. Notably, over the 10 runs, our proposed attack always outperforms baselines for the threat model of observation perturbation and the Cartpole domain for the threat model of action perturbation, while still superior to the baselines despite losing two times to the flip baseline on the Fish domain.

To have a better sense on the numbers, we give some quick examples below. For instance, as shown in Table 1a and b, we show that the average total loss of walker head height is almost unaffected for the three baselines -if the walker successfully stand or walk, its head height usually has to be greater than 1.2 at every time step, which is 1440 for one episode -while our attack can successfully lower the walker head height by achieving an average of total loss of 258(468), which is roughly 0.51(0.68) per time step for the stand (walk) task. Similarly, for the humanoid results, a successful humanoid usually has head height greater than 1.4, equivalently a total loss of 1960 for one episode, and Table 1a shows that the d4pg agent is robust to the perturbations generated from the three modelfree baselines while being vulnerable to our proposed attack. Indeed, as shown in Figure 2 , the walker and humanoid falls down quickly (head height is close to zero) under our specially-designed attack while remaining unaffected for all the other baselines.

4.2. DISCUSSION:

Evaluating on the total reward. Often times, the reward function is a complicated function and its exact definition is often unavailable. Learning the reward function is also an active research field, which is not in the coverage of this paper. Nevertheless, as long as we have some knowledge of unsafe states (which is often the case in practice), then we can define unsafe states that are related to low reward and thus performing attacks based on unsafe states (i.e. minimizing the total loss of distance to unsafe states) would naturally translate to decreasing the total reward of agent. As demonstrated in Table 2 , the results have the same trend of the total loss result in Table 1 , where our proposed attack significantly outperforms all the other three baselines. In particular, our method can lower the average total reward up to 4.96× compared to the baselines result, while the baseline results are close to the perfect total reward of 1000.

Evaluating the effect of planning length. To investigate model effect over time, we perform ablation studies on the planning/unroll length T of our proposed model-based attack in three examples: (I) cartpole.balance (II) walker.walk and (III) walker.stand.

(I) Cartpole balance. Our learned models are very accurate (test MSE error on the order of 10 -6 ).

We observed that the prediction error of our learned model compared to the true model (the MuJoCo simulator) is around 10% for 100 steps. Hence, we can choose T to be very large (e.g. 20-100) and our experiments show that the result of T = 100 is slightly better, see Appendix A.4.

(II) Walker walk. This task is much more complicated than (I), and our learned model is less accurate (test MSE is 0.447). For 10 steps, the prediction error of our learned model compared to the true model is already more than 100%, and hence using a small T for planning would be more reasonable. Table 3a shows that T = 1 indeed gives the best attack results (decreases the loss by 3.2× and decreases the reward by 3.6× compared to the best baseline (randB)) and the attack becomes less powerful as T increases. Nevertheless, even with T = 10, our proposed technique still outperforms the best baseline (randB) by 1.4× both in the total loss and total reward. 3b show that with the more accurate walker.stand model (compared to the walker.walk model), T = 10 gives the best avg total loss& reward , which are 13.4× and 4.9× smaller than the best baseline rand-B. Note that even with T = 1, the worst choice among all our reported T , the result is still 3.5× and 2.9× better than the best baselines, demonstrating the effectiveness of our proposed approach.

The main takeaway from these experiments is that when the model is accurate, we can use larger T in our proposed attack; while when the model is less accurate, smaller T is more effective (as in the Walker.walk example). However, even under the most unfavorable hyperparameters, our proposed attack still outperforms all the baselines by a large margin.

Evaluating on the effectiveness of attack. We study the setting where attackers are less powerful -they can only attack every 2 time steps instead of every transition. Table 4 shows that our proposed attack is indeed much stronger than the baselines even when the attackers power is limited to attack every 2 time steps: (1) compared to the best results among three baselines, our attack gives 1.53× smaller avg total loss (2) the mean reward of all the baselines is close to perfect reward, while our attacks can achieve 1.43× smaller average total reward compared to the best baseline.

Evaluating on the efficiency of attack. We also study the efficiency of the attack in terms of sample complexity, i.e. how many episodes do we need to perform an effective attack? Here we adopt the convention in control suite (Tassa et al., 2018) where one episode corresponds to 1000 time steps (samples) and learn the neural network dynamical model f with different number of episodes.

Figure 3 in Appendix A.1 plots the total head height loss of the walker (task stand) for 3 baselines and our method with dynamical model f trained with three different number of samples: {5e5, 1000, 5000} episodes. We note that the sweep of hyper parameters is the same for all the three models, and the only difference is the number of training samples. The results show that for the baselines rand-U and flip, the total losses are roughly at the order of 1400-1500, while a stronger baseline rand-B still has total losses of 900-1200. However, if we solve Eq. equation 3 with f trained by 5e5 or 1e6 samples, the total losses can be decreased to the order of 400-700 and are already winning over the three baselines by a significant margin. Same as our expectation, if we use more samples (e.g. 5e6, which is 5-10 times more), to learn a more accurate dynamics model, then it is beneficial to our attack method -the total losses can be further decreased by more than 2× and are at the order of 50-250 over 10 different runs. See Appendix A.1 for more details.

Here we also give a comparison between our model-based attack to existing works (Uesato et al., 2018; Gleave et al., 2019) on the sample complexity. In (Uesato et al., 2018) , 3e5 episodes of training data is used to learn the adversarial value function, which is roughly 1000× more data than even our strongest adversary (with 5e3 episodes). Similarly, (Gleave et al., 2019) use roughly 2e4 episodes to train an adversary via deep RL, which is roughly 4× more data than ours2 .

5. CONCLUSIONS AND FUTURE WORKS:

In this paper, we study the problem of adversarial attacks in deep RL with continuous control for two commonly-used threat models. We proposed the first model-based attack algorithm and showed that our formulation can be easily solved by off-the-shelf gradient-based solvers. Extensive experiments on 4 MuJoCo domains show that our proposed algorithm outperforms all model-free based attack baselines by a large margin. We hope our discovery of the vulnerability of deep RL agent can bring more safety awareness to researchers when they design algorithms to train deep RL agents.

There are several interesting future directions can be investigated based on this work, including learning reward functions to facilitate a more effective attack, extending our current approach to develop effective black-box attacks, and incorporating our proposed attack algorithm to adversarial training of the deep RL agents. In particular, we think there are three important challenges that need to be addressed to study adversarial training of RL agents along with our proposed attacks: (1) The adversary and model need to be jointly updated. How do we balance these two updates, and make sure the adversary is well-trained at each point in training? (2) How to avoid cycles in the training process due to the agent overfitting to the current adversary? (3) How to ensure the adversary doesn't overly prevent exploration/balance unperturbed vs. robust performance?

A APPENDIX:

A.1 MORE ILLUSTRATION ON FIGURE 3 The meaning of Fig 3 is to show how the accuracy of the learned models affects our proposed technique:

1. we first learned 3 models with 3 different number of samples: 5e5, 1e6, 5e6 and we found that with more training samples (e.g. 5e6, equivalently 5000 episodes), we are able to learn a more accurate model than the one with 5e5 training samples;

2. we plot the attack results of total loss for our technique with 3 learned models (denoted as PGD, num train) as well as the baselines (randU, randB, Flip) on 10 different runs (initializations).

We show with the more accurate learned model (5e6 training samples), we are able to achieve a stronger attack (the total losses are at the order of 50-200 over 10 different runs) than the less accurate learned model (e.g. 5e5 training samples). However, even with a less accurate learned model, the total losses are on the order of 400-700, which already outperforms the best baselines by a margin of 1.3-2 times. This result in Fig 3 also suggest that a very accurate model isn't necessarily needed in our proposed method to achieve effective attack. Of course, if the learned model is more accurate, then we are able to degrade agent's performance even more. For the baselines (rand-U and rand-B), the adversary generates 1000 trajectories with random noise directly and we report the best loss/reward at the end of each episode. The detailed steps are listed below:

Step 1: The perturbations are generated from a uniform distribution or a bernoulli distribution within the range [-eps, eps] for each trajectory, and we record the total reward and total loss for each trajectory from the true environment (the MuJoCo simulator)

Step 2: Take the best (lowest) total reward/loss among 1000 trajectories and report in Table 1 and 2 .

We note that here we assume the baseline adversary has an "unfair advantage" since they have access to the true reward (and then take the best attack result among 1000 trials), whereas our techniques do not have access to this information. Without this advantage, the baseline adversaries (rand-B, rand-U) may be weaker if they use their learned model to find the best attack sequence. In any case, Table 1 and 2 demonstrate that our proposed attack can successfully uncover vulnerabilities of deep RL agents while the baselines cannot.

For the baseline flip, we add the perturbation (with the opposite sign and magnitude ) on the original state/action and project the perturbed state/action are within its limits.

A.3 SCORE OF LEARNED POLICY WITHOUT ATTACKS:

We use default total timesteps = 1000, and the maximum total reward is 1000. We report the total reward of the d4pg agents used in this paper below. The agents are well-trained and have total reward close to 1000, which outperforms agents trained by other learning algorithms on the same tasks (e.g. DDPG, A3C in Sec 6 (Tassa et al., 2018) ; PPO in Sec 5 (Abdolmaleki et al., 2018) ), and thus the agents in this paper can be regarded as state-of-the-art RL agents for these continuous control domain tasks. The attack results in 

:

Alternatively, standard optimal control methods such as Linear Quadratic Regulator (LQR) and iterative Linear Quadratic Regulator (i-LQR) can also be applied to solve Equations 3 and 4 approximately.

It is only a qualitative comparison in the sample complexity regimes since the applications are not the same. This is in-line with the theoretical perspective that model-based approaches are expected to be more sample-efficient than the model-free counterparts

Figures and Tables:

(a) Attack observations of agent. (b) Attack actions of agent.

Figure 1: Two commonly-used threat models.

Figure 2: Video frames of best attacks in each baseline among 10 runs for the Walker.walk example.Only our proposed attack can constantly make the Walker fall down (since we are minimizing its head height to be zero).

Figure 3: Compare sample size on the Walker.stand in 10 different initialization in the environment. The x-axis is the kth initialization and the y-axis is the total loss of corresponding initialization.

. We demonstrate results on 4 different environments in Mu-Input: pre-trained policy π, MaxSampleSize n s , environment env, trainable parameters W 2: Output: learned dynamical model f (s, a; W ) 3: S agent ← Collect trajectories(π, n s , env) 4: S random ← Collect trajectories(random policy, n s , env) 5: f (s, a; W ) ← supervised learning algorithm(S agent ∪ S random , W ) 6: Return f (s, a; W ) Input: pre-trained policy π, learned dynamical model f (s, a; W ), threat model, maximum perturbation magnitude , unroll length T , apply perturbation length n (≤ T ) 2: Output: a sequence of perturbation δ 1 , . . . , δ n 3: if threat model is observation manipulation (Eq. 1) then

Table content: <table><tr><td>Algorithm 2 learn dynamics</td></tr><tr><td>1Algorithm 3 model based attack</td></tr><tr><td>1: 4:</td></tr></table>

Compare three model-free attack baselines (rand-U, rand-B, flip) and our algorithm (Ours) in 4 different domains and tasks. We report the following statistics over 10 different runs: mean, standard deviation, averaged ratio, and best attack (number of times having smallest loss over 10 different runs). Results show that our attack outperforms all the model-free attack baselines for the observation manipulation threat model by a large margin for all the statistics. Our proposed attack is also superior on the action manipulation threat model and win over most of the evaluation metrics.

Table content: <table><tr><td/><td colspan="6">(a) Observation manipulation: mean and standard deviation (in parenthesis)</td></tr><tr><td/><td colspan="2">Total loss</td><td>rand-U</td><td>rand-B</td><td>flip</td><td>Ours</td></tr><tr><td colspan="2">Walker</td><td>stand</td><td>1462 (70)</td><td>1126 (86)</td><td colspan="2">1458 (24) 258 (55)</td></tr><tr><td/><td/><td>walk</td><td>1517 (22)</td><td>1231 (31)</td><td colspan="2">1601 (18) 466 (42)</td></tr><tr><td colspan="2">Humanoid</td><td>stand</td><td>1986 (28)</td><td colspan="3">1808 (189) 1997 (5)</td><td>516 (318)</td></tr><tr><td/><td/><td>walk</td><td>1935 (22)</td><td>1921 (31)</td><td colspan="2">1982 (9)</td><td>1457 (146)</td></tr><tr><td colspan="2">Cartpole</td><td colspan="5">balance 4000 (0.02) 3999 (0.04) 3989 (2)</td><td>2101 (64)</td></tr><tr><td/><td/><td colspan="2">swingup 3530 (1)</td><td>3525 (1)</td><td colspan="2">3516 (1)</td><td>2032 (172)</td></tr><tr><td>Walker</td><td>stand</td><td>0.18</td><td/><td>0.23</td><td>0.18</td><td>Ours: 10/10, others: 0/10</td></tr><tr><td/><td>walk</td><td>0.31</td><td/><td>0.38</td><td>0.29</td><td>Ours: 10/10, others: 0/10</td></tr><tr><td>Humanoid</td><td>stand</td><td>0.26</td><td/><td>0.29</td><td>0.26</td><td>Ours: 10/10, others: 0/10</td></tr><tr><td/><td>walk</td><td>0.75</td><td/><td>0.76</td><td>0.74</td><td>Ours: 10/10, others: 0/10</td></tr><tr><td>Cartpole</td><td>balance</td><td>0.53</td><td/><td>0.53</td><td>0.53</td><td>Ours: 10/10, others: 0/10</td></tr><tr><td/><td>swingup</td><td>0.58</td><td/><td>0.58</td><td>0.58</td><td>Ours: 10/10, others: 0/10</td></tr><tr><td/><td colspan="6">(c) Action manipulation: mean and standard deviation (in parenthesis)</td></tr><tr><td/><td colspan="2">Total loss</td><td>rand-U</td><td>rand-B</td><td>flip</td><td>Ours</td></tr><tr><td colspan="2">Cartpole</td><td colspan="5">balance 4000 (0.03) 3999 (0.08) 3046 (1005) 1917 (102)</td></tr><tr><td/><td colspan="2">swingup</td><td>3571 (1)</td><td>3487 (7)</td><td colspan="2">1433 (4)</td><td>1388 (50)</td></tr><tr><td/><td>Fish</td><td>upright</td><td>935 (27)</td><td>936 (24)</td><td colspan="2">907 (22)</td><td>824 (84)</td></tr><tr><td/><td/><td colspan="5">(d) Action manipulation: averaged ratio and rank-1</td></tr><tr><td colspan="6">Total loss (avg ratio) Ours/rand-U Ours/rand-B Ours/flip</td><td>best attack</td></tr><tr><td>Cartpole</td><td>balance</td><td>0.48</td><td/><td>0.48</td><td>0.63</td><td>Ours: 10/10, others: 0/10</td></tr><tr><td/><td>swingup</td><td>0.39</td><td/><td>0.40</td><td>0.97</td><td>Ours: 10/10, others: 0/10</td></tr><tr><td>Fish</td><td>upright</td><td>0.88</td><td/><td>0.88</td><td>0.91</td><td>Ours: 8/10, flip: 2/10</td></tr></table>

Compare three attack baselines (rand-U, rand-B, flip) and our algorithm (Ours) in three different domains and tasks. Performance statistics of 10 different runs are reported.

Table content: <table><tr><td/><td colspan="6">(a) The mean and standard deviation (in parenthesis) over 10 different runs</td></tr><tr><td/><td colspan="2">Total reward</td><td>rand-U</td><td>rand-B</td><td>flip</td><td>Ours</td></tr><tr><td/><td>Walker</td><td>stand walk</td><td>937 (41) 941 (23)</td><td>744 (48) 796 (21)</td><td>993 (8) 981 (9)</td><td>235 (38) 225 (50)</td></tr><tr><td/><td>Humanoid</td><td>stand walk</td><td>927 (21) 934 (22)</td><td>809 (85) 913 (21)</td><td>959 (5) 966 (6)</td><td>193 (114) 608 (66)</td></tr><tr><td/><td>Cartpole</td><td colspan="5">balance 995 (0.17) 986 (0.16) 985 (3) swingup 873 (0.75) 851 (2) 852 (0.29) 353 (61) 385 (6)</td></tr><tr><td colspan="7">(b) Average ratio and number of times our algorithm being the best attack over 10 runs.</td></tr><tr><td colspan="6">Total reward (avg ratio) Ours/rand-U Ours/rand-B Ours/flip</td><td>best attack</td></tr><tr><td>Walker</td><td>stand walk</td><td>0.25 0.24</td><td/><td>0.32 0.28</td><td>0.24 0.23</td><td>Ours: 10/10, others: 0/10 Ours: 10/10, others: 0/10</td></tr><tr><td>Humanoid</td><td>stand walk</td><td>0.21 0.65</td><td/><td>0.24 0.67</td><td>0.20 0.63</td><td>Ours: 10/10, others: 0/10 Ours: 10/10, others: 0/10</td></tr><tr><td>Cartpole</td><td>balance swingup</td><td>0.39 0.41</td><td/><td>0.39 0.42</td><td>0.39 0.42</td><td>Ours: 10/10, others: 0/10 Ours: 10/10, others: 0/10</td></tr></table>

Ablation study on the planning length T . Compare 3 attack baselines (rand-U, rand-B, flip) and our algorithm (Ours) and report performance statistics of 10 different runs.

Table content: <table><tr><td/><td colspan="5">(a) domain: Walker, task: walk (observation perturbation)</td></tr><tr><td>Walker.walk</td><td/><td>Total loss</td><td/><td/><td>Total reward</td></tr><tr><td/><td colspan="5">mean std med min max mean std med min max</td></tr><tr><td>Ours, T = 1</td><td colspan="3">468 79 489 286 567</td><td colspan="2">222 45 227 135 300</td></tr><tr><td>Ours, T = 2</td><td colspan="3">604 31 611 535 643</td><td colspan="2">353 51 362 253 441</td></tr><tr><td>Ours, T = 5</td><td colspan="3">761 65 771 617 837</td><td colspan="2">483 60 496 348 540</td></tr><tr><td>Ours, T = 10</td><td colspan="3">881 68 886 753 975</td><td colspan="2">568 48 579 469 623</td></tr><tr><td>Ours, T = 15</td><td colspan="3">874 93 891 723 1002</td><td colspan="2">583 58 604 483 647</td></tr><tr><td>Ours, T = 20</td><td colspan="3">937 62 950 804 993</td><td colspan="2">634 41 638 559 687</td></tr><tr><td>rand-U</td><td colspan="3">1517 22 1522 1461 1542</td><td colspan="2">941 23 945 885 965</td></tr><tr><td>rand-B</td><td colspan="3">1231 31 1234 1189 1272</td><td colspan="2">796 21 796 766 824</td></tr><tr><td>flip</td><td colspan="3">1601 18 1604 1562 1619</td><td>981</td><td>9 984 961 991</td></tr><tr><td/><td colspan="5">(b) domain: Walker, task: stand (observation perturbation)</td></tr><tr><td>Walker.stand</td><td/><td>Total loss</td><td/><td/><td>Total reward</td></tr><tr><td/><td colspan="5">mean std med min max mean std med min max</td></tr><tr><td>Ours, T = 1</td><td colspan="3">322 84 319 202 453</td><td colspan="2">257 67 265 163 366</td></tr><tr><td>Ours, T = 2</td><td colspan="3">279 55 264 223 391</td><td colspan="2">246 40 232 200 322</td></tr><tr><td>Ours, T = 5</td><td colspan="2">163 53 154</td><td>93 246</td><td colspan="2">193 27 188 154 238</td></tr><tr><td>Ours, T = 10</td><td>84 46</td><td>67</td><td>42 165</td><td colspan="2">153 24 142 132 194</td></tr><tr><td>Ours, T = 15</td><td>101 40</td><td>82</td><td>57 157</td><td colspan="2">164 23 152 143 201</td></tr><tr><td>Ours, T = 20</td><td>117 41</td><td>98</td><td>68 193</td><td colspan="2">170 21 161 149 207</td></tr><tr><td>rand-U</td><td colspan="3">1462 70 1454 1341 1561</td><td colspan="2">938 41 932 866 999</td></tr><tr><td>rand-B</td><td colspan="3">1126 86 1130 973 1244</td><td colspan="2">744 48 744 664 809</td></tr><tr><td>flip</td><td colspan="3">1458 24 1451 1428 1501</td><td>993</td><td>8 997 979 999</td></tr></table>

Less frequency attack. Report statistics of 10 different runs with different initial states in the walker domain with task stand.

Table content: <table><tr><td/><td>Total loss</td><td/><td>Total reward</td></tr><tr><td/><td colspan="3">mean std med min max mean std med min max</td></tr><tr><td>Ours</td><td>934 152 886 769 1187</td><td colspan="2">648 95 622 559 799</td></tr><tr><td colspan="2">rand-U 1511 35 1502 1468 1558</td><td colspan="2">970 20 964 947 999</td></tr><tr><td colspan="2">rand-B 1431 77 1430 1282 1541</td><td colspan="2">924 41 923 840 981</td></tr><tr><td>flip</td><td>1532 15 1537 1496 1546</td><td>996</td><td>5 999 984 1000</td></tr><tr><td colspan="4">(III) Walker stand. The learned model is slightly more accurate than the (II) (test MSE is 0.089) in</td></tr><tr><td colspan="2">this task. Interestingly, Table</td><td/><td/></tr></table>

Table 1 and 2 in our manuscript are hence suggested to be representative.

Table content: <table><tr><td>Domain</td><td>Task</td><td>Total reward</td></tr><tr><td>Walker</td><td>stand</td><td>994</td></tr><tr><td>Walker</td><td>walk</td><td>987</td></tr><tr><td>Humanoid</td><td>stand</td><td>972</td></tr><tr><td>Humanoid</td><td>walk</td><td>967</td></tr><tr><td>Cartpole</td><td>balance</td><td>1000</td></tr><tr><td colspan="2">Cartpole swingup</td><td>883</td></tr><tr><td>Fish</td><td>upright</td><td>962</td></tr><tr><td colspan="3">A.4 ADDITIONAL EXPERIMENTS ON ABLATION STUDY</td></tr></table>

Cartpole balance (action perturbation)

Table content: <table><tr><td>Total loss</td><td>mean</td><td>std med min max</td></tr><tr><td>Ours, T = 20</td><td>2173</td><td>51 2189 2087 2239</td></tr><tr><td colspan="3">Ours, T = 100 1951 113 1924 1851 2192</td></tr><tr><td>rand-U</td><td>4000</td><td>0 4000 4000 4000</td></tr><tr><td>rand-B</td><td>3999</td><td>0 3999 3999 3999</td></tr><tr><td>flip</td><td colspan="2">3046 1005 3074 2060 3999</td></tr></table>

Peer review (iclr2020):
*Synopsis*: This paper looks at a new framework for adversarial attacks on deep reinforcement learning agents under continuous action spaces. They propose a model based approach which adds noise to either the observation or actions of the agent to push the agent to predefined target states. They then report results against several model-free/unlearned baselines on MuJoCo tasks using a policy learned through D4PG. Main contributions: - Adversarial attacks for Deep RL in continuous action spaces. *Review* The paper is well written, and has some interesting discussion/insight into attacking deep RL agents in continuous actions spaces. I think the authors are headed in the right direction, but compared to prior work in adversarial attacks for deep RL agents (i.e. the Huang and Lin) I have a few concerns that I feel the authors need to better explain/motivate in their paper. I am recommending this paper be rejected based on the following concerns. I am willing to raise my score if some of these are addressed by the authors in subsequent revisions 1. This algorithm requires the pre-trained policy to plan attacks (which may be a high bar for such an adversarial attack). It would be a nice addition to include similar results with *black-box* adversarial attacks, as mentioned in the Huang. 2. Another issue, addressed in the Lin paper, is this attack seems to require perturbation on every time step in a proposed trajectory. As mentioned by Lin, this is probably unrealistic and would cause the attacker to be detected. It would be another nice contribution to include variants that don*t require perturbations on each transition. 3. Another unfortunate requirement is a learned model (or a way to simulate trajectories). From the Model Based RL literature, we know learning such a model is quite difficult and often unrealistic given our current approaches. While this is problematic, I think the paper could systematically test this looking at what happens as the model becomes less accurate over time. This could provide some nice results showing an accurate model isn*t necessarily needed and anneal concerns over having to learn such a model. 4. It is unclear if the baselines measured against are meaningful in this setting, and I*m also a bit unclear how they are generated/implemented. Specifically, the random trajectories require you to return the generated trajectory with the smallest loss/reward. It is unclear how the adversary knows this information. Is it known through a model or some other simulation? Also the flip baseline could use a bit more explanation. I think these details can be safely placed in the appendix, but should appear somewhere in the final version. 5. I*m not sure the comparison to sample efficiency to the Gleave or Uesato papers are meaningful. For Gleave, the threat model explored is much different where they do not have access to the agent*s observation or action streams and instead learn policies to affect the other agent in game scenarios. This is very different. Also, the Uesato is not adversarially attacking the agent, but attempting to find failure cases for the agent, which I again feel is very different from what you are trying to accomplish. I would remove this discussion and the claim at the end of the conclusion. Other suggestions: S1. It would be helpful to include the score of the learned policy without any attacks, to see how well the baselines are performing (this will help readers understand if these are reasonable/meaningful baselines). S2. I*m unclear what figure three is adding to the paper, and am actually uncertain what the y-axis means. I don*t think this is a wise use of the 9th page, and this plot could probably be relegated to the appendix. S3. As in prior work, it would be useful to see how well this line of attack works for multiple learning algorithms. Some potential candidates could be: PPO, TRPO, SAC, etc...