Learning Key Steps to Attack Deep Reinforcement Learning Agents
Authors: Chien-Min Yu
Hsuan-Tien Lin
Venue:  (2020)
Keywords: deep reinforcement learning
adversarial attacks

Abstract: Deep reinforcement learning agents are known to be vulnerable to adversarial attacks. In particular, recent studies have shown that attacking a few key steps is effective for decreasing the agent's cumulative reward. However, all existing attacking methods find those key steps with human-designed heuristics, and it is not clear how more effective key steps can be identified. This paper introduces a novel reinforcement learning framework that learns more effective key steps through interacting with the agent. The proposed framework does not require any human heuristics nor knowledge, and can be flexibly coupled with any white-box or black-box adversarial attack scenarios. Experiments on benchmark Atari games across different scenarios demonstrate that the proposed framework is superior to existing methods for identifying more effective key steps.

Under review as a conference paper at ICLR 2020
LEARNING KEY STEPS TO ATTACK DEEP REINFORCEMENT LEARNING AGENTS
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep reinforcement learning agents are known to be vulnerable to adversarial at- tacks. In particular, recent studies have shown that attacking a few key steps is effective for decreasing the agent's cumulative reward. However, all existing at- tacking methods find those key steps with human-designed heuristics, and it is not clear how more effective key steps can be identified. This paper introduces novel reinforcement learning framework that learns key steps through interacting with the agent. The proposed framework does not require any human heuristics nor knowledge, and can be flexibly coupled with any white-box or black-box adver- sarial attack scenarios. Experiments on benchmark Atari games across different scenarios demonstrate that the proposed framework is comparable and could be superior existing methods for learning key steps.
INTRODUCTION
Reinforcement learning (RL) is a framework for sequential decision problems, where an agent in- teracts with an unknown environment and tries to maximize the total reward receives. With the rapid development of deep learning, RL agents parametrized by neural networks, usually referred to as deep RL agents, are able learn complex policies from raw inputs (Mnih al. 2015). Recently, deep RL agents have shown great success across various domains, such as achieving superhuman performance on games (Mnih et al., 2015, Silver et al., 2016; 2018), completing complex robotic tasks (Levine et al. 2016),, optimizing patient treatments (Escandell-
et al. 2014 Raghu cet al. 2017), and developing autonomous driving skills (Pan et al. 2017 Isele et al. 2018).
However, if we wish to deploy these RL agents into security applications, must take their reliability into consideration. It is discovered that neural network classifiers may make errors on deliberately crafted inputs, known as adversarial examples (Biggio al.,, 2013) (Szegdyy al. 2013) Behzadan & Munir 2017a). Deep RL is no exception: Many works have demonstrated that deep RL agents are also vulnerable when attacked by adversarial examples (Behzadan & Munir 2017a Huang et al.] 2017), raising serious concerns on the reliability of these agents for security-critical applications. For example, there may be severe consequences if RL-agent-driven autonomous
vehicle is compromised by adversarial examples fed from a malicious attacker.
Given the sequential nature of RL, not necessary to attack at every time step to degrade the agent's 's performance significantly (Lin et al. 2017. Kos & Song 2017). The reason is that not all decisions made by the agent are equally important. Some decisions may critical the agent, such as those for long-term planning or immediate reward gathering; some other decisions may not have much effect on the environment nor the rewards, and thus attacking those steps would not affects the agent's 's performance. The critical steps will be named key steps in this work, and attacking these steps decreases the cumulative reward that the agent could gather.
Figure SigureDillustnn iillustrates key step and non-key step in Atari Pong. where the main difference between them is whether the ball has already passed through the agent. If the attacker is aware of this attacking the key steps. By attacking fewer steps, the
difference, the attacker can concentrate attacker reduces the costs of generating, injecting, and hiding the adversarial examples. Therefore, an intelligent attacker should prefer attacking key steps over non-key steps.
A natural question arises: How does an attacker identify the key steps to attack? As an initial attempt to address this question, [in et al. (2017) formulate the problem from optimization perspective,

---

Under review as a conference paper at ICLR 2020
(a) Key step
(b) Non-key step
Figure I: An example of the key step and non-key step in Atari Pong. (a) The ball moving in the direction indicated by the red arrow. If the agent does move down this step or some earlier steps, the ball would pass through, making the agent lose a point. (b) The ball has already passed through the agent. Any decision this step does save the situation and does not affect the environment much
and propose heuristic that attacks the agent when it strongly prefers one action. Concurrently, Kos & Song (2017) propose another heuristic that attacks when the action appears rewarding to the agent. However, since these heuristics are designed based human knowledge, it is unclear whether there exist more effective key steps. In this work, we study the possibilities of finding more effective key steps, and of finding them automatically without human knowledge. We confirm both possibilities by introducing an RI. framework where the attacker learns the key steps from scratch through interacting with the agent.
In particular, we study the key-step identificatinn problem proposed by Lin et al. (2017). We first make crucial observation that this constrained optimization problem can be converted into another form that matches the objective of RL. Then, we design corresponding RL environment to train the attacker. Our contributions are summarized as follows:
We formulate the key-step identification problem in an unconstrained form, and introduce an RL framework that solves it directly without incorporating human knowledge. The proposed RL framework independent of how adversarial examples are generated, and thus could be combined with white-box or black-box attacks.
We justify learning framework by giving toy example where those human-designe heuristics fail to identify the key steps.
To test the effectiveness of our framework, we conduct experiments on four benchmark Atari games. The results suggest that the attacker trained by our framework learns effective key steps and has the potential to outperform prior methods.
BACKGROUND AND RELATED WORK
2. REINFORCEMENT LEARNING BACKGROUND
We begin with an introduction to the Markov Decision Process (MDP) (Sutton & Barto 2018). For set Se S, we (SS) to denote the set of probability distributions over S. Consider agent interacting with an environment: At each step f, the agent picks an action a, based on current state 8t, and then the environment returns reward r; and next state 81+1- We formalize this interaction
MDP
M, which is a 5-tuple (S, A. P, R, 7). where $ is a finite set of states, A. is a finite set of actions, P: S A (SS) defines the transition dynamics from 8, 8t+1, R: S A ARR) defines the reward distribution, and 7 E [0, 1] is discount factor. For convenience, we use r:: (St, az) + E,, R(r..(sr. to denote the mean reward function. We specify an agent by its (stochastic) policy S + A(A), and define its Q-value function as
("t, 04) =
Σ
2 A (s ac+b)
k=0
2

---

Under review as a conference paper at ICLR 2020
where 8, P( Q4-1) and a; (T;;)) for all t. The agent's goal is to find a policy that maximizes the expected cumulative reward, or expected return, defined by Q" x(s1)) starting from an initial state $1. For finite-horizonte MDPs, use T to denote the maximum number of steps, and define an episode to be a sample of (81, 01; /1, 8T; 0r. PT, 8T+1).
To learn the optimal poliey, deep Q-network (DQN, Mnih et al. 2013) first approximates the Q-value function by a neural network that takes a state as input and outputs the Q-values for all actions. The policy is then induced by picking the action with the largest Q-value each state. Given transition step (St, az, Tt, 8141), 1), the network fo is trained by minimizing the square loss
Lo (1r+ 11++7mmn max fo(st+1)a) fo(s))a))
where we use So(s)a denote the a-th entry of the Q-values fs()..
2.2 ADVERSARIAL ATTACK BACKGROUND
In the context of adversarial attacks, an attacker tries craft adversarial examples that the target model would misclassify. In gradient-baae based adversarial attacks, the attacker generates adversarial examples by adding small perturbations to data examples, such that the perturbations are impercep- tible to human eyes. Take the fast gradient sign method (FGSM) by [Goodfellow et al. (2014) for example. Given a target model fo parametrized by and an image-label pair (æ, y). the FGSM computes the perturbation as
= e (.,,(0, (0,,,),,
where is scaling factor that controls the norm of the perturbation, and [(, 2; y) is objec- tive function that depends on the attacker's goal. In untargetted attacks, the attacker aims to min- imize the target model's classification accuracy. Thus, the objective J(0, x. y) is substituted with DKI. [ fz))), the Kullback-Leibler divergence between the label's one-hot encoding e(v) and the predicted probabilities fa(x). However, this type of white-box attacks requires full knowledge of the network parameter 0, which may be hidden in practical tasks.
In contrast, black-box attacks assume no access to and study other conditions, such as the ability to query model outputs (Chen al.
2017). One black-box attack of particular interest the substitute model approach (Papernot et al. 2016a 2017), where the attacker computes perturbation using substitute model that is trained to perform the same task as the target model. This approach is based on the finding that adversarial examples are transferable; that is, if an adversarial example fools model, it usually fools another models trained to perform the same task, regardless of the network architecture (Szegedy et al., 2013). Consequently, once the attacker gains access to the training set, block-box attacks reduce white-box attacks with lower attack success rate.
2.3 ADVERSARIAL ATTACK ON DEEP RL AGENTS
As opposed to traditional adversarial attacks, attacking an RL agent is a unique and challenging task. When we attack classifier, we typically aim to minimize the classification accuracy the probability that the classifier predicts some given class. When we attack an RL. agent, however, usually do not care about the individual actions that the agent picks. In this paper, we choose to minimize the agent's cumulative reward (Huang et al. 2017 Lin et al. (011)) as the attacker's goal. Other possible goals include luring the agent into designated state (Lin et 2017), or misguiding the agent to optimize an adversarial reward (Tretschk et al.) 20118).
maximize
Previous works have investigated the effects of white-box and black-box attacks against RL agents on the Atari benchmark. Given full knowledge of the target agent's 's network parameters, an attacker can craft adversarial examples to fool the target agent into picking wrong actions (Behzadan & Munir [017r] Huang et al., 2017 Kos & Song 2017 Mandlekar et al., ( et al.] 2018] Hussenot et al. 2019) using the FGSM. Other white-box methods such as the Jacobian saliency map algorithm (Papernot al. 2016b) and the attack proposed by Carlini & Wagner (2017) are also effective against RL agents (Behzadan & Munir 2017a| Lin et al. 2017). Even the attacker cannot access the target agent's network parameters, adversarial examples can still be crafted with a substitute agent that is trained in the same environment as the target agent (Behzadan & Munir 2017al Huang al. 2017) similar the substitute model approach.
3

---

Under review as a conference paper at ICLR 2020
Another line of studies tries improve the RL agent's robustness with adversarial attacks. Several works improve the agent's robustness to visual perturbations through adversarial training (Kos & Song, 2017 Mandlekar et al. 2017| Behzadan & Munir 2017b] Pattanaik et al. 2018), which is defense technique that adds adversarial examples into training sets (Goodfellow et al. 2014). On the other hand, Pinto al. (0177) propose to improve the agent's robustess to environment changes by training the agent and an adversary in an alternating procedure, where the attacks are operated on the environment dynamics. Tessler et al. (2019) study the agent's robustness to action-space perturbations in continuous control domains, and achieve better performance even in the absence of the adversary.
LEARNING KEY STEPS TO ATTACK
3.1 THE KEY-STEP IDENTIFICATION PROBLEM
Suppose that an attacker would like to attack target agent with policy и by adding perturbations to the states. We use 1) to denote an arbitrary attack method that maps from state to perturbation. Let the budget, the maximum number of attacks permitted in one episode. The key-step identification problem (Lin et al. 2017) can be formulated as follows:
min by
Σ rss at)
a, I(s b.n(s:)), S211 P(s 41 ST
Σ S.S.
In this problem. the attacker aims to minimize the expected return of target agent under the budget constraint. The binary variable by indicates whether the perturbation m7())) is added to the state St, and is key step found by the attacker =.
Problem (1) is difficult combinatorial problem with exponentially large search space. Each de- cision the attacker changes the subsequent steps. Moreover, the transition dynamics and reward distribution may be random. Even if the attacker manages to search the whole space by brute force, the collected episodes are merely samples. The attacker still needs to estimate the expected return.
Due to these difficulties, previous works (Lin al. 2017; Kos & Song 2017) design simple heuris- ties to find the key steps. Suppose that the target agent is parametrized by a Q-network fo _ Kos & Song (2017) observe that steps with large Q-value are usually followed by large immediate reward, and hypothesize that an attack is effective at steps with large Q-value. They propose to set b, = the maximum Q-value max, fa(sz)a is larger than given threshold. On the other hand, Lin al. (017)) propose to add perturbations when the agent is confident about its action. They use the soli- max function convert the Q-values into a probability distribution (let softmax o fe), and set be = 1 if the probability gap max, (St)a min, T (5:)). )a is larger than given threshold.
3.2
AN RL FRAMEWORK FOR LEARNING THE KEY STEPS
To address these challenges, we design an RL framework that directly solves the key-step identifi- cation problem. Based on the Lagrange relaxation technique (Bertsekas) 1997). we first replace the hard budget constraint with soft penalty term:
min
E
Σ
Sts a+)
be
by by. (0.11)
(2)
I(s ben(s:)),
St1 P(St. VI ST
where À is a parameter that controls the penalty for each attack. Now Problem becomes sequen- tial decision problem without additional constraints. We then propose to solve this problem through
These heuristics be applicd attack other value-hased posecy-based agents in similar way.

---

Under review as a conference paper at ICLR 2020
state
b, (0,1)
environment
attack policy
action
reward
adversarial attack algorithm
attacker
possibly perturbed
perturbation
state
target agent
Figure 2: Interactions among the environment, the target agent, and the attacker. the attack policy outputs "1", the attacker would intercept the state from the environment and inject the perturbed state to the target agent. Different from prior methods, our method parametrizes the attack policy by neural network and trains it by RI. framework.
RL by training an attack policy that identifies the key steps. Figure illustrates the interactions among the environment, the target agent, and the attacker.
Here we describe our proposed RI. framework in details. Suppose that the target agent with policy trained to maximize the expected return in MDP M (S, A, P, 7) We propose to train the attack policy x' in another MDP M (S' A P R', 7/), where each component is defined as follows:
S' S, and 7/ 7,.
The attacker's action space A' {0,
For all E S', E A', the transition dynamics ('(s, = P(s, a), and the reward (²(s,)) = R(s, a) bλ, where a ~ ((s + by(s)) is the target agent's 's action.
=
The new environment M has the same state space M, but reduces to binary action space with the action "I" representing an attack at that step and "0" otherwise. The attacker's reward is the negative of the target agent's reward, plus penalty of for each attack. Therefore, maximizing the expected return in M' is equivalent to minimizing the objective in Problem (
Our proposed RL framework is general. We make no assumptions on the environment on the target agent. In addition, our RL framework can be freely paired with any RL algorithm and any attack method when training the attack policy. Thus, both white-box and black-box attack scenarios can be considered in this framework.
3.3 THE POTENTIAL OF LEARNING THE KEY STEPS
Here we justify our RL framework by giving toy example where the prior methods fail to identify the key steps. Consider simple MDP shown in Figure Suppose that we would like attack an agent with its policy and Q-value function given in the figure. If we set the budget constraint B = 2, the most effective attack policy should attack steps and However, both heuristics are unable to find these two key steps.
This toy example highlights the potential drawback of these heuristics. Since the attacker might change the agent's policy in the future steps, the agent's Q-value estimates in the current step could be inaccurate. As a result, heuristics based on these Q-value estimates may not find the most effective key steps. In contrast, framework trains the attacker from scratch. We do not rely on human- designed heuristics, and thus the learned attacker may be able to identify the most effective key steps in this example.
5

---

Under review as a conference paper at ICLR 2020
S1
agent
0.81
step
heuristes
optimal atack
S2
Sa
0.9
0.891
step
St
S$
So
stop
Se
S12
S
0.99
Figure 3: An MDP example where the two heuristics fail. The MDP components described as follows. A circle represents a state, and double circle represents a terminal state. At each state, the agent choose to go left or right, shown by the lines. Red numbers under the terminal states represent the reward an agent gets if the agent reaches that state. The discount factor y is set to 0.9. Suppose we would like attack agent with its estimated Q-values and policy shown by the blue numbers and blue arrows. Let the budget constraint B = 2. To minimize the agent's cumulative reward, smart attacker should attack steps and 3, guiding the agent into state $15. However, the heuristic proposed by Kos & Song (2017) would attack at steps 2 and 3 since the maximum Q-value larger at those steps. On the other hand, the heuristic proposed by Lin al. (207)) also attacks steps 2 and since the gap of Q-values is larger at those steps (so the probability gap is larger too). As result, both heuristics guide the agent into state 811. failing to find the best key steps.
B D
DOOO
Figure 4: An example of perturbation generated by the FGSM with 0.01. Left: Original frame. Middle: Perturbed frame. Right: Added perturbation (rescaled to [0, 1)).
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate the proposed RL framework on four Atari 2600 games in the Arcade Learning Envi- ronment (Bellemare al., 2013) (Pong, Space Invaders, Seaquest, and Riverraid), which covers variety of environments. The environment setting and preprocessing match the guidelines suggested by Machado al. (2018). In particular, use a sticky-action probability of 0.25 in these environ- ments. For the target agent, we take pretrained DQN agents from Dopamine (Castro et al.) 2018), which have the same network architecture Mnih et al. (2015). and fix them thereafter. We train the attack policy using DQN with this same architecture. It is worth mentioning that we train each attack policy for 10M frames, which is only 1/20 of the frames used train the target agents. Other hyperparameters reported in Appendix (Table
Throughout the experiments, we use the untargetted FGSM to generate perturbations due to its com- putation efficiency. Although we only test our framework in this setting, expect the framework to generalize to the case of targetted attack, or to other adversarial attack algorithms. We use Fool- box (Rauber et al., 2017) to generate the adversarial examples, with the norm constraint set to 0.01. Figure 4 shows an example of the perturbation, which imperceptible to human and can only be visualized after rescaled. To make the scenario more realistic, follow Hussenot al. (0019) and

---

Under review as a conference paper at ICLR 2020
260
bo arge-pret
redders
Ierpovnd
renden
5042
1539
1540
7503
-10
-12
2513
27 150
0.8
1.0
(a) Pong
(b) Space Invaders
(c) Seaquest
(d) Riverraid
Figure 5: Key-step identification with white-box attacks. The vertical axis is the target agent's undiscounted return, and the horizontal axis is the attack ratio (the number of attacks divided by the number of steps in an episode). Under the same attack ratio, the lower the target agent's undis- counted return is, the better the attack method does. All reported scores are averaged over ten testing episodes and shown with one standard deviation. The results of the two heuristics are obtained by setting ten different thresholds on their corresponding criteria. For our method, we train five attack policies using penalty parameters E {10', 10" 10-1, 10-2, 10-3), and plot the score for each learned policy, with testing episodes shown by red dots.
2039
AXCS
gerviis
2000
argezsila
1539
12559
30009
-5
0000
530
505
2503
-00
04
31
0.0
1.0
50
5.
3.
50
(a) Pong
(b) Space Invaders
(c) Seaquest
(d) Riverraid
Figure 6: step identification with black-box attacks. See Figure for descriptions.
perturb the target agent's 's observation, rather than the whole state. That is, although the target agent stack the latest four frames as its current state, the attacker is only allowed to add perturbation to the latest frame.
4.2 PERFORMANCE COMPARISON OF THE ATTACK POLICY
We compare the attack policy trained by our RL framework, to random baseline that attacks uni- formly at random, and to two heuristics, large-value (Kos & Song 2017) and large-prob-gay (Lin) et al. 2017). Figure 5 plots the results in the white-box setting, where the attacker has full knowl- edge of the target agent's 's network parameters. The attack policy learned in our RI. framework shows comparable performance to human-designed heuristics across different attack ratios consis- tently, and achieves superior performance in Pong and Riverraid. These results suggest that the key steps are learnable, and that it is possible to identify more effective key steps using our method.
We also test our framework with black-box attacks, where the attacker cannot access the target agent's 's network parameters. We adopt the substitute model approach, and take the substitute agent from Dopamine, which is a DQN agent trained with a different random seed. The substitute agent is used to compute the perturbations and the statistics needed in the two heuristics. The results are shown in Figure 6 Compared to white-box attacks, black-box attacks are of less success rate, and thus the effect of the attacks is weakened (cf. Figure 5). Still, our method is able to perform comparably to (in Space Invaders, Seaquest, Riverraid) or outperform (in Pong) previous methods.
4.3 THE EFFECT THE PENALTY PARAMETER
To investigate the penalty parameter X, we summarize the performance of the learned attack policies with white-box attacks in T Results obtained with black-box attacks can be found in Appendix (Table 3) As À decreases, our RL framework tends to learn an attack policy with higher attacker ratio. Training with too large values of would restrict the attack policy from launching any attacks. Therefore, choosing an appropriate À is important for learning effective key steps. Empirically,
7

---

Under review as a conference paper at ICLR 2020
Comparison of different with white-box attacks. episodes with the standard deviation shown in parentheses.
Table 1:
The results are averaged over
10'
10"
10-1
10-2
10
return
16.3
(2.1)
16.3
-20.5
(0.7)
-20.6
(1.2)
-20.9
Pong
attack ratio (%)
0.0
(0.0)
0.0 (0.0)
5.3
(1.0)
9.7
(4.7)
15.8 (3.1)
Space Invaders
return
1693.0 (820.0)
714.0
(4387)
201.5
(152.5)
139.5
(124.6)
254.0
(147.4)
attack ratio (%)
0.0 (0.0)
26.0 (49)
50.0 (9.4)
71.9 (5.3)
77.1 (8.1)
Seaquest
1288.0 (483.0)
414.0
(183.3)
84.0 (72.6)
122.0
(94.0)
54.0
(34.7)
attack ratio (%)
0.0 (0.0)
21.8 (4.6)
39.8 (7.7)
61.1
(5.2)
47.7
(5.2)
Riverraid
888.0 (636.1)
407.0
(127.3)
546.0
(113.6)
616.0
(328.7)
464.0
(295.9)
attack ratio (%)
27.8 (5.8)
35.3 (7.0)
(5.0)
92.2
(5.6)
82.5
(2.6)
114
stack
300
000
0.5
1.3
35
2.
0.0
0.55
6.30
115
021
90
05
10
35
24
25
61
03
85
(a) Pong with white-box attacks
(b) Pong with black-box attacks
Figure 7: Behavior comparison of our method (with X 1002 2) previous heuristics. Left: His- togram of the maximum Q-value computed by large-value. Right: Histogram of the probability gap computed by purge-prob-gap. (all: steps in one episode; attack: steps that our method attacks)
observe that setting À around the same order of magnitude as
Potential reward loss Original return of target agent Number of steps per episode
Minimum return of environment
would produce stable results (see Table 4 and Figure g in Appendix). This observation could guide the choice of À in different environments.
4.4 BEHAVIOR COMPARISON THE ATTACK POLICY
To understand to what extent the attack policy learned in our RL. framework is different from the heuristics, we attack the target agent by the learned attack policy, and compute the statistics used by the two heuristics at each step. Figure 7 plots the histogram of those statistics in one episode of Pong. We can see that the key steps found by our method spread across different intervals, rather than focusing the largest intervals. This result suggests that our method does not simply mimic the heuristics, but indeed learns unique key steps to attack. The results of other environments are similar and can be found in Appendix (Figure 9.
plots
DISCUSSION
We show that the key-step identification problem can be solved by training an attack policy through RL. Compared to existing works, our method learns different key steps without any human knowl- edge. Results on Atari benchmarks validate our belief that the proposed method may learn more effective key steps. This raises safety concerns on the real-world applications of deep RL agents. One possible future direction is to study how we can improve the RL agent's 's robustness to this kind of key-step attacks. For example, combining the idea of alternating training (Pinto et al,, 2017 Tessler et al. 2019) into our RL framework could be fruitful research direction. Since the RL agent's 's robustness is mostly studied in continuous environments (e.g.. MuJoCo), we hope that this work attracts more attention into studying this topic in environments with discrete actions.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In International Conference on Machine Learning and Data Mining Pattern Recogni- tion, Pp. 262-275. Springer, 2017a.
Vahid Behzadan and Arslan Munir. Whatever does not kill deep reinforcement learning, makes it stronger. arXiv preprint arXiv:1712.09344, 2017b.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):
334 334, 1997.
Battista Biggio, Igino Corona, Davide Maiorea, Blaine Nelson, Nedim Šmdiić Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. In Joint European conference on machine learning and knowledge discovery databuses, pp. 387-402. Springer, 2013.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Belle- mare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint
arXiv:1812.06// 10, 2018.
Yu Chen, Huan Zhang, Yash Sharma. Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, pp. 15-26. ACM, 2017.
Pablo Fscandell-Monteroo, Milena Chermisi, Jose M Martinez-Martinez, Juan Gomez-Sanchis, Carlo Barbieri, Emilio Soria-Olivas, Flavio Mari, Joan Vila-Francés, Andrea Stopper, Emanuele Gatti, et al. Optimization of anemia treatment in hemodialysis patients via reinforcement learning. Artificial intelligence in medicine, 62(1):47-66 2014.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 41X1v:1412.6572, 2014.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv: 1 702.02284, 2017.
Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Targeted attacks on deep reinforcement learning agents through adversarial observations. arXiv preprint arXiv: 12282, 2019.
David Isele, Reza Rahimi, Akansel Cosgun, Kaushik Subramanian, and Kikuo Fujimura. Navigating occluded intersections with autonomous vehicles using deep reinforcement learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2034-2039. IEEE, 2018.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint arXiv:1705.06452 2017.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17()):133-1333, 2016.
Yen-Chen Lin, Zhang-Wer Wei Hong. Yuan- Liao, Meng-Li Shih, Ming-Yu Yu Liu, and Min Sun. Tac- tics of adversarial attack on deep reinforcement learning agents. arXiv preprint 17.3::700.0001 2017.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.

---

Under review as a conference paper at ICLR 2020
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of poysically-plausibe perturbations. In 2017 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 1932-3939 IEEE, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Xinlei Pan, Yurong You, Ziyan Wang, and Cewu Lu. Virtual to real reinforcement learning for autonomous driving. arXiv preprint arXiv: 170400320. 2017.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv: 100v::160,.07277, 2016a.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Sympo- sium on Security and Privacy (EuroS&P), pp. 372-387. IEEE, 2016b.
Nicolas Papernot, Patrick McDaniel, lan Goodfellow, Somesh Jha, 7. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506-519. ACM, 2017.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2040-2042. International Foun- dation for Autonomous Agents and Multiagent Systems, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2817-2826. JMLR. org. 2017.
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint 2017.
1 1.006002,
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. arXiv preprint arXiv:1707.0413/ 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre. Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140- 1144, 2018
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint drX1v:1312.6199, 2013.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control. arXiv preprini arXiv:1901.091 19011.9144, 2019.
Edgar Tretschk, Seong Joon Oh, and Mario Fritz. Sequential attacks on agents for long-tem term adver- sarial goals, arXiv preprint 1805. 2018.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Table 2: Hyperparameters used during training the attack policy in our RL framework.
Hyperparameter
Value
optimizer
Adam
learning rate
0.0001
batch size
32
discount factor
0.99
training steps
10000000
target network update frequency
1000
replay buffer size
100000
learning start step
20000
learning frequency exploration type
4
e-greedy
epsilon decay type
linear decay
exploration decay horizon
250000
minimum epsilon during training
0.01
epsilon during testing
0.001
adversarial attack method
FGSM
maximum norm constraint for perturbation
0.01
perturbation searching intervals
100
Table 3: Comparison of different with black-box attacks. episodes with the standard deviation shown in parentheses.
The results are averaged over ten
A
10
10"
10
10
10
return
16.3
(2.1)
14.7
(2.2)
-17.2
(2.5)
-20.2
(1.0)
-19.7
(1.6)
Pong
attack ratio (%)
0.0
(0.0)
0.1
3.1
(0.4)
12.4
(3.5)
31.5
Space
return
787.0
(457.6)
571.0
(294.4)
539.5
(309.2)
422.5
(267.9)
269.5
(183.6)
Invaders
attack ratio (%)
6.6
(1.2)
9.3
(2.8)
46.6
(4.7)
73.9
(7.1)
57.7
(6.3)
Seaquest
return
1122.0
(375.0)
1038.0
(463.8)
1252.0
(524.2)
1056.0
(340.)
978.0
(310.2)
attack ratio (%)
0.6
(0.3)
2.3
(1.3)
39.0
46.6
(5.2)
47.2
(4.3)
Riverraid
return
9493.0 (3442.7)
9891.0
(2584.1)
7069.0
(2488.5)
9353.0
(3229.2)
11176.0
(2942.4)
attack ratio (%)
68.0
(4.6)
42.8
(4.0)
95.7
(1.9)
56.2
(3.1)
59.9
(3.5)
Table 4: Statistics of the target agents averaged over ten episodes.
Original return of target agent
Minumum return of environment
Number steps per episode
Potential reward loss
Pong Space Invaders
18.4
-21.0
2498.9
1.58 x 10-2
1063.0
0.0
1196.5
8.88 x 10-1
Seaquest
1280.0
0.0
1531.5
8.36 x 10-1
Riverraid
12912.0
0.0
1772.6
7.28 x 10
11

---

Under review as a conference paper at ICLR 2020
(a) Pong with white-box attacks
Wo www.uuooammn
(b) Pong with black-box attacks
www.
M
-200
(c) Space Invaders with white- box attacks
H MWlarmp
-00
(d) Space Invaders with black- -box attacks
(c) Seaquest with white-hox attacks
2000
2007
(l) Seaquest with black-box attacks
2000
1000
(g) Riverraid with white-box attacks
(h) Riverraid with black-box attacks
Figure 8: Attack policy's undiscounted return in MDP M' during training (A 10 100 10-1 10-2 10-3 from left to right). The attack policy is tested every 100000 steps and report scores averaged over ten testing episodes.
12

---

Under review as a conference paper at ICLR 2020
(a) Space Invaders with white-hox attacks
(b) Space Invaders with black-hox attacks
100
600
0.22
6.0
oos
000
003
634
ais
(c) Seaquest with white-hox attacks
(d) Seaquest with black-hox attacks
2.0
200
Riverrii with white-hox attacks
(f) Riverraid with black-box attacks
Figure 9: Behavior comparison of our method (with X = 10-3 previous heuristics. Left: His- togram of the maximum Q-value computed by large-value. Right: Histogram of the probability gap computed by large-prob-gon gap. (all: all steps in one episode; attack: steps that our method attacks)
13


Reviews:
[{"review_id": "SJgW8qeTYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nLearning Key Steps to Attack Deep Reinforcement Learning Agents\n\nThis paper proposes an extension of existing discrete action image space adversarial attack algorithms.\nInstead of choosing the steps by heuristic, the authors propose to choose the key steps by augmenting the reward function with a penalty to decrease the ratio of attacks.\n\nI tend to vote rejection for this paper, given that the proposed algorithms seem incremental compared to the existing algorithms, and the experiments seem not sufficient enough to support the core claim proposed in the paper.\n\nPros:\n- The paper is well written, with sufficient background and related work section for the paper to be self-contained.\n- The proposed framework is an interesting and practical framework for attacking RL agents.\n\nCons:\n- The experiment section is insufficient.\nMore specifically:\n1) Results from Figure 5 and Figure 6 seem to disagree with what the authors claim in the paper. In many (the majority) of the environments, the proposed algorithm has only trivial improvement and even worse performance under the same attack rate.\n\n2) It is not very convincing when only one result sample is plotted in Figure 5 and Figure 6.\nI think it is necessary to show the performance of the proposed algorithm under different attack rate.\nA wide range of candidate penalty parameter lambda should be tested, so that a curve can be fitted for the proposed algorithm similar to the baselines (similar to the one shown in Table 1, but with much more test values).\n\n3) Related to 2), it seems the Lagrange relaxation makes it hard to control the attack rate in the proposed algorithms. How sensitive it is to control the attack rate?\n\n3) Can the authors elaborate on why the algorithms is not too sensitive to the value of penalty in section 4.5?\nTable 1, where the performance is almost the same for different penalty parameter, does not necessarily show that the algorithms is not too sensitive to the choice of the penalty parameter.\nAs mentioned by the author, -21 is the minimum reward (or random reward) an agent can get from Pong.\n\n\nIn general, given the current status of the paper, where there is a lot of room for improvement of experiment section, I will vote for a rejection.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJehO3OJ9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to learn the ‘key-steps’ at which to to apply an adversarial attack on a reinforcement learning agent. The framing of this problem is a Lagrangian relaxation of a constrained minimization problem which takes the form of an RL problem itself, where the attacking agent’s reward is the negative reward of the target agents plus a penalty (lambda, hyper-parameter) for choosing to attack. The attacking agent’s action space is binary, attack or no attack.\n\nThe RL approach is compared with a random attack policy and two heuristic methods for attacking agents in games on the Atari benchmark.\n\nThe setting addressed in this work, where the attacker only learns whether/when to attack or not is a greatly simplified version of the full problem. It is (in my opinion, but feel free to correct me) likely that the type of perturbation also being learned has even greater potential, but is (obviously) much harder.\n\nAdditionally, although the authors mention this as future work, I think the co-training setting is particularly interesting. As the authors suggest, this could lead to more robust target agents, but additionally I wonder if the type and difficulty of attacks would vary as the target agent trains.\n\nBesides the (picky) complaints, I think the problem formulation is quite reasonable. Again, I find the larger problem extremely interesting, but perhaps this is far too intractable right now. The formulation is straightforward, so while I recognize it as a contribution, it is not a particularly large one.\n\nOn the other hand, the experimental results appear somewhat weak to me. \n\nIf you vary the lambda parameter (as in Table 1, but for all games) you should be able to get similar line plots in Figure 5&6 for the RL approach, which would give a much better comparison for the trade-offs as you get for the heuristic methods. I think this comparison would be very interesting and strengthen the existing results in those figures.\n\nThe results, currently, do not appear very significant because (1) the gap between the RL solution and the heuristics is very small and (2) these *appear* to be single runs without standard deviations displayed. Can you argue for why these results *are* in fact significant (statistically or otherwise)?\n\nAnother question raised by the results is how performance of the attacking policy varies with training. The authors point out that the training is quite small compared to the target policy, but is that because it has already found the best solution it can in that time? How would it improve with more training?\n\nSmall note on Figure 7, I think the point for these would be better made by normalizing the respective histograms.\n\nUpdate:\n\nThank you for your responses and updating with new results. I think these provide a much better picture of the performance of the RL-based system. Although I am revising my score upward, I still think this is generally a rejection. Obviously I still think the full problem is interesting, but even the key step identification problem would be publishable if the performance was improved or further analysis helped me understand why RL is not doing better (since the heuristics are after all just heuristics). Good luck on future versions of this work.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkeJyj8wtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contribution:\n\nThis work presents a method for performing budgeted attacks on a RL agent where an attacker can apply a perturbation to a limited subset of the observations of the said agent.\nTo select the \"key steps\" that needs to be corrupted, they train an opponent RL agent, with a reward crafted such that it should minimize the number of attacks as well as the cumulated reward of the target agent.\nTheir attack works either in a white-box setting, using FGSM, or in a black-box setting, using a substitute model.\n\nThey then proceed to compare to to prior work that use heuristics based on the policy values to select the steps to attack.\n\nThe paper is overall well written and easy to follow.\n\nReview:\n\n\nOne major limitation of the work is that the attack rate is not readily modifiable. It uses a penalty term in the reward function with a tunable weight $\\lambda$, but changing this weight seemingly requires retraining the opponent from scratch, which is unpractical. By constrast, note that the previous attack methods allow to change the attack rate at will.\nMore importantly, there is no clear relationship between the value of lambda and the attack rate. With the same lambda, depending on the game, the opponent settles on attack rates varying between 10% (on pong) and 70% (on Riverraid).\n\n\nThe results themselves show only marginal improvement over the baselines, and in the absence of clear error bars / confidence intervals, it is difficult to state the significance of the method. In the particular case of Space Invaders (figure 5b), the proposed method seems tied to the best heuristic (at attack-rate 70%), but the said heuristic reaches the same performance level for much lower attack-rate (as low as 40-50%), implying that the presented method did not do a good job at minimizing the number of attacks.\nOverall, a rigorous way of comparing the methods need to be devised. Maybe something like an AUC with proper confidence bounds could do the trick?\n\n\nIn equation (2), you present a target objective function designed using Lagrange relaxation. However, the RL algorithm uses decay (\\gamma = 0.99), which means that the resulting function that is effectively minimized is different. Could you clarify the impact of the decay on the lagrange-relaxed objective function?\n\n\nCould you clarify a bit the section 4.3 on black-box attacks? It seems that you are using a substitute model attack, but it's not clear to me how the substitute is obtained. Is it the same model? How is it trained? Is the attack robust to differences in the algorithm?\n\nFinally, I'm a bit surprised by the choice of DQN as the base algorithm, especially since the chosen framework (Dopamine), offers significantly stronger algorithms (Rainbow or IQN). DQN doesn't even reach perfect score on Pong, which means that the raw policy itself is a bit brittle, since it looses 5 points. Did you try to apply this method on more robust policies?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers adversarial attacks in deep reinforcement learning, and specifically focuses on the problem of identifying key steps to attack. The paper poses learning these key steps as an RL problem with a cost for the attacker choosing to attack.

The reviewers agreed that this was an interesting problem setup, and the ability to learn these attacks without heuristics is promising. The main concern, which was felt was not adequately addressed in the rebuttals, was that the results need to be more than just competitive with heuristic approaches.

The fact that the attack ratio cannot be reliably changed, even with varying $\lambda$ still presents a major hurdle in the evaluation of the proposed method.

For the aforementioned reasons, I recommend rejecting this paper.
Decision: Reject