Bounded Myopic Adversaries for Deep Reinforcement Learning Agents
Authors: Ezgi Korkmaz
Henrik Sandberg
Gyorgy Dan
Venue:  (2021)
Keywords: deep reinforcement learning
adversarial

Abstract: Adversarial attacks against deep neural networks have been widely studied. Adversarial examples for deep reinforcement learning (DeepRL) have significant security implications, due to the deployment of these algorithms in many application domains. In this work we formalize an optimal myopic adversary for deep reinforcement learning agents. Our adversary attempts to find a bounded perturbation of the state which minimizes the value of the action taken by the agent. We show with experiments in various games in the Atari environment that our attack formulation achieves significantly larger impact as compared to the current  state-of-the-art. Furthermore, this enables us to lower the bounds by several orders of magnitude on the perturbation needed to efficiently achieve significant impacts on DeepRL agents.

Under review as a conference paper at ICLR 2021
BOUNDED MYOPIC ADVERSARIES FOR DEEP REINFORCEMENT LEARNING AGENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial attacks against deep neural networks have been widely studied. Ad- versarial examples for deep reinforcement learning (DeepRL) have significant security implications, due to the deployment of these algorithms in many applica- tion domains. In this work we formalize an optimal myopic adversary for deep reinforcement learning agents. Our adversary attempts to find a bounded pertur- bation of the state which minimizes the value of the action taken by the agent. We show with experiments in various games in the Atari environment that our attack formulation achieves significantly larger impact as compared to the current state-of-the-art. Furthermore, this enables us to lower the bounds by several orders of magnitude on the perturbation needed efficiently achieve significant impacts on DeepRL agents.
INTRODUCTION
Deep Neural Networks (DNN) have become a powerful tool and currently DNNs are widely used in
speech recognition (Hannun et al. 2014), computer vision (Krizhevsky et al. 2012), natural language processing (Sutskever et al. 2014), and self learning systems as deep reinforcement learning agents
(Mnih et al. (2015), Mnih al. (2016), Schulman al. (2015). Lillicrap al. (2015)).
Along with the overwhelming success of DNNs in various domains there has also been a line of research investigating their weaknesses. Szegedy et al. a (2014) observed that adding imperceptible perturbations to images can lead DNN to misclassify the input image. The authors argue that the existence of these so called adversarial examples is a form of overfitting. In particular, they hypothesize that very complicated neural network behaves well on the training set, but nonetheless, performs poorly on the testing set enabling exploitation by the attacker. However, they discovered different DNN models were misclassifying the same adversarial examples and assigning them the same class instead of making random mistakes. This led Goodfellow et al. (2015) to propose that the DNN models were actually learning approximately linear functions resulting underfitting the data.
Recent work by Mnih et al. (2015) introduced the use of DNNs as function approximators in reinforcement learning, improving the state of the art in this area. Because these deep reinforcement learning agents utilize DNNs, they are also susceptible to this type of adversarial examples. Currently, deep reinforcement learning has been applied to many areas such as network system control (Jay et al. 2019)) Chu et al. (2020). Chinchali et al. (2018). financial trading Noonan (2017), blockchain protocol security Hou al. (2019), grid operation and security (Duan al. (2019). Huang al. (2019), cloud computing Chen et al. (2018), robotics (Gu et al. (2017). Kalashnikov et al. (2018), autonomous driving Dosovitsky et al. (2017), and medical treatment and diagnosis Tseng al. (2017), Popova et al. (2018), Thananjeyae eyaan et al. (017)) Daochang & Jiang (2018). Ghesu et al. (2017). A more particular scenario where adversarial perturbations might be significant interest is financial trading market where the DeepRL. agent is trained on observations consisting of the order book. In such a setting it is possible to compromise the whole trading system with an extremely small subset of adversaries. In particular, the -norm bounded perturbations dicussed in our paper have sparse solutions, and thus can be used as a basis for an attack in such a scenario. Moreover, the magnitude of the E norr bounded perturbations produced by our attack is orders of magnitude smaller than previous approaches, and thus our proposed perturbations result in a stealth attack more likely to evade automatic anomaly detection schemes.

---

Under review as a conference paper at ICLR 2021
Considering the wide spectrum of deep reinforcement learning algorithm deployment it is crucial to investigate the resilience of these algorithms before they are used in real world application first step to understand these algorithms and build
domains. Morcover, adversarial formulations generalizable, reliable and robust deep reinforcement learning agents. Therefore, in this paper we study adversarial attack formulations for deep reinforcement learning agents and make the following contributions:
We define the optimal myopic adversary. whose aim is to minimize the value of the action taken by the agent in each state, and formulate the optimization problem that this adversary seeks to solve.
We introduce differentiable approximation for the optimal myopic adversarial formulation.
We compare the impact results of attack formulation previous formulations in different games in the Atari environment.
We show that the new formulation finds a better direction for the adversarial perturbation and increases the attack impact for bounded perturbations. (Conversely, our formulation decreases the magnitude of the pertubation required to efficiently achieve a significant impact.)
2 RELATED WORK AND BACKGROUND
2.1 ADVERSARIAL REINFORCEMENT LEARNING
Adversarial reinforcement learning is an active line of research directed towards discovering the weaknesses of deep reinforcement learning algorithms. Gleave et al. (2020) model the interaction between the agent and the adversary as a two player Markov game and solve the reinforcement learning problem for the adversary via Proximal Policy Optimization introduced by |Schulman et al. (2017). They fix the victim agent's policy and only allow the adversary to take natural actions to disrupt the agent instead of using l -orm bound pixel perturbations. Pinto et al. (2017) model the adversary and the victim as a two player zero-sum discounted Markov game and train the victim in the presence of the adversary make the victim more robust. Mandlekar al. (2017) use gradient based perturbation to make the agent more robust as compared to random perturbations. HHuang et al. (2017) and Kos & Song (2017) use the fast gradient sign method (FGSM) show deep reinforcement learning agents are vulnarable to adversarial perturbations. Pattanaik et al. (2018) use a gradient based formulation to increase the robustness of deep reinforcement learning agents.
2.2 ADVERSARIAL ATTACK METHODS
Goodfellow et al. (2055) introduced the fast gradient method (FGM)
V V_J(Z,y) JI,
x² c
for crafting adversarial examples for image classification by taking the gradient of the cost function J (z, y) used to train the neural network in the direction of the input. Here x is the input and y is the output label for image classification. As mentioned in the previous section FGM was first adapted to the deep reinforcement learning setting by Huang et al. (2017). Subsequently, Pattanaik et al. (2018) introduced a variant of FGM, which few random samples are taken in the gradient direction, and the best is chosen. However, the main difference between the approach of Huang et al. (2017) and Pattanaik et al.] (2018) is in the choice of the cost function J used to determine the gradient direction. In the next section will outline the different cost functions used in these two different formulations.
2.3 ADVERSARIAL ATTACK FORMULATIONS
In bounded attack formulation for deep reinforcement learning, the aim is to try to find a perturbed state Sadv in ball
2

---

Under review as a conference paper at ICLR 2021
De.p(s) = (aadv
|[ - s|pp S c).
that minimizes the expected cumulative reward of the agent. It is important to note that the agent will always try to take the best action depending only on its perception of the state, and independent from the unperturbed state. Therefore, in the perturbed state the agent will still choose the action, a" (Sadv) = arg max, (sadv)))) a), which maximizes the state action value function in state aadv-
It has been active line of research to find the right direction for the adversarial perturbation in the deep reinforcement learning domain. The first attack of this form was formulated by Huang et al. (2017) and concurrently by Kos & Song (017) by trying minimize the probability of the best possible action in the given state,
a" as(s) = arg max Q(s,
min E E..
I (8ndvs a (s)).
Note that ITss a) is the softmax policy of the agent given by
(().a(a))
TT a)
(4)
Σ
where is called the temperature constant. When the temperature constant is not relevant to the discussion will drop the subseript and use the notation (s, a). It is important to note that r(s. a) is not the actual policy used by the agent. Indeed the DRL agent deterministically chooses the action a" maximizing Q(s, a) (or equivalently a)). The softmax operation is only introduced in order to calculate the adversarial perturbation direction. [in et al. (2017) formulated another attack with unbounded perturbation based on the Carlini & Wagner (2017) attack where the goal is minimize the perturbation subject to choosing any action other than the best.
min Sa E (s.(
|| sllp
subject
a" (s) + a" (Saav)).
Pattanaik et al, (2018) formulated yet another attack which aims to maximize the probability of the worst possible action in the given state,
ane (5) arg min Q(s, a)
(5)
max Suby E (s))(s)
TI (SSadv an (s)),
and further showed that their attack formulation
5
effective than
Pattanaik et al. 2018) also introduce the notion of targeted attacks to the reinforcement learning domain. In their paper they take the cross entropy loss between the optimal policy in the given state and their adversarial probability distribution and try to increase the probability of aw- However, just trying to increase the probability of aw the softmax policy i.e. (Sadv, a.,) is not sufficient target aw in the actual policy followed by the agent. In fact the agent can end up in state where,
IT (ndv, aw) > T(S, aw:) au + arg max (Sudr: a).
(6)
Although (T a.))) has increased, the action an will not taken. However, might be still possible to find perturbed state <'yy for which,
3

---

Under review as a conference paper at ICLR 2021
T (uuv; aw) > (8 Ow) Ow = arg max πT (8 a).
Therefore, maximizing the probability of taking the worst possible action in the given state not actually the right formulation to find the correct direction for adversarial perturbation.
3 OPTIMAL MYOPIC ADVERSARIAL FORMULATION
To address the problem described in Section 2.3 we define an optimal myopic adversary to be an adversary which aims minimize the value of the action taken by the agent myopically for each state. The value of the action chosen by the agent in the unperturbed state is Q a a*(s)) and the value (measured in the unperturbed state) of the action chosen by the agent under the influence the adversarial observation is (s, a" (Saav)). The difference between these is the actual impact of the attack. Therefore, in each state s the optimal myopic adversary must solve the following optimization problem
arg max [Q(s, a (s)) Q(s, a" (sadv) {)]. (s)
(8)
By we may rewrite ( in terms the softmax policies t (1,, a),
arg max [I((s, a (*)) TT,, (8aav))) Sa D. (s))
(9)
Since (Ts, a" (s)) does not depend on Sadv. is equivalent to solving
min
(ss m a)})-
(10)
a
3.1 EFFICIENT APPROXIMATION OF OPTIMAL MYOPIC ADVERSARIAL FORMULATION
Having an arg max operator in the cost function is unpleasant, since it is non-differentiable. Instead can approximate arg max by decreasing the temperature T for गe (Sutse a),
lim TT.. (Sadv, a) 1 max mx (x) (a). Tat =0
(11)
The intuition for (11 is that, by ( decrease the temperature the value of e(Q(s,a")TTa) for the action a" which maximizes Q(s, will dominate the other actions. Thus π (s, arg max, (Tr(Sav a)}) can be approximated by,
(T,, arg max max//T(xxssssss/// a)})
Σ (s, larg (ax
(12)
lim
Σ T(s
Tian Tha (Sadv,
(13)
Tel +0
Therefore, our original optimization problem can be expressed as,
min 788, arg (sx a)}) (e)
min
lim
Σ (s,
Tha (Sadvs a). (14)
DD, (e) Tal +0
In practice we will not decreasing Taay to as this would be equal to applying the non-differentiable argmax operation. Instead will replace the arg max operation with the approximation TI. (8, a)
4

---

Under review as a conference paper at ICLR 2021
21073
384002
20
Figure| 1: Games used in the experiments from Atari Arcade Environment. Games from left to right in order: Roadrunner, Riverraid, Bankkeist, Seaquest, Amidar, Beamrider, Pong and UpandDown.
for small value of adv. In general it is not guarenteed that the minimum of the cost function in (14) with this approximation will be close to the minimum of the original cost function with the arg max operation. To see why, note that this approximation is equivalent to first switching the limit and minimum (14) to obtain,
lim Tab + (5)
min
ITss, () TT, (Sadv; a),
(15)
and second replacing the limit with small value of Tad. There are two possible issues with this approach. First of all, due to the non-convexity of the minimand, exchanging the limit and minimum may not yield an equality between (14) and (15). Secondly, even if this exchange is legitimate, is not clear how to choose sufficiently small value for Tad in order obtain a good approximation to (15). However, we show in our experiments that using this approximation in the cost function gives state-of-the-art results.
There is another caveat which applies to all myopic adversaries including those in prior work. The Q-value is good estimate of the discounted future rewards of the agent, only assuming that the agent continues to take the action maximizing the Q-value in future states. Since myopic attacks are applied in each state, this may make minimizing the Q-value non-optimal attack strategy when the future is taken into account. There is also always the risk that the agent's 's Q-value is miscalibrated. However, our experiments show that our myopic formulation performs well despite these potential limitations.
3.2 EXPERIMENTAL SETUP
In our experiments we averaged over 10 episodes for each Atari game (Bellemare et al. 2013) in Figure [ from the Open AI gym environment (Brockman et al., 2016). Agents are trained with Double DQN (Wang et al.) 2016). We compared the attack impact of our Myopic formulation with the previous formulations of Huang et al. (207)) and Pattanaik et al. (018). The attack impact has been normalized by comparing
unattacked agent, which chooses the action corresponding the
maximum state-action value in each state, with an agent that chooses the action corresponding to the minimum state-action value in each state. Formally, let Rmax be the average return for the agent who always chooses the best action in given state, let Roin be the average return for the agent who always chooses the worst possible action in given state, and let R2 be the average return of the agent under attack. We define the impact I, as
Rmax
Ra
(16)
R
max
Rmnn
This normalization was chosen because we observed that in Atari environments agents can still collect stochastic rewards even when choosing the worst possible action in each state until the game ends. See more details of the setup Appendix A.2
3.3 ADVERSARIAL TEMPERATURE
Based on the discussion in Section 3 we expect that Tady decreases the function in (14) becomes a better approximation of the arg max function. Thus this results a higher attack impact as Tadv decreases. However, after certain threshold the function in (1) becomes too close the arg max
5

---

Under review as a conference paper at ICLR 2021
Tenpperoture Constrt meeess
"srppenaour Commert Incedts
0.4
1.55
6.0
04-
8.3
0.5
5.41
140
17a
DPan
20
Figure 2: Attack impacts temperature constant for different games 42-nrrm bound with 10-8 Left: Riverraid. Middle: Roadrunner. Right: Seaquest.
Table 1: Left: Attack impacts for three attack formulations with & bound and 10-8 Right:
Attack impacts for three attack formulations with & norm bound and 10-8
Games
Huang
Pattanaik
Myopic
Games
Huang
Pattanaik
Myopic
Amidar
0.048-10.11 19
0.514:10.29
0.932-10.02
Amidur
0.039=0.22
0.502-10.32
0.941
Bankheist
0.111++0.14
0.373++.115
0.624++0.08
Bankheist
0.014++0.007
0.0411=0.12
0.596+0.06
Beumrider
0.083=0033
0.455=0.27
0.663+10.13
Beamrider
-0.052:+0.55
0.374+0.13
0.581
Riverraid
0.079:=0.19
0.379:0.27
0.589:1:0.12
Riverrid
0.062=0.29
0.369::0.21
0.549:10.1
RoadRunner
0.1874:0.16
0.387:10.16
0.557::0.11
RoadRunner
0.153=00112
0.453:10.15
0.482:0.07
Pong
0.0:10.016
0.067:10.06
0.920:10.08
Seaquest
0.386=0.25
0.468:0.233
0.57610.11
Seaquest
0.305-1-0.26
0.524+00.23
0.697-+0.16
Pong
0.000=0.02
0.043++0.03
0.916+0.009
UpNDown
0.074.0.3
0.4761:0.16
0. 0.865::0.66
UpNDown
-0.007:10.28
0.43610.28
0.892+0.07
Impads
Bound va Impact
Frund
18
k3t
-20
Figure 3: Attack impact vs logarithm base 10 of fn-norm bound. Left: Amidar. Middle: Pong. Right: UpNDown.
function which is non-differentiable. Therefore, in practice, beyond this threshold the quality of the solutions given by the gradient based optimization will decrease, and the attack impact will be lower. Indeed this can be observed in Figure 2 In our experiments we chose Tadv to maximize the impact by grid search.
3.4 IMPACT COMPARISON FOR & -ORM BOUNDED PERTURBATIONS
Table shows the mean and standard deviation of the impact values under E and fy-norm bounds. The tables show that the proposed attack results in higher mean impact for all games and under all norms, and results in lower standard deviation almost always. In particular, this an indication that our myopic attack formulation achieves higher impact more consistently than the previous formulations. More results on -orm bound can be found in Section A.T of the Appendix.
Figure 3 shows the attack impact function of the perturbation bound for each formulation in three games. As e decreases, our myopic formulation exhibits higher impact relative to the other formulations. Recall that Goodfellow et al. (2015) argue that small adversarial perturbations shift the input across approximately linear decision boundaries learned by neural networks. Therefore. having a higher impact with smaller norm bound is evidence that myopic formulation finds better direction (i.e. one that points more directly at such a decision boundary) to search for the perturbation.

---

Under review as a conference paper at ICLR 2021
Aueoage vine Arosstiliy of Sanaed Artione
Average Errprrical Prodabiity Forntd Actiono
Averag Emsrical Probabilty of Ranted Actions
33000
fe de 2008s
ayantty det Aww.Quaaec
30110
33075
3000
030
(a) Pong
(b) BankHeist
(c) UpNDown
Avaraga Empliial hicbetlity of Nenkad Actions
Average Emspiicst Probaliiity by Rankeed Actions
Anoruge trrprikal Prubabilitty of Fanrbodd Atlient
1.114
1312-
1210 Negt 2000
0.00
0.00
0.00
30
(d) RiverRaid
(e) Amidar
(I) BeamRider
Figure 4: Up: Average empirical probabilities of ranked actions for three different formulations. Left: Bankheist. Middle: Pong. Right: UpNDown. Down: Expected probability of ranked actions of three different formulations. Left: Riverraid. Middle: Amidar. Right: BeamRider.
3.5 DISTRIBUTION ON ACTIONS TAKEN
In order to understand the superior performance of the proposed adversarial formulation it is worth- while to analyze the distribution of the actions taken by the agents under attack. For each formulation, we recorded the empirical probability p(s, ox) that the attacked agent chooses the g ranked action The ranking of the actions is according to their value in the unperturbed state. Without perturba- tion agents would always take their first ranked action, any deviation from this a consequence of the adversarial perturbations. We average these values over 10 episodes to obtain the average empirical probability that the t action taken Ee-p (p(s, as)). Here P is the distribution on the episodes e of the game induced by the stochastic nature of the Atari games. We plot the results in Figure
The legends in Figure 4 show the average empirical probabilities of taking the best action a". can be seen that in general EE-p Ip(s, a²)] is lower and Ee~plp(s,, aw))] is higher for our Myopic formulation when compared to other attack formulations. We realized that the formulations which have higher impact at the end of the game might still be choosing the best action more often than the ones which have lower attack impact as shown in the legend of Figure 4 for the Pong game. In this case both Huang al. (017)) and|Pattaaaii et al. (2018) cause the agent choose the best action a less frequently than our formulation, but still end up with lower impact. If we look at Figure it can be seen that Ee-p[p(s [p(s, a. a²rh }] is much higher for both Huang et al. (017)) and Pattanaik et al. 2018 compared to our Myopic formulation. However, the key to achieving a greater impact is to cause the agent to choose the lowest ranked actions more frequently. As can be seen from Figure 4 our Myopic formulation does this more successfully. More detailed results can be found in Appendix
A.5 A.5
3.6 STATE-ACTION VALUES OVER TIME
In this section we investigate state-action values of the agents over time without an attack, under attack with the Pattanaik et al. (018)) formulation, and under attack with our myopic formulation.
In Table 2 it is interesting to observe that under attack the mean of the state-action values over the episodes might be higher despite the average return being lower. One might think that an attack with greater impact might visit lower valued states on average. However, in our experiments we have found that the gap between Ee-p!Q(s, (s))] and Ee-p (ss, a" (Saav) relative to the gap
7

---

Under review as a conference paper at ICLR 2021
Table 2: Average Q-values for the best, worst, the adversarial actions and impacts, loss in the Q-values caused by the adversarial influence, and impacts over the episodes.
B[Q(s, a (x)) B[Q(ss (SADV)
E[Q(s, ow]l
Qss
IMPACT
RIVERRAID
UNATTACKED
9.724
8.911
0
0 -0.079
HUANG ET. AL
9.656
9.653
8.8577
0.0039
PATTANAIK ET. AL.
10.092
10.072
9.341
0.0258
0.379
MYOPIC
10.013
9.977
9.283
0.0495
0.589
SEAQUEST
UNATTACKED
9.724
8.911
0
0 0.305
HUANG ET. AL
10.6787
10.6787
10.100
0.0043
PATTANAIK ET. AL.
10.092
10.072
9.341
0.0267
0.524
MYOPIC
10.013
9.977
9.283
0.0506
0.697
BANKHEIST
UNATTACKED
6.6863
5.9273
0
0 0.111
HUANG AL
6.602
6.5998
5.907
0.0039
PATTANAIK ET.
6.0517
6.0403
5.604
0.025
0.373
MYOPIC
5.935
5.9153
5.5520
0.517
0.624
AMIDAR
UNATTACKED
0.1188
-0.1487
0
0 0.048
HUANG ET. AL
0.115
0.114
-0.147
0.0049
PATTANAIK ET. AL..
0.139
0.138
-0.163
0.0466
0.514
MYOPIC
0.2239
0.1953
-0.2564
0.059
0.932
between Ee<p (Qss, a" (s))] and Ee<p (Q(s, aw)] is much more significant factor in determining the magnitude of the impact. Therefore, we define the quantity
EemplQ a" (8)) PemplQ((> a (8ate)
Q loss
(17)
E Ee ((s, a" (s)) Ee aw)l
Observe in Table the magnitudes Eeep [Q(s, a" (s))), Eenp awr), and Re (((ss,a"("sa)" (s a"
not strictly correlated to the impacts for each formulation. However, Qloss is always higher for our Myopic attack compared to the previous formulations. Recall that in Equation (3), our original
optimization problem designed to maximize the gap between Q(s, a* (s)) and Q(, a" (sav)) in each state. The results Table 2 are evidence both that we achieve this original goal and that solving initial optimization problem for each state leads to lower average return the end of the game. More results on this matter can be found Appendix AA.4.
The mean Q-value per episode under attack increases in some games. We believe that this may be given state is only an accurate representation of the expected
due to the fact that the Q-value rewards of an unattacked agent. When the agent is under attack, there might be states with high Q-value which are dangerous for the attacked agent (e.g. states where the attack causes the agent to immediately lose the game). More results can be found Appendix A.3
FUTURE EXTENSIONS CONTINUOUS ACTION SETS
We note in this section that, at least from a mathematical point of view, it is possible to extend our formulation continuous control tasks. Observe that the adversarial objective in (8) applies to continuous control problems too. The difficulty we are describing in (IO) is not an issue in continuous control tasks. For instance, in Deep Deterministic Policy Gradient (DDPG) Proximal Policy Optimization (PPO) the adversarial policy is already approximated by the actor network pg(s). That is: (Saay) (M0(sa)) Unlike in the case of discrete action sets, this means that the solution the problem in can be approximated through gradient descent following,
aQ(s, a) da
V(Q(s, uo (saav)
( V (SLo
(18)
8

---

Under review as a conference paper at ICLR 2021
where the gradient is taken with respect to Sadv- In a way it is actually easier to construct the adversarial examples in continuous control tasks because the learning algorithm already produces a differentiable approximation (s) to the argmax operation used in action selection. In our paper focused on the derivation for the case of a discrete action set because the optimization problem in (TO) is harder to solve.
5 CONCLUSION
In this paper we studied formulations of adversarial attacks on deep reinforcement learning agents and defined the optimal myopic adversary which incorporates the action taken in the adversarial state into the cost function. By introducing differentiable approximation to the value of the action taken by the agent under the influence the adversary we find a direction for adversarial perturbation which more effectively decreases the Q-value of the agent in the given state. In experiments demonstrated the efficacy of our formulation as compared to the previous formulations in the Atari environment for various games. Adversarial formulations are inital steps towards building resilient and reliable DeepRI. agents, and we believe our adversarial formulation can help to set a new baseline towards the robustification of DeepRI. algorithms.
REFERENCES
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael. Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research. pp. 253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv: 015440 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57, 2017.
Xianfu Chen, Honggang Zhang, Celimuge Wu, Shiwen Mao, Yusheng Ji, and Medhi. Bennis. Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning. IEEE Internet of Things Journal 6, pp. 4005-4018, 2018.
Sandeep Chinchali, Pan Hu, Tianshu Chu, Manu Sharma, Manu Bansal, Rakesh Misra, Marco Pavone, and Sachin. Katti. Cellular network traffic scheduling with deep reinforcement learning. In Thirly-Second AAAI Conference on Artificial Intelligence. 2018
Tianshu Chu, Sandeep Chinchali, and Sachin Katti. Multi-agent reinforcement learning for networked system control. In Proceedings of the International Conference on Learning Representations
(ICLR), 2020.
Liu Daochang and Tingting. Jiang. Deep reinforcement learning for surgical gesture segmentation and classification. In International conference on medical image computing and computer-assisted intervention., pp. 247-255. 25..ingge Cham, 2018
Alexey Dosovitsky, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Proceedings of the Conference on Robot Learning (CoRL), 78: 1-16, 2017.
Jiajun Duan, Di Shi, Ruisheng Diao, Haifeng Li, Zhiwei Wang, Bei Zhang, Desong Bian, and Zhehan. Yi. reee-reinforcemen--leaingg based autonomous voltage control for power grid operations.
IEEE Transactions on Power Systems, 2019.
Florin-Cristian Ghesu, Bogdan Georgescu, Yefeng Zheng, Sasa Grbic, Andreas Maier, Joachim Hornegger, and Dorin. Comaniciu. Multi-scale deep reinforcement learning for real-time 3d- landmark detection in ct scans. IEEE transactions on pattern analysis and machine intelligence 41, pp. 176-189, 2017.
Adam Gleave, Michael Dennis, Cody Wild, Kant Neel, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. International Conference on Learning Reprexentations ICLR, 2020.

---

Under review as a conference paper at ICLR 2021
Ian Goodfellow, Jonathan Shelens, and Christian Szegedy. Explaning and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Shixiang Gu, Timothy Holly, Ethan andLillicrap, and Sergey, Levine. Deep reinforcement learning for robotic manipulation with asynchronous off- aff-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 2389-3396, 2017.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Diamos Greg, Erich Else, Ryan Prenger, Sanjeev Satheesh, Sengupta Shubho, Ada Coates, and Andrew Ng. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.556; 2014.
Charlie Hou, Mingxun Zhou, Yan Ji, Phil Daian, Florian Tramer, Giulia Fanti, and Ari. Juels. Squirrl: Automating attack discovery on blockchain incentive mechanisms with deep reinforcement learning. arXiv preprint 1912.0179 2019.
Qiuhua Huang, Renke Huang, Weituo Hao, Tan Jie, Rui Fan, and Zhenyu. Huang. Adaptive power system emergency control using deep reinforcement learning. IEEE Transactions Smart Grid (2019)., 2019.
Sandy Huang, Nicholas Papernot, Yan Goodfellow, lan an Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. Workshop Track of the 5th International Conference on Learning Representations, 2017.
Nathan Jay, Noga Rotman, Michael Godfrey, Brighten Schapira, and Aviv. Tamar. deep reinforce- ment learning perspective on internet congestion control. In International Conference on Machine Learning, pp. 3050-3059, 2019.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint Stf::1800.10292 2018.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. International Conference on Learning Representations, 2017.
Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 2012.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan. Wierstra. Continuous control with deep reinforcement learning. arXivpreprint arXiv:1509.02971 2015.
Yen-Chen Lin, Hong Zhang-Wei Yuan-Hong Liao, Meng-Li Shih, ing-Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. Proceedings of the Twenty-Sixe International Joint Conference Artificial Intelligence, pp. 3756-3762, 2017.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically-plausibe perturbations. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3932-3939. 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, arc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Antonoglou, Helen King. Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518: 529-533, 2015.
Volodymyr Mnih, Adria Badia Puigdomenech, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International Conference on Machine Learning, pp. 1938-1937, 2016.
Laura Noonan. Jpmorgan develops robot to execute trades. Financial Times, pp. 1928-1937) July 2017.
10

---

Under review as a conference paper at ICLR 2021
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, and Bommannan Gautham. Robust deep reinforce- ment learning with adversarial attacks. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2040-2044 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce-
ment learning. International Conference on Learning Representations ICLR, 2017.
Mariya Popova, Olexandr Isayev, and Alexander. Tropsha. Deep reinforcement learning for de novo drug design. Science advances 4, 78, 2018
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference Machine Learning (ICML-15), 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg, Klimov. Proximal policy optimization algorithms. arXiv: (etv::77770634700 [cs. 2017.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 2014.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dimutru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the International Conference Learning Representations (ICLR), 2014.
Brijen Thananjeyan, Animesh Garg, Sanjay Krishnan, Carolyn Chen, Lauren Miller, and Ken. Gold- berg. Multilateral surgical pattern cutting in 2d orthotropic gauze with deep reinforcement learning policies for tensioning. In 2017 IEEE International Conference on Robotics and Automation (ICRA)., pp. 2371-2378, 2017.
Huan-Hsin Tseng, Sunan Cui Yi Luo, Chien Jen-Tzung, Randall K, Ten Haken, and Issam El Naqa. Deep reinforcement learning for automated radiation adaptation in lung cancer. Medical Physics, (12):6690-677) 6705, 2017.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando. De Freitas. Dueling network architectures for deep reinforcement learning. Internation Conference on Machine Learning ICML,, pp. 1995-2003, 2016.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 Lo -NORM BOUND
Table 3: Attack impacts for three attack formulations with &oo norm bound and 10-
GAMES
HUANG
PATTANAIK
MYOPIC
AMIDAR
0,208:10.14
0.290.1.337
0.536:10.25
BANKHEIST
0.1974-0.12
0.410-10099
0.527:40.16
BEAMRIDER
0.258:10.27
0.27910.26
0.327:10.19
0.081110.26
0.323-0.27
0.429-00229
ROADRUNNER
0.125--0.16 16
0.445+0.13
0.553++0.1
PONG
-0.002:40.01
0.065+0.00
0.086:
SEAQUEST
0.34410.25
0.449-1.24
0.450:0,23
UPNDOWN
0.244110.28
0.369
0.532.10.16
One observation from these results is that the myopic formulation performs worse in the foc -orm bound than it does in the ε --normm bound and the f2-norm bound when = 10-8 A priori one might the myopic formulation to best in & -oom bound because the unit lx ball contains the unit balls of the l and & norms. To investigate this further we examined the behaviour of f2-noo to-norm bound and lo -oom bound while varying in Figure (5). Observe that at very small scales the E--no fo-morm bounded perturbation has larger impact, but larger scales the foo norm bounded perturbation generally performs better than the f-rnrrm bounded perturbation. We link this phenomenon to the decision boundary behaviour in different scales.
expect the performance
Myopic Norm Bound Comparison for Roadrunner
Myopic Norm Bound Comparison for Bankheist
0.9
0.0
0.7
0.7
0.8
0.*
0.5
0.4
0.3
-10
0.2
loguolel
logulel
Figure Left: Impact change with varying e for & -norm bounded and f norm bounded perturbation in Myopic formulation for Roadrunner game. Right: Impact change with varying e for foc -oom bounded and f2-m bounded perturbation in Myopic formulation for Bankheist game.
Recall that in the 22-norm bound use a perturbation in the gradient direction of length However, in the -oom bound we use a perturbation given by times the sign of the gradient. This corresponds to the -nom bounded perturbation which causes the maximum possible change for linear function. Compared to previous formulations decreasing the temperature in the softmax makes our objective function more nonlinear. Thus, at smaller scales it is crucial for the perturbation to be in the exact direction of the gradient, since the decision boundary behaves nonlinearly. However, larger scales the decision boundary is approximately linear, which gives a better result in the f -nom bound.
12

---

Under review as a conference paper at ICLR 2021
A.2 RANDOMIZED ITERATIVE SEARCH AND IMPACTS
It can be seen from Figure 6 that the impact of the Pattanaik et al. (0018) formulation increases as n increases, while myopic formulation has a greater impact even when equal to In partieular, this is again an indication that our cost function provides a better direction in which to search for adversarial perturbations compared previous formulations.
1.0
Mypp
impacts
0.6
0.4
0.2
10
30
50
Figure 6: Attack impacts vs number of iteration of random search (n) for Pong.
Our attack, as well as those of Huang et al. (2017) and Pattanaik et al. (2018), are all based on computing the gradient of the respective cost functions and choosing point along the gradient direction to the adversarial perturbation. Pattaaaik et al. (2018) used a randomized search, where random points sampled in the gradient direction and the one with lowest Q-value is chosen as the perturbation. We follow Pattanaik et al. (0088) in using randomized search, and in all previous results mentioned above have set n In Table 4 we set three different formulations. Even in this more restrictive setting the performance of our formulation remains high, while the performance of Pattanaik al. (2018) degrades significantly. In particular, this demonstrates that the gradient of our cost function is a better direction to find an adversarial perturbation.
and compare the impact of the
Table 4: Attack impacts for three different attack formulations with & norm bound and 10 10 =
GAMES
HUANG
PATTANAIK
MYOPIC
AMIDAR
0.050::0. 25
0.13820.31
0.941
0.18910.13
0.24710.15
0.48710.09
BEAMRIDER
0.001=0.40
0.096+0.36
0.634=0.15
RIVERRAID
0.173=0.23
0.23420.21
0.367::016
ROADRUNNER
0.035::0.15 0.09010.12
0.15110.12
PONG
0.173::0.23 0.01410.03
0.887
SEAQUEST
0.321
0.290+0.4
0.50240..2
UrNDOWN
0.475=0024
0.615+0.10
0.91110.04
A.3 GAMES AND AGENT BEHAVIOUR
In this section we share our observations on the behaviour of the trained agent under attack. In Figure 9 the agent performs well until it suddenly decides to stand still and wait for the enemy to arrive. Similarly, in Figure 7 the trained agent performs quite well again until it decides to jump in front of the truck. Finally. in Figure 8 the trained agent forgets to recharge its oxygen even though it is earning many points from shooting the fishes and saving the divers.
13

---

Under review as a conference paper at ICLR 2021
10300
11300
11500
Figure 7: Example game from Atari Arcade Environment. The trained agent jumps in front of the car even though the agent not in the same lane with the car in RoadRunner game.
3970
4000
4030
*
OXTGEN
OXYGEN
Figure 8: Example game from Atari Arcade Environment. The trained agent forgets to recharge its oxygen even though its condition to recharge is not critical in Seaquest game.
1
1
1
Figure 9: Example game from Atari Arcade Environment. moving until the enemy reaches the agent in Amidar game.
The trained agent just waits without
14

---

Under review as a conference paper at ICLR 2021
A.4 Q- Q-VALUES OVER TIME
In this section we plot state-action values of the agents over time without an attack, under attack with the Pattanaik al. 2018)) formulation, and under attack with myopic formulation. Figure 12 interesting to observe that under attack the mean of the state-action values over the episodes might be higher despite the average return being lower.
iiciinnio Values
2010
15
1003
Figure 10: State-action values vs time graph for Atari Arcade Game Seaquest. Left: Unattacked agent. Middle: Pattanaik al. (2018) adversarial formulation. Right: Myopic attack.
Vakur
Figure 11: State-action values time graph for Atari Arcade Game Riverraid. Left: Unattacked agent. Middle: Pattanaik et al. (2018) adversarial formulation. Right: Myopic attack.
Actice Volaen
State Statedatioon do Time
130
70
400
Figure 12: State-action values vs time graph for Atari Arcade Game Bankheist. Left: Unattacked agent. Middle: Pattanaik al. (2018) adversarial formulation. Right: Myopic attack.
15

---

Under review as a conference paper at ICLR 2021
A.5 RESULTS ON AVERAGE EMPIRICAL PROBABILITIES ACTIONS
In this section we provide a detailed table on the average empirical probabilities of the best action a² and the worst action aw It can be that in general (e>n[p(s 8s a a²)] is lower and Be< (m)) is higher for our Myopic formulation when compared to other attack formulations.
Table 5: Average empirical probabilities of a" and Owr and impacts for three different formulation for Riverraid and BeamRider from Atari environment with 2--morm bounded perturbation and e 10
Beep (p(s, a")]
Be Remppp((s.n aww)
Impact
Riverraid
Huang el. al.
0.9904-L0.16
0.00226++0.0003
-0.0796 +0.19
Pattanaik et. al. Myopic
0.9443=0.29
0.00285 #0.0014
0.3797
0.897++0.16
0.00369+0.0001
0.5897+0..2
BeamRider
Huang el. al.
0.989-40.244
0.0042++0.0005
0.0847+0:29
Pattanaik et. al.
0.948110.200
0.0059+1.0022
0,4553
Myopic
0.826++0.18 195
0.0186+-0.008
0.6636+0..335
Amidar
Huang et. al.
0,9872:10.168
0.001110.0008
0.04762:10.11
Pattanaik et. al. Myopic
0.9377--0.34 0.0052++0.002 00022
0.5145+0.126
0.8566++0.211
0.01225+000035
0.9320+0.198
BeikH Huang et. al.
0.986110.100
0.00035710.00059
0.111
Pattanaik et. al. Myopic
0.9426++0.012
0.00261+00011
0.373+0.11
0.897410.122
0.0043310.0036
0.624
Pong
Huang et. al.
0.468410.056
0.002011.001
0.0:0.0016
Pattanaik et. al. Myopic
0.5151+00.06
0.01028 +:00004
0.067+00.66
0.5695::0.159
0.0622110.003
0.920110008
16

---

Under review as a conference paper at ICLR 2021
A.6 SENSITIVITY ANALYSIS
In this section we attempt to gain an insight into which pixels in the visited states are most sensitive perturbation. For each pixel i.j we measure the drop in the state action values when perturbing that pixel by a small amount. Formally, let Osen arg max, (* a), where ssenitivity is equal except that small perturbation 7 has been added to the 2, j-th pixel of s. Then we measure
Q(s,a") - Q(s, Osen)
(19)
We plot the values from 19 in Figure 15 Note that any
value (i.e. any lighter colored pixel the plot) indicates that a 7 perturbation to the corresponding pixel will cause the agent to take an action different from the optimal one. Lighter colored pixels correspond to larger drops in Q-value.
0.16
0.14
0.12
5033 3022
0.20
0.10
0.15
0.08
www.cooo..oooo CZo0
0.10
0.04
0.02
121620242202 02200240227000000
Figure 13: Sensitivity Analysis of RoadRunner. Left: 7 10-8, Right: ? 10-10
17

---

Under review as a conference paper at ICLR 2021
A.7 PERTURBATION HEATMAPS
In thise section we demonstrate the heatmaps of the raw pixel perturbations and the preprocessed frames from which the perturbations computed.
1e-10
5.0
2.5
2
0
-2
-3.0
$400
-7.
--6
0 4 8
Figure 14: Perturbation heatmaps of myopic formulation. Left: RoadRunner. Right: Riverraid.
530
Figure 15: Preprocessed frames. Left: RoadRunner. Right: Riverraid.
18


Reviews:
[{"review_id": "xy6jUe8HN9l", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  This paper proposes to build an adversary to find a bounded perturbation of the state that minimizes the value of the action taken by the reinforcement learning agent. This approach enables us to lower the bounds by several orders of magnitude on the perturbation needed to efficiently achieve significant impacts on DeepRL agents.\n\nStrong aspects:  \n1.     The approach is simple and straightforward with good numerical performance.\n2.     The writing is easy to follow and experiments are thorough.\n\nWeak aspects:\n1.     It is strange that, for a state of thousands of dimensions, a tiny perturbation within a ball having a radius of $10^-10$, can make maximum possible deterioration in terms of return (i.e., Fig. 3). Can the authors provide more insights? For example, the sensitivity analysis of the return / Q value with regard to the states might help.\n\n2.     How does the proposed approach compare to other more recent baselines such as Gleave et al. (2020)?\n\n3.     Is the metric in (16) a common one? What if we simply plot the return under various perturbations, like that in Huang et al. (2017)?\n\nMinor points:\n\n- The locations and / or sizes of the legends in Fig. 2, 3 and 4 and the size of axis labels can be adjusted accordingly to make them easier to read.\n\n\nI have read the authors' response and the associated discussions, and based on that  raised my evaluation by 1", "extra_scores": {}, "extra_text": {}}, {"review_id": "ucSNz4TxLon", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes using a better quantitative metric to conduct an attack on a DRL learner. The attack is limited to an attack on a state (not over multiple states) and aims to lower the Q value from this state by making the worst action be the action chosen to be played. The experiment on Atari games show promising results.\n\nPros:\n- The idea is effective, finding the right attack objective is interesting and surprising that was not considered earlier.\n- Good description of why prior methods do not achieve the optimal attack\n- The experiments show good results\n\nCons:\n- The techniques are not very novel, softmax (with temperature) as a soft differentiable version of argmax is very well known.\n- A comparison to non-myopic attack would make paper stronger (https://arxiv.org/pdf/1907.09470.pdf)\n- The legends in the figure are just too small to be readable\n- Would have been good to show attacks on more complex problems.\n\nQuestions:\n- This attack is for a particular state, which state is chosen for this attack? Is it towards the start of the game or end of the game?\n- Why is E[Q(s; aw)] different for different approaches in Table 2, and also for E[Q(s; a*(s))]? These values should not depend on the attack.\n- Aren't the variances too high in Table 1? Is 10 episodes enough - why not more?", "extra_scores": {}, "extra_text": {}}, {"review_id": "dHt15HdpK_y", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work introduces a new method for observation-perturbation adversarial attacks against deep neural network policies. The idealized version of the method is an optimal attack assuming (a) the agent's Q-values are calibrated; (b) the attacker can only act once, at the current time step. While the attack is in fact conducted at every time step, so is not \"myopic\", this assumption greatly simplifies the problem. Moreover, the work approximates the (deterministic, hard-max) policy with a low-temperature softmax policy to enable gradient-based methods to work. Evaluating in a range of Atari games, the method has a significantly larger \"impact\" (normalized return) than previous methods, across $\\ell_1$, $\\ell_2$ and $\\ell_{\\infty}$ norms. Moreover, the results suggest the attack succeeds because of increasing the probability of low-ranked actions, consistent with the attack's derivation.\n\nStrengths:\n  - Empirical results show a substantial improvement over a reasonable choice of baseline methods.\n  - Experiments also support author's claims for why the method works (especially section 3.4 but also to a lesser extent section 3.5) -- important given the approximations needed from the derivation.\n  - The paper gives a good intuition for why the method should work, and clearly highlights the limitations of previous methods.\n\nWeaknesses:\n  - The justification of the threat model could be improved. I appreciate the list of examples in paragraph 3 of the intro for where RL applications have been studied, but why is this threat model actually relevant to these example? For example, take financial trading: an attacker cannot perturb other peoples orders in the market data feed (unless they've compromised the exchange -- in which case there are much simpler routes to exploitation!), but can insert their own orders with almost arbitrary price/size. This does not fit an $\\ell_p$ norm, which assumes modifying any field is equally possible. I would suggest picking a use case where this threat model does hold and going into detail on it -- or if there is none, then be more up front about this limitation.\n  - Contrary to section 4 I am not convinced this method can be easily applied to continuous control tasks. First, policies for continuous control need not be stochastic. Typically, the policy outputs parameters of a distribution. At training one samples from this distribution -- but at inference one often takes the mode, deterministically, since it improves performance. Even if we allow for a stochastic policy, one has the new challenge of a much larger action space.\n\n    My best guess is the method could be made to work, given that e.g. Pattanaik's method is not all that different and worked on MuJoCo. But I expect it to require some non-trivial tuning, and possibly new algorithmic insights. This section should really be supported by experiment, or have the claim reduced in scope to there being no *obvious theoretical hurdles* to its application.\n  - Paper should be more up-front about assumptions being made. I did like the last paragraph of section 3 (before section 3.1). But you make other assumptions implicitly. For example, the agent's Q-values may be miscalibrated, such that the worst action for the agent is not actually the one the agent assigns lowest Q-value to! This will make your method suboptimal (even in the myopic case). Additionally, the fact you attack at each time step but choose the attack myopically must be leaving some value on the table. Indeed, the fact that expected Q-values often rise (table 2) under the attack is some evidence of this. I think the paper is a good submission without fixing these limitations (though it would be much stronger if they were addressed), but it's important they're clearly signposted so that they can be addressed in future work.\n\nI consider this paper borderline but overall am leaning towards accept. There is a clear theoretical reason why the idealized method should perform better than previous methods (e.g. Huang, Pattaniak) and the empirical results support that the approximation is stronger than baselines. The threat model of this (and prior work) seems chosen more for mathematical simplicity than realism, but may help lay the ground for future work in more realistic settings. The paper could do a better job of communicating the assumptions and limitations, but assuming this is addressed then it seems worth disseminating to the ICLR community.\n\nA few questions:\n  - Do you have any additional insights into why the mean Q-value per episode often increases under attack? This is a very counterintuitive result -- why should choosing worse actions lead you to higher-valued states? Some things that might be worth looking into (though I certainly do not expect all of these during the relatively brief discussion period):\n    + What return do you get if you perform the attack for 1-timestep (or a smaller number of timesteps) and then run the unattacked agent? This is what the Q-value is actually estimating -- but Q-values learned by deep RL are often very uncalibrated, especially off-distribution.\n    + What happens in other environments? It strikes me that most Atari games are quite hard to get \"stuck\" in: even if you die you respawn. Some environments usually used to test safe exploration might be helpful with this.\n    + What happens with attacks that are non-myopic, e.g. the \"enchanting attack\" of Lin et al (2017)?\n  - Do you have any experiments in continuous control tasks that would back up section 4, even if preliminary?\n  - Why are \\ell_p norm perturbations of observations an important threat model to study? I know it is widely studied, but it is also widely criticized -- e.g. Gilmer et al (2018) -- and for RL in particular it seems rare for an attacker to be able to directly perturb observations. Are there cases where this attack is realistic? Are there non-security reasons to care, e.g. to improve robustness to natural \"perturbations\"? If so, can we validate this in a realistic setting, e.g. does vulnerability to your method also predict failures of sim2real transfer, or sensitivity to non-adversarial noise?\n  - What do the perturbations actually look like? Do they look like anything reasonable, e.g. introducing a fake \"ball\"? Given you claim the perturbation direction identified by this method is likely stronger, it'd be interesting to know what neural networks find most \"persuasive\".\n\nDetailed feedback:\n  - Need to use parenthetical citations \\citep in places rather than in-line citations \\citet: e.g. first paragraph in the intro “speech recognition Hannun et al. (2014)” -> “speech recognition (Hannun et al, 2014)”; first paragraph of 3.1.\n  - “perturbations to image”->”perturbations to images”.\n  - Intro: focus it more on your method. It's a good summary of some of the history of adversarial examples – but is this that relevant? Most readers (and certainly your reviewers) will be familiar with adversarial examples. Adversarial examples in RL might need some handholding being less common, but can probably also be assumed. \n  - Related, could benefit from an explicit statement of what your threat model is early on (an adversary making bounded perturbations of the observations of an RL agent). I appreciate the list of diverse scenarios RL has been applied it, but I would suggest making this more focused on those that are actually deployed on nearing deployment (especially if safety critical).\n  - Related Work: 2.1 is a nice succinct summary of existing work but it'd be helpful to the reader to place it in relation to your own work. In particular it seems your method is much more closely related to Huang, Kos & Song and Pattanaik et al than it is to Gleave et al or Pinto et al. It therefore might be useful to treat these as two groups (“multi-agent” adversarial RL v.s. “observation” adversarial RL, say); discuss why you chose to work under the observation threat model; and then dig into details on how your approach differs from Huang, Kos, Pattanaik (e.g. is it the same threat model but stronger empirical results? do you make stronger/weaker assumptions?).\n  - “$\\pi(s,a)$ is not the actual policy used by the agent” – bit confusing, can you change notation to make this explicit e.g. $\\pi_{\\text{soft}}$? Also good to be clear the difference is just soft vs hard-max (I assume), rather than e.g. $\\pi$ being learned by behavioral cloning the policy under attack. Perhaps explicitly state earlier your attack is white-box?\n  - I liked section 2.3: it was an easy to read summary that clearly explains the problems with existing work.\n  - Section 3: your method is not really “optimal” since you approximate the hard-max policy with a soft-max policy of low temperature, so the section is a bit misleading. It might be better to frame it as: (1) definition of optimal attack; (2) tractable approximation to this. I do appreciate the discussion in the last paragraph of section 3 (before section 3.1) of why this is an approximation.\n  - Figure 4: can you use the subcaption package to give each individual barchart its own label? This is easier than having to consult the main caption. It's also worth making explicit that these plots do not inculde the best action – I was wondering for a while where the rest of the probability mass was! I'd be inclined to even add the best action to the plot (perhaps visually distinguished in some way, and using a log-scale for probabilities if it would compress it too much).\n  - Table 2: caption would benefit from being expanded.\n\nUpdate after author response: The changes made have improved the clarity of the paper, such as making assumptions and the threat model more explicit, and the heatmap addition provides a nice qualitative insight. However, I am inclined to agree with other reviewers that the paper's contribution is incremental. Given this I am retaining my score of marginally above acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PcMKzTc55-N", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes an optimal myopic adversary for deep reinforcement learning agent, in which the adversary finds a bounded perturbation of the state that minimizes the value of the action taken by the agent. The authors introduce a differentiable approximation for the optimal myopic adversarial formulation that leads to a better direction for the adversarial perturbation and increases the attack impact for bounded perturbations. Empirically, the authors show with experiments in various games in the Atari environment that the attack formulation achieves significantly larger impact as compared to the current state-of-the-art. \n\nThe paper is easy to follow, and the topic investigated looks interesting. However, a few major issues unfortunately in the draft discourage me to accept the paper.\n\n1. The novelty in the paper is marginal. The conclusions drew from this paper are completely based on the experimental results. Though the formulation looks interesting, in-depth analysis is missing in the paper. No major theoretical results have been reported to provide stronger support for the bounded myopic adversary.\n\n2. In Section 3, the authors have known that Eq (14) and Eq (15) may not be equivalent to each other, they fail to provide more analysis and discussion on how to address it, instead, only relying on the empirical results. Also, how to choose a sufficiently small adversarial temperature constant is not clear, as suggested in the paper. It gives a sense that the authors only focused on the applicability of the formulation, while ignoring necessary theoretical justification.\n\n3. No empirical results for the continuous action set. The authors have mentioned the decent applicability of the myopic adversary in the continuous tasks. They should also have shown some results to support their claim, as failing to do this makes the paper look incomplete. Unless the proposed is only devoted to the discrete tasks.\n\nMinor point:\n\n4. Regarding the experimental results, I wonder why the authors put Huang and Pattanaik together for Atari games. Though this works, it would be better to see both discrete and continuous tasks separately for these two different baselines. Also, the evaluation metric in the paper is Impact instead of returns, which is not popular in literature. I suggest the authors to include this in the appendix.\n\n*********************************\nAfter carefully considering the rebuttal from the authors, I am going to maintain the score based on my evaluation and also the current paper draft. Though the authors have tried to addressed the comments, the paper still requires more improvements, including theoretical novelty and experimental results.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most reviewers are positive about this work, though they believe it is somewhat incremental, and its theoretical contributions are minor. None of the reviewers are very excited about this work. Overall, the PC believes this is a borderline paper.

Minor note: During the discussions, the paper by Xiao et al., "Characterizing Attacks on Deep Reinforcement Learning" (2019) was brought up. The authors claimed that they did not compare with that paper because the best attack there (obs-fgsm-wb) had already been studied. In a later stage of discussions, one of the reviewers stated that the method obs-nn-wb in that paper performed better in some domains. Even though this is not a major issue, it is advisable to the authors to make sure that this is indeed the case, and if it is, provide proper comparison with that paper.

We encourage the authors to consider the reviewers' comments to improve the paper and resubmit to a future venue.
Decision: Reject