Adversarial Policies: Attacking Deep Reinforcement Learning
Authors: Adam Gleave
Michael Dennis
Cody Wild
Neel Kant
Sergey Levine
Stuart Russell
Venue:  (2020)
Keywords: deep RL
adversarial examples
security
multi-agent

Abstract: Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at https://adversarialpolicies.github.io/.

Published as a conference paper at ICLR 2020
ADVERSARIAL POLICIES: ATTACKING DEEP REINFORCEMENT LEARNING
Adam Gleave" Michael Dennis Cody Wild Neel Kant Sergey Levine Stuart Russell
University of California, Berkeley
ABSTRACT
Deep reinforcement learning (RL) policies are known to be vulnerable to adversar- ial perturbations their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's 's observa- tions. This might lead one to wonder: is it possible attack an RL agent simply by choosing an adversarial policy acting in multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of of-thh-art victims trained via self-play to be robust opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in digenniienii environments, and induce substantially different activations in the victim policy network than when the victim plays against normal opponent. Fine-tuning protects vietim against specific adversary, but the attack method can be successfully reapplied to find new adversarial policy. github. io/
Videos are available at hntttps:: brttsss/addeeraaiaaapool.
INTRODUCTION
The discovery of adversarial examples for image classifiers prompted new field of research into adversarial attacks and defenses (Szegedy et 2014). Recent work has shown that deep RL policies are also vulnerable to adversarial perturbations of image observations (Huang et al. 2017: Kos and Song 2017). However, real-world RL agents inhabit natural environments populated by other agents, including humans, who can only modify another agent's observations via their actions. We explore whether it's possible attack victim policy by building an adversarial policy that takes actions in shared environment, inducing natural observations which have adversarial effects on the victim.
RL has been applied in settings as varied autonomous driving (Dosovitskiy et al. (017), nego- tiation (Lewis et al.) (017) and automated trading (Noonan 2017). In domains such as these, an attacker cannot usually directly modify the victim policy's 's input. For example, in autonomous driving pedestrians and other drivers can take actions in the world that affect the camera image, but only in a physically realistic fashion. They cannot add noise to arbitrary pixels, or make a building disappear. Similarly, in financial trading an attacker can send orders to an exchange which will appear in the victim's 's market data feed, but the attacker cannot modify observations of a third party's 's orders.
As proof of concept, we show the existence of adversarial policies in zero-sum simulated robotics games with proprioceptive observations (Bansal et al., (018a)) The state-of- of-the-an victim policies were trained via self-play to be robust opponents. We train each adversarial policy using model-free RL against fixed black-box victim. We find the adversarial policies reliably beat their vietim, despite training for less than 3% of the timesteps initially used to train the victim policies.
Critically, we find the adversaries win by creating natural observations that are adversarial, and not by becoming generally strong opponents. Qualitatively, the adversaries fall to the ground in contorted positions, as illustrated in Figure I rather than learning to run, kick or block like normal opponents. This strategy does not work when the victim is 'masked' and cannot the adversary's 's position, suggesting that the adversary succeeds by manipulating a victim's observations through its actions.
Having observed these results, wanted to understand the sensitivity of the attack to the dimen- sionality of the victim's observations. We find that victim policies in higher-dimension Humanoid
Corresponding author. F-mail: www.eaeeeeeeeee...ee..

---

Published as a conference paper at ICLR 2020
Victim
Normal
Norunl Opponent
Victim
Adversarial
adverrsaiia
Opponent
Figure 1: Illustrative snapshots of victim (in blue) against normal and adversarial opponents (in red). The victim wins if it crosses the finish line; otherwise, the opponent wins. Despite never standing up, the adversarial opponent wins 86% of episodes, far above the normal opponent's 47% win rate.
enviromments are substantially more vulnerable adversarial policies than in lower-dimensional Ant environments. To gain insight into why adversarial policies succeed, we analyze the activations of the victim's policy network using a Gaussian Mixture Model and t-SNE (Maaten and Hinton 22008). We find adversarial policies induce significantly different activations than normal opponents, and that the adversarial activations are typically more widely dispersed between timesteps than normal activations.
natural defense is to fine-tune the victim against the adversary. We find this protects against that particular adversary, but that repeating the attack method finds adversary the fine-tuned victim vulnerable to. However, this new adversary differs qualitatively by physically interfering with the victim. This suggests repeated fine-tuning might provide protection against range of adversaries.
Our paper makes three contributions. First, we propose a novel, physically realistic threat model for adversarial examples in RL. Second, we demonstrate the existence of adversarial policies in this threat model for several simulated robotics games. Our adversarial policies reliably beat the victim, despite training with less than 3% many timesteps and generating seemingly random behavior. Third, we conduct a detailed unalysis of why the adversarial policies work. We show they create natural observations that are adversarial to the victim and push the activations of the victim's policy network offfdistribution. Additionally, we find policies are easier to attack in high-dimensional environments.
As deep RL increasingly deployed in environments with potential adversaries, we believe is important that practitioners are aware of this previously unrecognized threat model. Moreover, even in benign settings, we believe adversarial policies can useful tool for uncovering unexpected policy failure modes. Finally, which could improve robustness relative to conventional self-play by training against adversaries that exploit weaknesses undiscovered by the distribution of similar opponents present during self-play.
excited by the potential of adversarial training using adversarial policies,
RELATED WORK
Most study of adversarial examples has focused on small Ep norm perturbations to images, which Szegedy et al.) (2014) discovered cause a variety of models confidently misclassify the image. even though the changes are visually imperceptible to human. Gilmer et al. (2018a) argued that attackers are not limited small perturbations, and can instead construct new images or search for naturally misclassified images. Similarly, |eesato et al. 2018) argue that the near-ubiquitous Ép model is merely a convenient local approximation for the true worst-case risk. We follow Goodffllow et al. (2017) in viewing adversarial examples more broadly, "inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake."
The little prior work studying adversarial examples in RL has assumed an lp f--norm threat model. Huang et al. (2017) and Kos and Song (2017) showed that deep RL policies are vulnerable to small perturbations in image observations. Recent work by Lin et al. (201)) generates sequence of
2

---

Published as a conference paper at ICLR 2020
perturbations guiding the victim to target state. Our work differs from these previous approaches by using a physically realistic threat model that disallows direct modification of the victim's 's observations.
Lanctot et al. (2017) showed agents may become tightly coupled to the agents they were trained with. Like adversarial policies, this results in seemingly strong polices failing against new opponents. However, the victims attack win against range of opponents, and are not coupled in this way.
Adversarial training is a common defense to adversarial examples, achieving state-of ofate-of-thheart the-art robust- ness in image classification (Xie et al. (211)). Prior work has also applied adversarial training to improve the robustness of deep RL policies, where the adversary exerts a force vector on the victim varies dynamics parameters such friction (Pinto et al.. 2017 Mandlekar al. 2017 Pattanaik iet al. 2018 Our defense of fine-tuning the victim against the adversary is inspired by this work.
This work follows rich tradition of worst-case analysis in RL. In robust MDPs, the transition function is chosen adversarially from an uncertainty set (Bagnell et al. 2001] Tamar et al., 2014). Doyle et al. (1996) solve the converse problem: finding the set of transition functions for which a policy is optimal. Methods also exist to verify controllers or find counterexample to specification. [Bastani et al.) (2018) verify decision trees distilled from RL policies, whileGGhosh et al. (2018) test black-box closed- loop simulations. |Ravanbakhsh et al (2016) can even synthesize controllers robust to adversarial disturbances. Unfortunately, these techniques are only practical in simple environments with low- dimensional adversarial disturbances. By contrast, while our method lacks formal guarantees, it can test policies in complex multi-agent tasks and naturally scales with improvements in RL algorithms.
FRAMEWORK
We model the victim playing against opponent in two-player Markov game (Shapley 1953). Our threat model assumes the attacker can control the opponent, in which case we call the opponent an adversary. We denote the adversary and victim by subscript a and respectively. The game = = (S. (Aa, Ap).T, T, (Ra, Rv)) consists of state set S, action sets Aa and Ag. and joint state transition function T
Aa A, + A (S) where A (S) a probability distribution on S. The reward function R; : S x A. x Av x S + IR for player E {a,") ) depends on the current state, next state and both player's actions. Each player wishes to maximize their (discounted) sum of rewards.
The adversary is allowed unlimited black-box access to actions sampled from Te. but is not given any white-box information such as weights or activations. We further assume the vietim agent follows a fixed stochastic policy Tu. corresponding to the common case of a pre-trained model deployed with static weights. Note that in safety critical systems, where attacks like these would be most concerning, it is standard practice to validate model and then freeze it, to ensure that the deployed model does not develop any new issues due to retraining. Therefore, a fixed victim is a realistic reflection of what we might see with RL-trained policies in real-world settings, such as with autonomous vehicles.
Since the victim policy n, is held fixed, the two-player Markov game M reduces to a single-player MDP M. (S, A., Ta, R²,) that the attacker must solve. The state and action space of the adversary are the same as in M, while the transition and reward function have the victim policy n. embedded:
Ta (s, aa) = T (s, da, a.) and R² ao.. s') = Ra (s, Cas ay s'),
where the victim's action is sampled from the stochastic policy a, Tp( s). The goal of the attacker is to find an adversarial policy na maximizing the sum of discounted rewards:
Σ (s a ((t+))), where (t+11 Ta(o" .a(()) and da ~ Ta (.) 1=0
Note the MDP's dynamics To will be unknown even if the Markov game's dynamics T are known since the victim policy Tu is black-box. Consequently, the attacker must solve an RL problem.
4 FINDING ADVERSARIAL POLICIES
We demonstrate the existence of adversarial policies in zero-sum simulated robotics games. First, we describe how the victim policies were trained and the environments they operate in. Subsequently, provide details of our attack method in these environments, and describe several baselines. Finally, we present quantitative and qualitative evaluation of the adversarial policies and baseline opponents.
3

---

Published as a conference paper at ICLR 2020
(a) Kick and Defend
(b) You Shall Not Pass
(c) Sumo Humans
(d) Sumo Ants
Figure 2: Illustrations of the zero-sum simulated robotics games from Bansal et al. (201 2a) we use
for evaluation. Environments are further described in Section 4.1
4.1 ENVIRONMENTS AND VICTIM POLICIES
We attack victim policies for the zero-sum simulated robotics games created by Bansal al. (2018a), illustrated in Figure? 2 The victims were trained in pairs via self-play against random old versions of their opponent, for between 680 and 1 3300 million timesteps. We use the pre-trained policy weights released in the "agent zoo" of Bansa et al. (2018)). In symmetric environments, the zoo agents are labeled ZooN where N a random seed. In asymmetric environments, they are labeled ZooVN and ZcoON representing the Victim and Opponent agents.
All environments are two-player games the MuJoCo robotics simulator. Both agents observe the position, velocity and contact forces of joints in their body. and the position of their opponent's joints. The episodes end when a win condition is triggered, or after a time limit, in which case the agents draw. We evaluate in all environments from Bansal et al. (2018a) except for Run to Goal, which omit as the setup is identical to You Shall Not Pass except for the win condition. We describe the environments below, and specify the number of trained zoo policies and their type (MLP or LSTM):
Kick and Defend (3, LSTM). A soccer penalty shootout between two Humanoid robots. The positions of the kicker, goalie and ball randomly initialized. The kicker wins if the ball goes between the goalposts; otherwise, the goalic wins, provided it remains within units of the goal.
You Shall Not Pass (1, MLP). Two Humanoid agents are initialized facing each other. The runner wins if it reaches the finish line; the blocker wins if it does not.
Sumo Humans (3, LSTM). Two Humanoid agents compete on a round arena. The players' positions are randomly initialized. A player wins by remaining standing after their opponent has fallen
Sumo Ants (4, LSTM). The same task as Sumo Humans, but with 'Ant' quadrupedal robot bodies. We use this task in Section 5.2 to investigate the importance of dimensionality to this attack method.
4.2 METHODS EVALUATED
Following the RL formulation in Section we train an adversarial policy maximize Equation using Proximal Policy Optimization (PPO; [Schulman et al.. (017). We give a sparse reward at the end of the episode, positive when the adversary wins the game and negative when it loses or ties. [ansal et al. (018a) trained the victim policies using similar reward, with an additional dense component at the start of training. We train for 20 million timesteps using the PPO implementation from Stable Basclines (Hill et al. 2019). The hyperparameters were selected through combination of manual tuning and a random search of 100 samples; see Section A in the appendix for details. We compare methods to three baselines: policy Rand taking random actions; lifeless policy Zero that exerts zero control; and all pre-trained policies Zoox x from Bansal et al. (018a)).
4.3 RESULTS
Quantitative Evaluation We find the adversarial policies reliably win against most victim policies, and outperform the pre-trained Zoo baseline for a majority of environments and victims. We report
Bansal et al. 2018a) consider the episode to end in tie if player falls before it is touched by opponent. Our win condition allows for attacks that indirectly modify observations without physical contact.
4

---

Published as a conference paper at ICLR 2020
Adversary (Adv)
Normal (Zoo)
Random (Rand)
Zero
Kick and Defend 2
You Shall Not Pass
Sumo Humans 2
100
Win rate (%4
50
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Timestep
1e7
Timestep
1e7
Timestep
1e7
Figure 3: Win rates while training adversary Adv against the median victim in each environment (based the difference between the win rate for Adv and 200). The adversary outperforms the 20o bascline against the median victim in Kick and Defend and You Shall Not Pass, and compet- itive Sumo Humans. For full results, see figure beloow or figure C.T in the supplementary material.
Key: The solid line shows the median win rate for Adv across random seeds, with the shaded region representing the minimum and maximum. The win rate is smoothed with a rolling average over 100 000 timesteps. Baselines are shown as horizontal dashed lines. Agents Rand and Zero take random and zero actions respectively. The Zoo baseline is whichever ZooM (Sumo) or ZooOM (other environments) agent achieves the highest win rate. The victim ZooN (Sumo) or ZooVN (other environments), where N is given in the title above each figure.
the win rate over time against the median victim in each environment in Figure 3 with full results in Figure C.IIin in the supplementary material. Win rates against all victims summarized in Figure
Qualitative Evaluation The adversarial policies beat the victim not by performing the intended task (e.g. blocking goal), but rather by exploiting weaknesses in the victim's 's policy. This effect is best seen by watching the videos at https:////// and Defend and You Shall Not Pass, the adversarial policy never stands up. The adversary instead wins by positioning their body to induce adversarial observations that cause the victim's 's policy to take poor actions. A robust victim could easily win, result we demonstrate in Section [5.]
wdvversate poliiiies g git.ub........ io/] In Kick
This flavor of attacks impossible in Sumo Humans, since the adversarial policy immediately loses it falls over. Faced with this control constraint, the adversarial policy learns a more high-level strategy: it kneels in the center in a stable position. Surprisingly, this is very effective against victim 1, which in 88% of cases falls over attempting to tackle the adversary. However, proves less effective against victims and 3, achieving only 62% and 45% win rate, below Zoo baselines. We further explore the importance of the number of dimensions the adversary can safely manipulate Section [.2
Distribution Shift One might wonder if the adversarial policies win because they are outside the training distribution of the victim. To test this, we evaluate victims against two simple off-distribution baselines: a random policy Rand (green) and lifeless policy Zero (red). These baselines win as often as 30% to 50% in Kick and Defend, but less than 1% of the time in Sumo and You Shall Nor Pass. This is well below the performance of our adversarial policies. We conclude that most victim policies are robust to off-distribttinn observations that are not adversarially optimized.
5 UNDERSTANDING ADVERSARIAL POLICIES
In the previous section we demonstrated adversarial policies exist for victims in range of competitive simulated robotics environments. In this section, we focus on understanding why these policies exist. Specifically, establish that adversarial policies manipulate the victim through their body position; that victims are more vulnerable to adversarial policies in high-dimensional environments; and that activations of the victim's 's policy network differ substantially when playing an adversarial opponent.
5

---

Published as a conference paper at ICLR 2020
100
ZooV1
79
53
50
32
30
ZooVI
81
76
7ooV2
93
63
46
80
Victim
ZooV3
31
84
50
64
29
ZoeSVI
99
84
84
12
18
60
ZooMV1
15
17
18
58
31
13
ZooDV1
11
91
88
57
ZooMV2
27
28
27
36
73
42
21
ZooMV3
24
47
61
19
ZooMV1
83
70
78
Adv1
Adv2
Adv3
Zoo01
Zao03 ZooO2
Rand
Zero
Adv1
AdvS1
AdvD1
Zoo01
Rand
Zero
(a) Kick and Defend.
(b) You Shall Not Pass.
100
Zool
87
9
37
15
80
80
0
Zool
8
7
40
36
38
7
Zoo2
11
9
40
40
4
Zoo2
10
63
14
34
54
71
0
Zoo3
5
6
6
8
40
34
42
44
4
3
Victim
Zoo3
10
Zoo4
10
9
8
8
39
35
40
42
4
3
ZooMI
24
58
93
ZooMI
5
5
5
62
51
58
61
2
4
40
ZooM2
4
3
61
51
54
56
3
3
ZooM2
32
39
39
46
69
ZooM3
3
56
52
56
4
20
ZooM3
62
68
75
66
82
90
ZooM4
62
56
3 2
Adv2
Adv3
Zool
Zooz
Zao3
Rand
Zero
Advi
Adv2
Adv3
Adv4
Zoo1
Zo02
Zoo3
Zoo4
Rand
Zero
Opponent
Opponent
(c) Sumo Humans.
(d) Sumo Ants.
Figure 4: Percentage of games by opponent (out of 1000), the maximal cell in each row is in red. Key: Agents ZooYN are pre-trained policies from Bansal et al. (2018a). where Y E {'V', ",',0","" denotes the agent plays (V)ictim, (O)pponent AdvN are the best adversarial policy of 5 seeds trained against the corresponding Zoo [V] N. Agents Rand and Zero are baseline agents taking random and zero actions respectively. Defended victims ZcoXYN, where X E S* 'D', 'M'), are derived from ZooYN by fine-tuning against a (S)ingle opponent AdvN, (D)ual opponents AdvN and Zoo [O] or by (M)asking the observations.
either side, and N is random seed. Opponents
Zoo
Zoo*2
Rand
Adv
ZooO2
Rand
Zoo Zoo"IV IV
Zoo*3
Adv
Mean Log Probability
-100
-200
Kick and Defend
You Shall Not Pass
Sumo Humans
Environment
(a) Gaussian Mixture Model (GMM): likelihood the activations of vietim's policy network are "normal". We collect activations for 20, 000 timesteps of vic- tim Zeo (V 1 playing against each opponent. We fit 20-component GMM activations induced [O] Error bars 95% confidence interval.
t-SNE activations of Kick and Defend vietim ZooV2 playing against different opponents. Model fit- ted with a perplexity of 250 to activations from 5000 timesteps against each opponent. See Figures C.3 an4[C.4|in the supplementary results for visualizations of other environments and victims.
Figure 5: Analysis of activations of the victim's policy network. Both figures show the adversary Adv induces off-distribution activations. Key: legends specify opponent the victim played against. Adv is the best adversary trained against the victim, and Rand is a policy taking random actions. Zoo +N corresponds to ZooN (Sumo) or ZooON (otherwise). Zoo+ 1T and Zoo + 1V are the train and validation datasets, drawn from Zool (Sumo) Zoo0l (otherwise).

---

Published as a conference paper at ICLR 2020
5.1 MASKED POLICIES
We have previously shown that adversarial policies are able to reliably win against victims. In this section, we demonstrate that they win by taking actions to induce natural observations that are adversarial to the victim, and not by physically interfering with the victim. To test this, we introduce 'masked' victim (labeled ZooMN or ZoOMVN) that is the same as the normal victim ZooN or ZooVN, except the observation of the adversary's position is set to static value corresponding to a typical initial position. We use the same adversarial policy against the normal and masked victim.
One would expect it to be beneficial to be able to see your opponent. Indeed, the masked victims do worse than a normal victim when playing normal opponents. For example, Figure 4b shows that in You Shall Not Pass the normal opponent Zoo01 wins 78% of the time against the masked victim 2ooMV1 1 but only 47% of the time against the normal victim ZooV1. However, the relationship is reversed when playing an adversary. The normal victim ZooV1 loses 86% of the time to adversary Advl whereas the masked victim ZocMV1 wins 99% of the time. This pattern is particularly clear in You Shall Not Pass, but the trend is similar in other environments.
This result is surprising as it implies highly non-transitive relationships may exist between policies even in games that seem to be transitive. A game is said to be transitive if policies can be ranked such that higher-ranked policies beat lower-ranked policies. Prima facie, the games in this paper seem transitive: professional human soccer players and sumo wrestlers can reliably beat amateurs. Despite this, there is a non- con-transitive relationship between adversarial policies, victims and masked victims. Consequently, we urge caution when using methods such as self-play that assume transitivity, and would recommend more general methods where practical (Balduzzi et al. 2019 Brown et al. (019).
Our findings also suggest trade-off in the size of the observation space. In benign environments, allowing more observation of the environment increases performance. However, this also makes the agent more vulnerable to adversaries. This is in contrast to an idealized Bayesian agent, where the value of information is always non-negative (Good, 1967). In the following section, we investigate further the connection between vulnerability to attack and the size of the observation space.
5.2 DIMENSIONALITY
A variety of work has concluded that classifiers are more vulnerable to adversarial examples on high- dimensional inputs (Gilmer et 2b Khoury and Hadfield- Menell 2018 Shafahi et al. (011)) We hypothesize similar result for RL policies: the greater the dimensionality of the component P of the observation space under control of the adversary, the more vulnerable the victim is to attack. We test this hypothesis in the Sumo environment, varying whether the agents are Ants Humanoids. The results in Figures 44 and [ndd/4dssu support the hypothesis. The adversary has a much lower win-rate in the low-dimensional Sumo Ants (dim P 15) environment than in the higher dimensional Suno Humans (dim P = 24) environment, where is the position of the adversary's 's joints.
5.3 VICTIM ACTIVATIONS
In Section 5.1 we showed that adversarial policies win by creating natural observations that are adversarial to the victim. In this section, we seek better understand why these observations are adversarial. We record activations from each victim's 's policy network playing a range of opponents, and analyze these using a Gaussian Mixture Model (GMM) and t-SNE visualization. See Section B in the supplementary material for details of training and hyperparameters.
We fit GMM on activations 2oo+1T collected playing against a normal opponent, 200l or ZooV1, holding out Zoo V for validation. Figure shows that the adversarial policy Adv induces activations with the lowest log-likelihoon
with random baseline Rand only slightly more probable. Normal opponents Zoo*2 2 and 2oo*3 3 induce activations with almost as high likelihood as the validation set Zoo+ 1V, except in Sumo Humans where they are as unlikely as Rand.
We plot t-SNE visualization of the activations of Kick and Defend victim ZooV2 in Figure 5b As expected from the density model results, there is a clear separation between between Adv, Rand and the normal opponent Zoo02. Intriguingly, Adv induces activations more widely dispersed than the random policy Rand, which in turn are more widely dispersed than 2oo02. We report on the full set of victim policies in Figures C.3 and C.4 in the supplementary material.
7

---

Published as a conference paper at ICLR 2020
6 DEFENDING AGAINST ADVERSARIAL POLICIES
The ease with which policies can be attacked highlights the need for effective defenses. A natural defense is to fine-tune the victim zoo policy against an adversary, which we term single training. We also investigate dual training, randomly picking either an adversary or a zoo policy at the start of each episode. The training procedure otherwise the same as for adversaries, described in Section 4.2
We report on the win rates in You Shall Not Pass in Figure 4b We find both the single ZooSVI and dual 2ooDV1 fine-tuned victims are robust to adversary Advi, with the adversary win rate dropping from 87% to around 10%. However, ZooSV1 catastrophically forgets how to play against the normal opponent Zo001. The dual fine-tuned vietim ZooDVI fares better, with opponent Zoo01 winning only 57% of the time. However, this is still an increase from Zoo01's is 48% win rate against the original victim ZooV1. This suggests ZooV1 may use features that are helpful against a normal opponent but which are easily manipulable (Ilyas et al. 2019).
Although the fine-tuned victims are robust to the original adversarial policy Adv1, they are still vulnerable to our attack method. New adversaries AdvSl and AdvD1 trained against ZooSV1 and ZooDV1 win at equal or greater rates than before, and transfer successfully to the original victim. However, the new adversaries AdvS1 and AdvDl are qualitatively different, tripping the victim up by lying prone on the ground, whereas Adv1 causes ZooV1 to fall without ever touching it.
7 DISCUSSION
Contributions. Our paper makes three key contributions. First, we have proposed a novel threat model of natural adversarial observations produced by an adversarial policy taking actions in shared environment. Second, we demonstrate that adversarial policies exist in a range of zero-sum simulated roboties games against state-of-the-art victims trained via self-play be robust to adversaries. Third, we verify the adversarial policies win by confusing the vietim, not by learning generally strong policy. Specifically, we find the adversary induces highly off-distribution activations in the victim, and that victim performance increases when it is blind to the adversary's 's position.
Self-play. While it may at first appear unsurprising that policy trained as an adversary against another RL policy would be able to exploit it, we believe that this observation is highly significant. The policies we have attacked were explicitly trained via self- eelf-play to be robust. Although it is known that self-play with deep RL may not converge, or converge only to local rather than global Nash, self-play has been used with great success in a number of works focused on playing adversarial games directly against humans (Silver et al.,] 2018 OpenAl 2018). Our work shows that even apparently strong self-play policies can harbor serious but hard to find failure modes, demonstrating these theoretical limitations are practically relevant and highlighting the need for careful testing.
Our attack provides some amount of testing by constructively lower-bounding the exploitability of a victim policy- its performance against its worst-case opponent - by training an adversary. Since the victim's 's win rate declines against our adversarial policy, we can confirm that the victim and its self-play opponent were not in a global Nash. Notably we expect our attack to succeed even for policies in a local Nash, as the adversary is trained starting from random point that is likely outside the victim's attractive basin.
Defense. We implemented a simple defense: fine-tuning the victim against the adversary. We find our attack can be successfully reapplied beat this defense, suggesting adversarial policies are difficult to eliminate. However, the defense does appear to protect against attacks that rely on confusing the victim: the new adversarial policy is forced to instead trip the victim up. We therefore believe that scaling up this defense is promising direction for future work. In particular, we envisage a variant of population- based training where new agents are continually added to the pool to promote diversity, and agents train against fixed opponent for a prolonged period of time to avoid local equilibria.
Conclusion. Overall, we are excited about the implications the adversarial policy model has for the robustness, security and understanding of deep RL policies. Our results show the existence of previously unrecognized problem in deep RL, and we hope this work encourages other researchers to investigate this area further. Videos and other supplementary material are available online at 16/ and our source code is available on GitHub pettps:/ggitin com http://hhhh..../////aaa//nnnaaaa haai////veeraaeee
poo ici es
8

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
We thank Jakob Foerster, Matthew Rahtz, Dylan Hadfield-Menell, Catherine Olsson, Jan Leike, Rohin Shah, Victoria Krakovna, Daniel Filan, Steven Wang, Dawn Song, Sam Toyer and Dan Hendrycks for their suggestion and helpful feedback on earlier drafts of this paper. We thank Chris Northwood for assistance developing the website accompanying this paper. We are also grateful to our anonymous reviewers for valuable feedback and encouragement to explore defenses in this paper.
REFERENCES
J. Andrew Bagnell, Andrew Y. Ng. and Jeff G. Schneider. Solving uncertain Markov decision processes. Technical Report CMU-RI-TR-01-- August 2001.
David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech M. Czarnecki, Julien Pérolat, Max Jader- berg, and Thore Graepel. Open-ended learning in symmetric zero-sum games, arXiv:1901 1.00 106v [cs.LG], 2019.
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent com- plexity via multi-agent competition. In Proceedings of the International Conference Learning Representations (ICLR), 2018a.
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Source code and model weights for emergent complexity via multi-agent competition, 2018b. URL https
Osbert Bastani, Yewen Pu, and Armando
Verifiable reinforcement learning via policy extraction. In Advances in Neural Information Processing Systems (NeurIPS), pages 2018.
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret mini- mization. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. Proceedings of the Conference on Robot Learning (CoRL), volume 78, pages 1-16, 2017.
John Doyle, James A. Primbs, Benjamin Shapiro, and Vesna Nevistic. Nonlinear games: examples Proceedings of IEEE Conference on Decision and Control (CDC),
and counterexamples. volume 4, pages 3915-3920, 1996.
Shromona Ghosh, Felix Berkenkamp, Gireeja Ranade, Shaz Qadeer, and Ashish Kapoor. Verifying controllers against adversarial examples with Bayesian optimization. In IEEE International Conference Robotics and Automation (ICRA), pages 7306-7313, 2018. 2018.
Justin Gilmer, Ryan P. Adams, Ian Goodfellow, David Andersen, and George E. Dahl. Motivating the rules of the game for adversarial example research. 1rXiv:1807.067322 [cs.LG], 2018a. 8a.
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel Schoenholz, Maithra Raghu, Martin Wattenberg. and Ian Goodfellow. Adversarial spheres.
[cs.CV], 2018b.
IJJ. Good. On the principle of total evidence. The British Journal for the Philosophy of Science, 17 (4):319-321, 1967.
Ian Goodfellow, Nicolas Papernot, Sandy Huang. Yan Duan, Pieter Abbeel, and Jack Clark. Attacking machine learning with adversarial examples. https:://ppeaal http:/ppnaa..cooaa....o
research/ 2017.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable Baselines. https://githute comb
hill- 1al1-a/se t/tassstaae stable baselines: 2019.
Sandy H. Huang. Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Picter Abbeel. Adversarial attacks on neural network policies. arXiv: 10Xi:17002.02284 [cs.L.G], 2017.

---

Published as a conference paper at ICLR 2020
Andrew Iyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. urXiv1105.0021755 [stat.ML]. 2019.
Marc Khoury and Dylan Hadfelld-Menel. On the geometry of adversarial examples. arXiv:1811.00525w [es.LG], 2018.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv:1705.06452v1 [stat.ML], 2017.
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel. unified game-theoretic approach to multiagent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), pages #190-4203. 2017.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? End-to-end learning of negotiation dialogues. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.
Yen-Chen Lin, Zhang- Wei Hong, Yuan-Hong Liao, Meng- Li Shih, Ming Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 3756-3762, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research (JMLR), (N6v))2577-2665, 2008.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of thysically-pllussie perturbations. In Proceedings of the JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3932-3939, 2017.
Laura Noonan. JPMorgan develops robot to execute trades. Financial Times, July 2017.
OpenAI. OpenAl Five. httts:///blll http:///bolooo.......... opw.a cwm wwcoom/ppeai--fiiii
2018.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Ro- bust deep reinforcement learning with adversarial attacks. In Proceedings of the International Conference on Autonomous Agents and MultiAgent System (AAMAS), pages 2040-2042, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the International Conference on Machine Learning (ICML), volume 70, pages 2817-2826, 2017.
Hadi Ravanbakhsh and Sriram Sankaranarayanan. Robust controller synthesis of switched systems using counterexample guided framework. In Proceedings of the International Conference on Embedded Software (EMSOFT), pages 8:1-8:10, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv: 707.06347v2 [cs.LG], 2017.
Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? In Proceedings of the International Conference Learning Representations (ICLR), 2019.
Lloyd S. Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 39(10): 1095-1100, 1953.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419): 140--1144111144,,1 2018.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
10

---

Published as a conference paper at ICLR 2020
Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust MDPs using function approximation. In Proceedings of the International Conference on Machine Learning (ICML), pages 2014.
Jonathan Uesato, Brendan O' Donoghuu, Pushmeet Kohli, and Aaron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In Proceedings of the International Conference on Machine Learning (ICML), volume 80, pages 5025-5034, 2018
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.

---

Published as a conference paper at ICLR 2020
A
TRAINING: HYPERPARAMETERS AND COMPUTATIONAL INFRASTRUCTURE
Parameter
Value
Search Range
Search Distribution
Total Timesteps
20 x 10"
[0, 40 x 10%]
Manual
Batch size
16 384
2048, 65 536]
Log uniform
Number of environments
8
1, 16]
Manual
Mini-batches
1, 128]
Log uniform Uniform
Epochs per update
1,11]
Learning rate
3 x 10-4
[ x 10 10-5, x 10-2]
Log uniform
Discount
0.99
Maximum Gradient Norm
0.5
Clip Range
0.2
Advantage Estimation Discount
0.95
Entropy coefficient
0.0
Value Function Loss Coefficient
0.5
Table A.I: Hyperparameters for Proximal Policy Optimization.
Table A.1 specifies the hyperparameters used for training. The number of environments cho- sen for performance reasons after observing diminishing returns from using more than 8 parallel environments. The total timesteps was chosen by inspection after observing diminishing returns additional training. The batch size, mini-batches, epochs per update, entropy coefficient and learning rate were tuned a random search with 100 samples on two environments, Kick and Defend and Sumo Humans. All other hyperparameters are the defaults in the PPO2 implementation in Stable Baselines (Hill al. 2019).
We repeated the hyperparameter sweep for fine-tuning victim policies for the defense experiments, but obtained similar results. For simplicity, we therefore chose use the same hyperparameters throughout.
We used mixture of in-house and cloud infrastructure to perform these experiments. takes around 8 hours to train an adversary for single victim using 4 cores of an Intel Xeon Platinum 8000 (Skylake) processor.
B ACTIVATION ANALYSIS: T-SNE AND GMM
We collect activations from all feed forward layers of the victim's 's policy network. This gives two 64-1ength vectors, which concatenate into a single 128-dimension vector for analysis with Gaussian Mixture Model and a t-SNE representation.
B.1 T-SNE HYPERPARAMETER SELECTION
We fit models with perplexity 5, 10, 20, 50, 75, 100, 250 and 1000. We chose 250 since qualitatively it produced the clearest visualization of data with moderate number of distinct clusters.
B.2 GAUSSIAN MIXTURE MODEL HYPERPARAMETER SELECTION
We models with 5, 10, 20, 40 and 80 components with full (unrestricted) and diagonal covariance matrix. We used the Bayesian Information Criterion (BIC) and average log-likelihood on held- out validation set as criteria for selecting hyperparameters. We found 20 components with a full covariance matrix achieved the lowest BIC and highest validation log lilkiiikhhoo in the majority of enviromment-victim pairs, and was the runner-up in the remainder.
C FIGURES
Supplementary figures are provided on the subsequent pages.
12

---

Published as a conference paper at ICLR 2020
Adversary (Adv)
Normal (Zoo)
Random (Rand)
Zero
Kick and Defend 1
Kick and Defend 2
Kick and Defend 3
100
win rate (%)
75
50
25
Sumo Humans
Sumo Humans
Sumo Humans
100
Win rafe (%a
75
50
25
Sumo Ants
Sumo Ants
Sumo Ants
100
Win rafe (%))
75
50
25
Sumo Ants 4
You Shall Not Pass
100
75
Win rafe
50
25
Timestep
1e7
Timestep
1e7
Figure C.I: Win rates while training adversary Adv. The adversary exceeds baseline win rates against most victims Kick and Defend and You Shall Not Puss, is competitive on Sumo Humans, but performs poorly in the dimeenenioonl Sumo Ants environment. Key: The solid line shows the median win rate for Adv across 5 random seeds, with the shaded region representing the minimum and maximum. The win rate is smoothed with rolling average 100 000 timesteps. Baselines shown as horizontal dashed lines. Agents Rand and Zero take random and zero actions respectively. The Zoo baseline is whichever ZooM (Sumo) or ZooOM (other environments) agent achieves the highest win rate. The victim is ZooN (Sumo) or ZooVN (other environments), where N is given in the title above each figure.
13

---

Published as a conference paper at ICLR 2020
Opponent Win
Victim Win
Ties
ZooV1
79
53
50
32
90
42
30
30
21
46
50
67
10
57
64
61
0
0
0
6
9
ZooV2
93
80
63
50
46
46
37
90
20
36
45
44
0
0
5
10
80
ZooV3
26
68
67
60
ZooMV1
15
17
18
58
79
31
12
13
84
83
82
40
19
69
84
78
0
0
0
4
40
ZooMV2
-27
25
21
73
72
63
24
20
ZooMV3
-26
24
23
47
50
61
21
19
74
75
77
51
48
36
74
72
5
10
Adv1
Adv2
Adv3
ZooO1
Zoo03 ZooO2
Rand Zero
Adyl
Ady2
Ady3
ZooO1
ZooO2 Zoo03
Rand
Zero
Adv1
Adv2
Adv3
ZooO1
ZooO2
Zo003
Rand
Zero
(a) Kick and Defend
Opponent Win
Victim Win
Ties
100
ZooV1
87
81
76
48
3
13
19
24
52
97
99
0
0
ZooSVI
9
99
84
84
12
18
91
16
16
88
82
0
0
0
60
ZooDV1
11
91
88
57
8
89
12
43
92
94
0
40
ZooMV1
83
70
78
99
17
30
22
99
99
20
Adv1
AdvS1
ZaaO1 AdvD1
Rand
Zero
AdvS1
Zoo01 AdvD1
Rand
Zero
Advl
AdvS1
AdvD1
Zoo0z
Rand
Zero
(b) You Shall Not Pass
Opponent Win
Vietim Win
Ties
100
Zoo1
37
15
80
80
0
12
91
63
85
19
18
100 100
0
0
0
Zoo2
76
100100
80
Zoo3
10
31
57
0
1
81
79
23
88
34
100
2
33
0
0
60
ZooM1
93
91
92
12
76
100
40
ZooM2
32
39
46
84
69
0
0
39
30
23
50
10
17
100
100
29
32
38
14
0
0
20
ZooM3
62
68
75
66
82
90
30
20
10
31
13
99
99
12
15
0
0
Advl
Adv2
Adv3
Zoo1
Zao2
Zoo3
Rand
Zero
Adv1
Adv2
Adv3
Zoo1
Zoo2
Zoo3
Rand
Zem
Adv2
Adv3
Zool
Zoo2
Zoo3
Rand
Zero
(c) Sumo Humans
Opponent Win
Victim Win
Ties
100
Zoo1
-11
40 36 38
59
57
57
44
47
48
38
67
70
30
34
36
25
16
17
13
16
27
Zoo2
-11
40
40
37
37
66
15
18
31
80
Zo03 -5
34 42
66 62
57
41
42
67
37
25
19
17
16
28
Zoo4
-10
8
39
40
42
3
72
75
74
44
49
43
43
76
75
16 19
18
18
16
16
16
14
20 22
60
ZoeMI
6
5
62
51
58
61
2 4
32
33
28
31
19
22
20
16
48 35
62
62
66
64
19
28
22
23
50 61
40
ZooM2
4
4
61 51
54
56
3
18
17
16
22
19
18
20
16
32 15
79 81
74
20 31
26
29
66 82
ZooM3
52
56
14
20
17
76
25 29
20
ZooM4
62
56
59
65
3
13
14
9
12
17
12
24 10
83 82
86 82
26 28 25
73 87
Adv1
Adv2
Adv3
Adv4
Zool
Zoo2
Zoo3
Zoo4
Rand
Zero
Adv1
Ady2
Adv3
Adv4
Zool
Zno2
Zoo4
Rand
Zero
Advl
Adv2
Adv3
Adv4
Zool
Zoo2
Zan3
Zoo4
Rand
Zero
(d) Sumo Ants
Figure C.2: Percentage of episodes (out of 1000) won by the opponent, the victim tied. The maximal opponent win rate in each row is in red. Victims are on the y-axis and opponents on the r-axis. Key: Agents ZooYN are pre- pre-trained policies from Bansal et al. (2018a). where Y (VV', 'O', "} denotes the agent plays (V)ictim, (O)pponent or either side, and N Is random seed. Opponents AdvN are the best adversarial policy of 5 seeds trained against the corresponding Zoo [V] N. Agents Rand and Zero are baseline agents taking random and zero actions respectively. Defended victims ZooXYN, where X E {'S", *D', 'M', are derived from ZooYN by fine-tuning against a (S)ingle opponent AdvN, (D)ual opponents AdvN and Zoo [O] N. by (M)asking the observations.
14

---

Published as a conference paper at ICLR 2020
Adversary (Adv)
Normal (Zoo)
Random (Rand)
(a) Kick and Defend, victim
(b) Kick and Defend, victim 2
(c) Kick and Defend, victim
(d) Sumo Humans, victim
(c) Sumo Humans, victim 2
(f) Sumo Humans, victim 3
(g) Sumo Ants, victim
(h) Sumo Ants, victim
ii) Sumo Ants, victim
(i) Sumo Ants, victim
(k) You Shall Not Pass, victim
Figure C.3: t-SNE activations of the victim when playing against different opponents. There is a clear separation between the activations induced by Adv and those of the normal opponent Zoo. Model fitted with a perplexity of 250 to activations from 5000 timesteps against each opponent. The victim is ZooN (Sumo) or ZoeVN (other environments). where N is given in the figure caption. Opponent Adv is the best adversary trained against the victim. Opponent Zoo corresponds to ZooN (Sumo) or ZooON (other environments). See Figure C.4 for activations for a single opponent at time.
15

---

Published as a conference paper at ICLR 2020
Adversary (Adv)
Normal (Zoo)
Random (Rand)
(a) Kick and Defend.
(b) You Shall Not Pass.
(c) Sumo Humans.
(d) Sumo Ants.
Figure C.4: t-SNE activations of victim Zool (Sumo) ZooV1 (other environments). The results are the same as in Figure C.3 but decomposed into individual opponents for clarity. Model fitted with perplexity of 250 to activations from 5000 timesteps against each opponent. Opponent Adv the best adversary trained against the victim. Opponent Zoo Zool (Sumo) or Zoo01 (other environments). See Figure C.3 flror results for other victims (one plot per victim).
16


Reviews:
[{"review_id": "rJlKI8M49H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors start with the valid observation that for controlled systems the threat model of explicitly flipping pixels or otherwise changing the inputs directly is less relevant than the case of identifying bad inputs in the environments (which the authors rename \"natural observations\") that cause the controller under attack to leave its stable region and end up in bad states for itself, unable to recover sometimes. This paper is an empirical exploration of this phenomenon, using as examples trained agents in a Gym environment and involving two humanoid robot simulations interacting with each other in various ways. The original policies are taken from previous training, so the main experimental contribution here is to solve an RL problem on the attacker's side to try to find the bad regions in the victim's control space, going via the means of the observation-based coupling.\n\nI think these are timely issues. I agree that, to the extent that DRL policies are being seriously considered as candidates for various practical problems, we should seriously ask questions about the (lack of) stability of solutions. \n\nI also think that the paper would be stronger if the authors considered the rich history in control design, formal methods and other communities where this approach to counter-example based thinking is fairly standard and in particular where the use of optimization methods to find 'bugs' has been studied in some depth. \n\nSo, for instance, the authors give us useful quantification to support the observation that the process of attacker RL finds ways to put the victim outside its stable region where it is naturally unstable and ineffective. However, they do not observe that there is a literature on systematically performing 'controller testing' via optimization using various techniques. Here are just a few examples to show the diversity:\nGhosh, S., Berkenkamp, F., Ranade, G., Qadeer, S., & Kapoor, A. (2018, May). Verifying controllers against adversarial examples with Bayesian optimization. In 2018 IEEE International Conference on Robotics and Automation (ICRA) (pp. 7306-7313). IEEE.\nRavanbakhsh, H., & Sankaranarayanan, S. (2016, October). Robust controller synthesis of switched systems using counterexample guided framework. In 2016 international conference on embedded software (EMSOFT) (pp. 1-10). IEEE.\n\nThe authors might find it interesting to note that this approach of viewing the control problem 'backwards' to generate new instances has a deeper history in control theory (of course without discussion of NNs etc), e.g., \nDoyle, J., Primbs, J. A., Shapiro, B., & Nevistic, V. (1996, December). Nonlinear games: examples and counterexamples. In Proceedings of 35th IEEE Conference on Decision and Control (Vol. 4, pp. 3915-3920). IEEE.\n\nAll that said, I find the results presented here to be plausible and pointing in the desired direction for exploring how to robustify DRL. The authors defer training using these examples to future work but I think the paper would be more self-contained if that were actually demonstrated here already. For reasons I mention above, the existence of this kind of weakness in controllers is not really surprising to people who have thought deeply about control, and it is only to be expected that NN based parameterisations of control would only be even more vulnerable. So, the more satisfying result would be the positive one that shows how to train DRL to be (more) robust under such attacks.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Syxtou3uFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Adversarial Policies: Attacking Deep Reinforcement Learning\n\nIn this paper, the authors tackle the problem of adversarial attack by controlling the adversarial policy in reinforcement learning.\nI tend to vote for acceptance for this paper, but I also want to point out there is a lot of room for a better experiment section.\n\nPros: \n- In general, the proposed attacking scenarios is novel and missing from the current adversarial attack research, which mostly attack by adding constraint noise in the image space, instead of directly train a new adversarial controlling policy.\n\n- The problem is well formulated and the experiments are also sufficient to support the claim (could be better though, see Cons below).\nThe visualization provides useful insight for us to deepen understanding of the policy attack.\n\n\nCons:\n\n- Experiments limited in sumo cases.\nThe sumo experiment is quite difficult to train by itself and does not cover the board range of difficulty in multi-agent reinforcement learning tasks.\nTo be more specific, it will help the generalization of the paper by including\n1) Low-dimensional multi-agent environments. It will be even better if the input space is image and the output space is discrete.\n2) The study does not consider cooperative multi-agent games. It is also a very important and natural extension to the paper to consider the malicious attack in cooperative games, which can be equally common in real life.\n\n- No solution is given to the problem to combat policy attack.\nSome of the results of easy baselines such as iteratively playing attack and defense should be included in the current version of the paper.\nIt doesn’t have to be successful, but should be quite necessary to increase the research value of the paper.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SylJdDmKtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thank the authors for the response. The update looks good to me. The discussion section looks much better now.\n----------------------------------------\nSummary\nThis paper conducts research on adversarial policy against a fixed and black-box policy (victim). In this setting, the victim has a fixed policy but the adversary has no access to its white-box information. Instead, the adversary can freely access the black-box policy of the victim. The experiments show that the trained adversary outperforms the baseline in some scenarios, with three interesting findings: 1) the adversary successfully found the weakness of the victim. In some scenarios, the adversary wins by doing some weird actions which make no sense, but those weird actions somehow make the victim fail; 2) the victim fails due to the weird (or I should say adversarial) observation from the adversary but not the physical inference. The authors demonstrate this by showing that the victim has a much higher success rate when the observation is replaced with a ‘normal’ opponent. I am kind of on the borderline, but still lean to accept paper.\nStrengths\n- This paper presents some experiments on adversarial learning when the policy of the victim is fixed and black-box with interesting findings, demonstrates that the adversary can successfully figure out the weakness of the victim.\n- The paper formalizes the problem into an MDP whose dynamics is unknown, which is clear.\n- This paper comes with many experiments supported by great demos, which clearly support the authors’ arguments.\nWeaknesses\n- The paper becomes messy in the end and does not come with a good conclusion. With a lot of experiments conducted, the author fails to summarize them into an (or a few) interesting conclusion(s), but end up with a page-long conclusion paragraph, which makes the paper less focused but like listing miscellaneous experiments. From the reader’s perspective, a (or a few) clear conclusion would be very helpful.\nPossible Improvements\nMy suggestion is to reorganize the second half of the paper, to make a few clear arguments. Currently, the paper looks pretty narrative, and the author might easily get lost while reading the paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper demonstrates that for deep RL problems one can construct adversarial examples where the examples don't really need to be even better than the best opponent. Surprisingly, sometimes, the adversarial opponent is less capable than normal opponents which the victim plays successfully against, yet they can disrupt the policies. The authors present a physically realistic threat model and demonstrate that  adversarial policies can exist in this threat
model.

The reviewers agree with this paper presents results (proof of concept) that is "timely" and the RL community will benefit from this result. Based on reviewers comment, I recommend to accept this paper. 
Decision: Accept (Poster)