Optimal Attacks on Reinforcement Learning Policies
Authors: Alessio Russo
Alexandre Proutiere
Venue:  (2020)

Abstract: Control policies, trained using the Deep Reinforcement Learning, have been recently shown to be vulnerable to adversarial attacks introducing even very small perturbations to the policy input. The attacks proposed so far have been designed using heuristics, and build on existing adversarial example crafting techniques used to dupe classifiers in supervised learning. In contrast, this paper investigates the problem of devising optimal attacks, depending on a well-defined attacker's objective, e.g., to minimize the main agent average reward. When the policy and the system dynamics, as well as rewards, are known to the attacker, a scenario referred to as a white-box attack, designing optimal attacks amounts to solving a Markov Decision Process. For what we call black-box attacks, where neither the policy nor the system is known, optimal attacks can be trained using Reinforcement Learning techniques. Through  numerical experiments, we demonstrate the efficiency of our attacks compared to existing attacks (usually based on Gradient methods). We further quantify the potential impact of attacks and establish its connection to the smoothness of the policy under attack. Smooth policies are naturally less prone to attacks (this explains why Lipschitz policies, with respect to the state, are more resilient). Finally, we show that from the main agent perspective, the system uncertainties and the attacker can be modelled as a Partially Observable Markov Decision Process. We actually demonstrate that using Reinforcement Learning techniques tailored to POMDP (e.g. using Recurrent Neural Networks) leads to more resilient policies. 

Under review as a conference paper at ICLR 2020
OPTIMAL ATTACKS ON REINFORCEMENT LEARNING POLICIES
Anonymous authors Paper under double-blind review
ABSTRACT
Control policies, trained using the Deep Reinforcement Learning, have been re- cently shown be vulnerable adversarial attacks introducing even very small perturbations to the policy imput. The attacks proposed so far have been designed using heuristics, and build on existing adversarial example crafting techniques used to dupe classifiers in supervised learning. In contrast, this paper investigates the problem of devising optimal attacks, depending on a well-defined attacker's s objective, c.g., to minimize the main agent average reward. When the policy and the system dynamics, as well as rewards, are known to the attacker, a scenario referred white-box attack, designing optimal attacks amounts to solving Markov Decision Process. For what we call black-box attacks, where neither the policy nor the system is known, optimal attacks be trained using Reinforcement Learning techniques. Through numerical experiments, we demonstrate the efficiency of our attacks compared to existing attacks (usually based on Gradient methods). We further quantify the potential impact of attacks and establish its connection to the smoothness of the policy under attack. Smooth policies are naturally less prone to attacks (this explains why Lipschitz policies, with respect to the state, are more resilient). Finally, we show that from the main agent perspective, the system uncertainties and the attacker can be modelled as Partially Observable Markov Decision Process. We actually demonstrate that using Reinforcement Learning techniques tailored to POMDP (e.g. using Recurrent Neural Networks) leads more resilient policies.
INTRODUCTION
Advances in Deep Reinforcement Learning (RL) have made it possible to train end-to-end policies achieving superhuman performance
large variety of tasks, such as playing Atari games Mnih et al. (2013; 2016), playing Go Silver et al. (2016; 2017), as well as controlling systems with continuous state and action spaces Lillicrap et al. (2016); Schulman et al. (2015); Levine et al. (2016). Recently, some of these policies have been shown to be vulnerable to adversarial attacks at test time see e.g. Huang et al. (2017); Pattanaik et al. (2018). Even if these attacks only introduce small perturbations to the successive inputs of the policies, they can significantly impact the average collected reward by the agent.
So far, attacks on RL policies have been designed using heuristic techniques, based on gradient methods, essentially the Fast Gradient Sign Method (FGSM). As such, they do not explicitly aim at minimizing the reward collected by the agent. In contrast, in this paper. we investigate the problem of casting optimal attacks with well-defined objectives, for example minimizing the average reward collected by the agent. We believe that casting optimal attacks is crucial when assessing the robustness of RL policies, since ideally, the agent should learn and apply policies that resist any possible attack (of course with limited and reasonable amplitude).
For given policy learnt by the agent, we show that the problem of devising an optimal attack can be formulated agent's 's reward function and policy. We are mainly interested in black-box attacks: to devise them, the adversary only observes the variables from which she builds her reward. For example, if the objective is to lead the system to a certain set of (bad) states, the adversary may just observe the states the system evolves. If the goal is to minimize the cumulative reward collected by the agent, the
Markov Decision Process (MDP), defined through the system dynamics, the

---

Under review as a conference paper at ICLR 2020
adversary may just observe the system states and the instantaneous rewards collected by the agent. In black-box attacks, the aforementioned MDP is unknown, and optimal attacks can be trained using RL algorithms. The action selected by the adversary in this MDP corresponds to a perturbed state to be fed to the agent's policy. As a consequence, the action space is typically very large. To deal with this issue, optimal attack is trained using DDPG Lillicrap et al. (2016), Deep RI. algorithm initially tailored to continuous action space.
We train and evaluate optimal attacks for some OpenAl Gym Brockman al. (2016) environments with discrete action spaces (discrete MountainCar, Cartpole, and Pong), and continuous state-action spaces (continuous MountainCar and continuous LunarLander). As expected, optimal attacks outper- form existing attacks.
We discuss how the methods used to train RL policies impact their resilience to attacks. We show that the damages caused by attacks are upper bounded by quantity directly related to the smoothness of the policy under attack'. Hence, training methods such as DDPG leading to smooth policies should be more robust. Finally, we remark that when under attack, the agent faces environment with uncertain state feedback, and her sequential decision problem can be modelled as a Partially Observable MDP (POMDP). This suggests that training methods specifically tailored to POMDP (e.g. DRQN Hausknecht & Stone (2015)) result in more resistant policies. These observations are confirmed by experiments.
2 RELATED WORK
Adversarial attacks on RL policies have received some attention over the last couple of years. As for now, attacks concern mainly the system at test time, where the agent has trained a policy using some Deep RL algorithm (except for Behzadan & Munir (2017), which considers an attack during training). The design of these attacks Huang et al. (2017); Pattanaik et al. (2018); Lin et al. (2017) are inspired by the Fast Gradient Sign Method (FGSM) Goodfellow et al. (2015), which was originally used to fool Deep Learning classifiers. Fast Gradient Methods (FGMs) are gradient methods that changes the input with the aim of optimizing some objective function. This function, and its gradient, are sampled case of an agent trained using Deep RL, this function can be hence related to distribution of actions given state (the actor-network outputs a distribution over actions), or the Q-value function (the critic outputs upproximate Q-values). In Huang et al. (2017), FGM is used to minimize the probability of the main agent selecting the optimal action according to the agent. On the other hand, in Pattanaik et al. (2018), the authors develop a gradient method that maximizes the probability of taking the action with minimal Q-value. The aforementioned attacks are based on the umperturbed policy of the agent, or its Q-value function, and do not consider optimizing the attack over all possible trajectories. Therefore, they can be considered a first-order approximation of an optimal attack. If the adversary wishes to minimize the average cumulative reward of the agent, she needs to work on the perturbed agent policy (to assess the impact of an attack, we need to compute the rewards after the attack).
using observations:
Proactive and reactive defence methods have been proposed to make RL policies robust to gradient attacks. Proactive methods are essentially similar to those in supervised learning, i.e., they are based on injecting adversarial inputs during training Kos & Song (2017); Pattanaik et al. (2018). A few reactive methods have also been proposed Lin et al. (2017). There, the idea is to train a separate network predicting the next state. This network is then used to replace the state input by the predicted state if this input is believed to be too corrupted. But introduce partial observability, and the problem is not Markov anymore. It is known Singh et al. (1994) that deterministic memory-less policies are inadequate to deal with partial observability, and some kind of memory is needed to find a robust policy. In this work, we make use of recurrent neural networks show that we can use techniques from the literature of Partially Observable MDPs to robustify RL policies.
show in the paper, adversarial examples
It is finally worth mentioning that there has been some work studying the problem of robust RL with a thhe-thhoretical perspective Morimoto & Doya (2005); Pinto et al. (2017). However, these papers
A policy is smooth when the action it selects smoothly varies with the state. Refer to proposition 5.1 for details.
2

---

Under review as a conference paper at ICLR 2020
consider very different setting where the adversary has direct impact the system (not indirectly through the agent as our work). a sense, they are more related to RL for multi-agent systems.
PRELIMINARIES
This section provides a brief technical background on Markov Decision Process and Deep Reinforce- ment Learning. The notions and methods introduced here are used extensively in the remainder of the paper.
Markov Decision Process. A Markov Decision Process (MDP) defines discrete-time controlled Markov chain. It is defined by tuple M = (S, (A,, E S), P, Po- q) S is the finite state space, A, is the finite set of actions available in state s, P represents the system dynamics where P(-ss, a) is the distribution of the next state given that the current state is and the selected action is a, Po the distribution the initial state, and g characterizes rewards where q(-Is, a) the distribution of the collected reward in state is when action selected. We denote by r(s, a) the expectation of this reward. We assume that the average reward is bounded: for all (8, a), |r(8, a) S R. A As, t(als) denotes the probability of choosing action in state s. The value function V of a policy π defines for every the average cumulative discounted reward under when the system starts in state (s(s) IE 7"r(s7, az [s0 = =s] where sf and a7 are the state and the selected action at time / under policy and E (0, 1) is the discount factor. The Q-value function Q" of policy r maps a (state, action) pair (s, a) to the average cumulative discounted reward obtained starting in state when the first selected action is and subsequent actions are chosen according (=, a) = r(s, a) +( [V= (s')]. For given MDP M, the objective to devise a policy with maximal value in every state.
policy for M maps the state and to a distribution of the selected action. For
Deep Reinforcement Learning To train policies with maximal rewards for MPDs with large state action spaces, Deep Reinforeement Learning consists in parametrizing the set of policies some particular functions of interest such as the value function or the Q-value function of a given policy by deep neural network. In this paper, consider various Deep RI techniques train either the policy of the agent or the attack. These include:
DQN Mnih al. (2013): the network aims at approximating the Q-function of the optimal policy.
DDPG Lillicrap et al. (2016): an actor-critic method developed to deal with continuous state-action spaces. To this aim, the actor returns a single action rather than distribution over actions.
DRQN Hausknecht & Stone (2015): introduces memory in DQN by replacing the (post-convolutionaa) layer by recurrent L.TM. The use of this recurrent structure allows deal with partial observ- ability, and DRQN works better with POMDPs.
4 OPTIMAL ADVERSARIAL ATTACKS
To attack given policy trained by the main agent, adversary can slightly modify the sequence of inputs of this policy. The changes are imposed to small, according to some distance, so that the attack remains difficult to detect. The adversary aims at optimizing inputs with precise objective in mind, for example, to cause as much damage as possible to the agent.
We denote by M (A,, E S), S.,P.oo. q) the MDP solved by the agent, i.e., the agent policy has been trained for M.
4.1 THE ATTACK MDP
To attack the policy æ the adversary proceeds as follows. At time t, the adversary observes the system state 8t. She then selects perturbed state 5t, which becomes the input of the agent policy The agent hence chooses an action according to T (-)).). The adversary successively collects a random reward defined depending the action selected by the agent. An attack is defined by mapping ø -+ where ø = ((s) is the perturbed state given that the true state is 8.
her objective, which is assumed to be function of the true state and
3

---

Under review as a conference paper at ICLR 2020
Constrained perturbations. For the attack to be imperceptible. the input should be only slightly modified. Formally, we assume that we can define a notion of distance d(s, s') between two states s, E S. The state space of most RL problems can be seen case d is just the Fuclidean distance. We impose that given a state & of the system, the adversary can only select perturbed state in A; = (5 S d(s, 8) I c). es the maximal amplitude of the state perturbation.
subset of a Euclidean space, in which
System dynamics and agent's reward under attack. Given that the state 8, s at time t and that the adversary selects a modified state 8 E Ag, the agent selects an action according to the distribution TT-(s). Thus. the system state evolves to random state with distribution PT (-(s,, 3) = Eg ((1ss)) The agent instantancous reward is random variable with distribution E. (a(()))(())
Adversary's reward. The attack is shaped depending on the adversary's objective. The adversary typically defines her reward as a function of the true state and the action selected by the agent, or more concisely as direct function of the reward collected by the agent. The adversary might be interested in guiding the agent to some specific states. In control systems, the adversary may wish to induce oscillations in the system output or to reduce its controllability (this can be realized by choosing a reward equal to the energy spent by the agent, i.c., proportional to a² if the agent selects a). The most natural objective for the adversary is to minimize the average cumulative reward collected by the agent. In this case, the adversary would set her instantaneous reward equal to the opposite of the agent's reward.
We denote by qeh (-1s, 3) the distribution of the reward collected by the adversary in state s when she modifies the input to 8, and by F (s, 8) its expectation. For example, when the adversary wishes to minimize the agent's average cumulative reward, we have (*, 5) = - >.. [(a)8)((s) a).
We have shown that designing an optimal attack corresponds to identifying policy that solves the following MDP: MM = (S, (A4,, s E S), Pr ....P....qT. )"). When the parameters of this MDP are known to the adversary, a scenario referred to as white box attack, finding the optimal attack accounts to solving the MDP, e.g., by using classical methods (value or policy iteration). More realistically, the adversary may ignore the parameters of M, and only observe the state evolution and her successive instantaneous rewards. This scenario is called black-box attack, and in this case, the adversary can identify the optimal attack using Reinforcement Learning algorithms.
For ease of notation, we will denote the value of the adversarial policy e as V (for the MDP M even though it depends on æ. Finally, observe that when the adversarial attack policy ø, then the system dynamics and rewards correspond to scenario where the main agent applies the perturbed policy π o, which is defined by the distributions (л o (1(s) := Eg-o(s) [-155)] E S. Hence, we can denote the value of the perturbed policy as Vx00 (for the MDP M).
4.2 MINIMIZING AGENT'S S AVERAGE REWARD
Next, we give interesting properties of the MDP M When the adversary aims minimizing the average cumulative reward of the agent, then the reward collected by the agent can be considered as a cost by the adversary. In this case, y (s, 8) - Σ. π T(a 5)r(s, a), and one can easily relate the value function of attack policy ø to that of the agent policy o 6b
(s))) (
Vs S.
The vavalu function of an attack policy Φ can also be related to the Q-value function of the agent policy under attack π Φ. Indeed: for all 8, S.
(* " (s, ( (+))),
= - r(s,a) + >E, It) [ [s')] = -Ba-r [) @Q (s, a)).
In particular, the agent policy is deterministic (=(+) E A, the action selected under л in state s), we simply have:
Q -2ro (8, x5)),
V.
Equation (2) has an important consequence when we train an attack policy using Deep Learning algorithms. It implies that evaluating the Q-value of attack policy & can be done by evaluating
4

---

Under review as a conference paper at ICLR 2020
the Q-value function of the perturbed agent policy ó. In the case of off-policy learning, the former evaluation would require to store in the replay buffer experiences of the type (s, s, r) (r is the observed reward), whereas the latter just stores (s, (()),r) r) (but it requires to observe the actions of the agent). This simplifies considerably when the state space is much larger than the action space.
4.3 TRAINING OPTIMAL ATTACKS
Training an optimal attack can be very complex, since it corresponds to solving an MDP where the action space size is similar to that of the state space. In our experiments, we use two (potentially combined) techniques to deal with this issue: (i) feature extraction when this is at all possible, and (ii) specific Deep RL techniques tailored to very large (or even continuous) state and action spaces, such as DDPG.
Feature extraction. One may extract important features of the system admitting low-dimensional representation. Formally, this means that by using some expert knowledge about the system, one can identify a bijective mapping between the state s in S, and its features in Z of lower dimension. In general, this mapping can also be chosen such that its image of the ball A is easy to represent in Z (typically this image would also be ball around 2, the feature representation of s). It is then enough to work in 2. When 2 is of reasonable size, one may then rely on existing value-based techniques (e.g. DQN) to train the attack. An example where features can be easily extracted is the Atari game of pong: the system state is just represented by the positions of the ball and the two bars. Finally, we assume to be very likely that an adversary trying to disrupt the agent policy would conduct an advanced feature engineering work before casting her attack.
Deep RL: DDPG. In many systems, may not be easy extract interesting features. In that case, one may rely on Deep RL algorithms to train the attack. The main issue here is the size of the action space. This size prevents us to use DQN (that would require to build a network with as many outputs as the number of possible actions), but also srriietgr and actor-critic methods that parametrize randomized policies such as TRPO (indeed we can simply not maintain distributions over actions). This leads to use DDPG, actor-critic method that parametrizes deterministic policies. In DDPG, we consider policies parametrized by w (Ø., is the policy parametrized by w), and its approximated Q-value function Qo parametrized by 6. We also maintain two corresponding target networks. The objective function to maximize is J (w) = Egcpp [V (s)]. @ is updated using the following gradient Silver et (2014):
V (ws = E Exepsr V4 (s)) [w
where de corresponds to dw with additional exploration noise, and pa denotes the discounted state visitation distribution under the policy & In the case where the adversary's 's objective is to minimize the average reward of the agent, view of (2), the above gradient becomes:
(w) -E E [VaQo(s, क (()
(4)
Finally, ensure that the selected perturbation computed by dw belongs As. add last fixed layer to the network that projects onto the ball centred in 0, with radius e. This is done by applying the function + min 11, [ x. to x + e, where x is the output before the projection layer, and is the exploration noise. After this projection, we add the state s to get (s))).
The pscudo-code of the algorithm is provided in Algorithm in the Appendix. Algorithm 2 presents version of the algorithm using the gradient (4). A diagram of the actor and critic networks are also provided in Figure& 8 of the Appendix.
bradient-based exploration. In case the adversary knows the policy π of the main agent, we can leverage this knowledge to tune the exploration noise e. Similarly, to gradient methods Huang et al. (2017); Pattanaik al. (2018), we suggest using the quantity V [I (-1ss, y) improve the exploration process, where J is the cross-entropy loss between T-|ss) and one-hot vector y that encodes the action with minimum probability in state s. This allows exploring directions that might minimize the value of the unperturbed policy, which boosts the rate at which the optimal attack is exploration learnt. At time f, the exploration noise could be set to e,, a convex combination of a white exploration
noise e, and f fst) g(- V.J (=((-))), )),, where g is normalizing function. Following this gradient introduces bias, hence we randomly choose when to use e, or ej (more details can be found in the Appendix).
5

---

Under review as a conference paper at ICLR 2020
5 ROBUSTNESS OF POLICIES
In this section, we briefly discuss which training methods should lead to agent policies more resilient to attacks. We first quantify the maximal impact of attack on the cumulative reward of the agent policy, and show that it is connected to the smoothness of the agent policy. This connection indicates that the smoother the policy is, the more resilient it is to attacks. Next, we show that from the agent perspective, an attack induces a POMDP. This suggests that if training the agent policy using Deep RL algorithms tailored to POMDP yields more resilient policies.
Impact of an attack and policy smoothness. When the agent policy π is attacked using o, one may compute the cumulative reward gathered by the agent. Indeed, the system evolves as if the agent policy was π Φ. The corresponding value function satisfies the Bellman equation: (s))(())
Ea-rodt- r(ss, a) + P( [V">)(s")) Starting from this observation, we derive an upper bound on the impact of an attack (see the Appendix for a proof):
Proposition 5.1. For any E S, let (n (s) maxse.A; (+1())) (-13)||TV. We have:
||==- loo 2 | lao
(R + (1+77
(5)
Assume now that is smooth in the sense that, for all s, S. |||( T T(11s')l S Ld(ss s'), then
Le -| /Tod 0c 22 521 1- (R+ (R+
(6)
In the above, (s) quantifies the potential impact of applying a feasible perturbation to the state s on the distribution of actions selected by the agent in this state (note that decreases 0 as goes to 0). The proposition establishes the intuitive fact that when the agent policy is smooth (i.e., varies smoothly with the input), it should be more resilient to attacks. Indeed, in our experiments, policies trained using Deep RL methods known to lead to smooth policies (e.g. DDPG for RL problems with continuous state and action spaces) resist better to attacks.
Indueed POMDP and DRQN. When the agent policy is under the attack o, the agent perceives the true state s only through its perturbation (s)). More precisely, the agent observes a perturbed state only, with probability (s)ss) 1 (0())=3} (in general O could also be a distribution over states if the attack is randomized). Therefore, an optimall perturbation policy é induces a deterministic POMDP. It is known Spaan (2012), 2); Astrom (1965); Singh et al. (1994) that solving POMDPs requires "memory", i.e., policies whose decisions depend on the past observations (non-Markovian). Hence, we expect that training the agent policy using Deep RL algorithms tailored to POMDP produces more resilient policies. In Hausknecht & Stone (2015) they empirically demonstrate that recurrent controllers have certain degree of robustness against missing information, even when trained with full state information. We use this to pro-actively extract robust features, not biased by a specific adversary policy Φ) as in adversarial training. This is confirmed in our experiments, comparing the impact of attacks on policies trained by DRQN to those trained using Deep RI. algorithms without recurrent structure (without memory).
EXPERIMENTAL EVALUATION
We evaluate our attacks on four OpenAI Gym Brockman et al. (2016) environments (A: discrete MountainCar, B: Cartpole, C: continuous MountainCar, and D: continuous LunarLander), and on E: Pong, an Atari 2600 game in the Arcade Learning Environment Bellemare et al. (2013). Appendix, we also present the results of our attacks for the toy example of the Grid World problem, to illustrate why gradient-based attack are sub-optimal.
Agent's policies. The agent policies are trained using DQN or DRQN in case of discrete action spaces (environments A, B, and F), and using DDPG when the action space is continuous (environments C and D). In each environment and for each training algorithm, have obtained from 3 policies for DQN, for DRQN (achieving at least 95% of the performance of the best policy reported in OpenAl Gym).
All - I|zv is the Total Variation distance, and for any V R!S! o|V max.es [V()).

---

Under review as a conference paper at ICLR 2020
CETTTTACC 1210
Cralect matert EPOM
Sytnauzaah 0905
Sain dan C09
CAI
Performancc
Performance
abrkk
Figure I: Average performance loss vs. attack amplitude e: (top-left) discrete MountainCar, (top- right) Cartpole, (bottom-left) continuous MountainCar, (bottom-right) LunarLanddrr, The attack policy ø has been trained only for 0.05 In red is shown the FGM attack, in blue the attack proposed in this paper.
Adversarial attack. For each environment A, B, C, and D, we train one adversarial attack using DDPG against one of the agent's policies, and for a perturbation constraint 0.05 (we the same normalization method as in Huang et al. (2017)). For results presented below, we use uniformly distributed noise; results with gradient-based noise discussed in the Appendix. To get attacks for other values of E, we do not retrain our attack but only change the projection layer in our network. For environment
represent the state as 4-dimensional feature vector, and train the attack using DQN in the feature space. The full experimental setup is presented in the Appendix.
6.1 OPTIMAL VS. GRADIENT-BASER ATTACKS
We compare our attacks to the FGM attacks (gradient-basedl proposed Pattanaik al. (2018) (we found that these are the most efficient). To test a particular attack, we run it against the 3 agent's policies, using random seeds (c.g. involved the state initialization), and 30 episodes. We hence generate up 1800 episodes, and average the cumulative reward of the agent.
In Figure 1, we plot the performance loss for different the perturbation amplitudes in the environments. Observe that the optimal attack consistently outperforms the gradient-based attack, and significantly impact the performance attack for 0.05, it seems that this attack generalizes well for various values of e. In Appendix, we present similar plots but not averaged over the agent's policies, and we do not see any real difference suggesting that attacks generalize well to different agent's policies.
the agent. Also note that since we have trained our
From Figure! 1, for environments with discrete action spaces (the top two sets of curves), the resilience of policies trained by DRQN confirmed: these policies harder attack.
Policies trained using DDPG for environments with continuous action spaces (the bottom two sets of curves Figure 1) more difficult perturb than those trained in environments with discrete action spaces. The gradient-basee attack does not seem to be efficient at all, at least in the range of attack amplitude и considered. In the case of LunarLander some state components were not normalized, which may explain why bigger value of is needed in order to see major performance loss (at least ε 0.1). Our optimal attack performs better, but the impact the attack not as significant as in environments with discrete action spaces: For instance, for discrete and continuous MountainCar, our attack with s = 0.07 decreases the performance of the agent by 30% and 13%, respectively.
6.2 ATTACK ON PONG, BASED FEATURE EXTRACTION
We consider the game of Pong where the state is 84x84 pixel image. The agent's policies were trained using DQN using as inputs the successive images. As for the attack, we extracted a 4- dimensional feature vector fully representing the state: this vector encodes the position of the centre
7

---

Under review as a conference paper at ICLR 2020
Unperturbed ituage
Perturbed imagee
Unperturbed image
Perturbed image
Training attack Comedce roward
corscce reword
Original eolore
Afversari adtors
Original adkees
Adbersakal cokore
-28
Nummbbr
splaodes
Artivm
Figure 2: Game of Pong. (Left part) Images and corresponding agent's action distributions before and after the attack. (Right part) Average agent's reward at different phases of the attack training.
of mass of the three relevant objects in the image (the ball and the two bars). Extracting the features is done using a high pass filter to detect the contours. We need 4-dimensional vectors since the two bars have a single degree of freedom, and the ball has two.
We limit the amplitude of the attack in the feature space directly. More precisely, we impose that the adversary just changes one of the 4 components of the feature vector by one pixel only. For example, the adversary can move up one bar by one pixel. We found that the amplitude of the resulting attack in the true state space (full images) corresponds to E = 0.013.
The attack is trained using DQN in the feature space. In Figure 2 (left part), present few true and perturbed images, and below the corresponding distributions of the actions selected by the agent without and with the attack (the probabilities softmax function with temperature T = 1). Moving objects by one pixel can perturb considerably the agent's policy. In Figure (right part), we show the evolution of the performance of the agent's policy under our attack during the training of the latter. Initially, the agent's 's policy achieves the maximum score, but after 500 episodes of training, the agent's score has become close the lowest possible score in this game (-20). This score can be reached in an even fewer number of episodes when tuning the training parameters. The gradient-based attacks performed in Huang et al. (2017) cannot be directly compared to ours, since these attacks change 4 frames at every time step, while ours modifies only the last observation. They also modify the values of every pixel in frames (using 32-bit precision over the reals [O.1]), whereas we change only the values of just a few (using 8-bit precision over the integers [0,255]).
calculated from the output of the network using
CONCLUSION
In this paper we have formulated the problem of devising an optimal black-box attack that does not require access to the underlying policy of the main agent. Previous attacks, such as FGM Huang et al. (2017), make of white-box assumptions, i.e., knowing the action-value function Q, the policy, of the main agent. In our formulation, we do not assume this knowledge. Deriving an optimal attack important in order to understand how build RL policies robust adversarial perturbations.
The problem has been formulated as Reinforcement Learning problem, where the goal of the adversary encapsulated in the adversarial reward function. The problem becomes intractable when we step out of toy problems: we propose variation of DDPG to compute the optimal attack. In the white-box case, instead of using FGM, we propose to use a gradient-based method improve exploration.
Adapting to such attacks requires solving a Partially Observable MDP, hence we have to resort to non-markovian policies Singh et al. (1994). It can be achieved by adopting recurrent layers, in DRQN Hausknecht Stone (2015). We also show that Lipschitz policies have desirable robustness properties.
We validated our algorithm on different environments, In all cases we have found out that our attack outperforms gradient methods. This is more evident in discrete action spaces, whilst in continuous spaces is more difficult to perturb for small values of e, which may be explained by the bound we provide for Lipschitz policies. Finally, policies that memory seem be more robust in general.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Karl Astrom. Optimal control of markov processes with incomplete state information. Journal of mathematical analysis and applications, 10(1):174-205, 1965.
Vahid Behzadan and Arslan Munir. Vulnerability deep reinforcement learning to policy induction attacks. In International Conference Machine Learning and Data Mining in Pattern Recognition, pp. 262-275. Springer, 2017.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint wxxvv1606.001500. 2016.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference Learning Representations, 2015.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposium Series, 2015.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Picter Abbeel. Adversarial attacks on neural network policies. Workshop Track of the 5th International Conference on Learning Representations, Toulon, France, 2017.
Jernej Kos and Dawn Song, Delving into adversarial attacks on deep policies. Workshop Track of the 5th International Conference on Learning Representations, Toulon, France, 2017.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1333 2016.
Timothy P Lillicrap. Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations, San Juan, Puerto Rico, 2016.
Yen-Chen Lin, Zhang-Was Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. Proceedings of the Twenty- Sixth International Joint Conference on Artificial Intelligence, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. NIPS Deep Learning Workshop. 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335-359 2005.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In Proceedings of the 17th International Con- ference on Autonomous Agents and MultiAgent Systems, pp. 2040-2042, International Foundation for Autonomous Agents and Multiagent Systems, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Eitarnin-VVolumee 70, pp. 2817-2826. JMLR. org, 2017.
Matthias Plappert. keras-rl. https:/ / / github. www./eeras-rl//eeeaea..
2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization, In International Conference on Machine Learning, pp. 1889-1897, 2015.

---

Under review as a conference paper at ICLR 2020
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershelvam,, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):4444889 2016. doi: 10. (0.1008/naturre699
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354 EP Oct 2017. URL https:
//doi. 1ttt:a/oo.....//ttteteeeeee222
Satinder Singh, Tommi Jaakkola, and Michael I Jordan. Learning without state-estimation in partially observable markovian decision processes. Machine Learning Proceedings 1994, pp. 284 292. Elsevier, 1994.
Matthijs TJ Spaan. Partially observable markov decision processes. In Reinforcement Learning, PP- 387-414. Springer, 2012.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 OTHER EXPERIMENTAL RESULTS
Gridworld. We use as toy-example x gridworld, with distance, and perturbation magnitude = 1.0. The goal of the game is to reach the top-left or bottom-right corner in the fewest number of steps. with reward inspired by Huang al. Huang et al. (2017) and Pattanaik al. Pattanaik et al. (2018), and compared them with our proposed method to find the optimal perturbation policy Ø. The implementation of those algorithms is shown in Algorithm 3, 4. In figure 3 is shown the undiscounted value of the various attacks: we can see the value the worst attack defined in Pattanaik et al. (2018) does not achieve the same values that are achieved from the optimal perturbation ø.
for each step spent in the game. We implemented both the gradient methods
-00
-00
-00
-20
-00
-00
100
2
3
4
100
11
-10
3
3
Figure 3: Un-discounted value of the perturbed policy of the main agent. Left: our proposed attack, middle: attack in Huang et al. (2017), right: attack in Pattanaik et al. (2018).
OpenAl Gym Environments. Figure presents the results for the discrete MountainCar environment A. There, we plot the distribution of the performance loss (in %) due to the attack for 0.05, when the agent policies have been trained using DQN (top) or DRQN (bottom).
Meen: -28.5%
Mean: 11.55%
No attack
5
0
-00 00%00%%
00.000%
-20 00%
0.00%
20.00%
method
Mean: 8%
Mean: -8.2%
Denstr
attack
10
0.00%
Figure 4: Performance loss distribution under the optimal and gradient- attacks with amplitude E 0.05 for the discrete MountainCar environment. The attacks against policies trained using DQN (top) and DRQN (bottom).
11

---

Under review as a conference paper at ICLR 2020
Gradient method Opkmal attack No attack
Mean: -14.5%
Density
Mean: -99.%%
-00.00%
00.00%%
-20 00%
0.00%
20.00%
0.05 DRON
Mean: -16. 8%
Mean: -8.2%
attack No attack
Density
-60.00%
00%
20, 00%0
0.00%
20.00%
Performance loss
Figure 5: Mountaincar environment distribution of the average discounted reward performance decrease for 0.05, with respect to the main agent's policy the attacker has trained for.
Me-nn:166.
Kean: -16.78%
Meain
Mean:
Mean -13.79 13%79%
-14
Mean:
33.8%%
Crudient
Grud en
20
10.0%
20.00%
-55
D0%
10%9
1..00%
0.00%
000%
25.02%
-15 5..00%
3..0%%
6.00%
5.00%
Performa nce loss
Performanc e loss
Figure 6: Cartpole environment distribution of the average discounted reward performance decrease for = 0.05. On the left we show the distribution with respect to the main agent's policy the attacker was trained on, and on the right average across all the main agent's policies.
In Figure 5-8 show the performance loss distribution for 0.05, and in Figure 9 the average performance loss for different values of e, all computed when the adversary attacks the agent's policy was trained for. Additionally, for environment B. D. we show the loss distribution averaged across all the main agent's 's policies.
Meean: -1 10.4%
Mean: -] 0%6
Mean: -11.0%
Mean: +] 556
12
13
33.00%
11.00%
30.00%
5.00%
6.0%%
5.00%
10 00%
20.00%
33..00%
-20.00%
3,00%
6.00%
5.00%
20 10.0%%
Performance
Performance
Figure 7: Continuous MountainCar environment distribution of the average discounted reward performance decrease for 0.05. On the left show the distribution with respect to the main agent's 's policy the attacker was trained on, and on the right an average across all the main agent's policies.
12

---

Under review as a conference paper at ICLR 2020
2.00
attack to sttck
0.4%G
attack sttck
1.75
Meem: +1
-11
1.25
Demiit
1.00
1.00
0.75
0.25
0.00
0.00
Performance loss
Performance loss
Figure 8: LunarLander environment distribution of the average discounted reward performance decrease for = = 0.05. On the left we show the distribution with respect to the main agent's policy the attacker trained on, and the right average across all the main agent's 's policies.
59
-0%
Perforannee loss.
Performance loss
13%
30%
Gracient method- DON
-25%
Gradient method DON
Gradient method - DRON
Gradient method DRQN
-50%
Optimal attack
Optimal
Optimal attack DRON
-35%
0.06
0.01
0.02
0.03
0.04
0.05
0.08
0.07
1376
Performance loss
Performance loss
5%
-5%
-15%
Gradient method - DDPG
Gradient method DDPC
Optimal
-20%
-25%
0.02
0.03
0.05
0.06
Figure 9: Average performance loss attack amplitude in the various environments, when the adversary attacks the main agent's policy it was trained on. (top-left) discrete MountainCar, (top-right) Cartpole, (bottom-left) continuous MountainCar, (bottom-right) LunarLander. Attacks against using policies obtained using DQN, DRQN, or DDPG.
Gradient based exploration Gradient-based exploration tries to exploit knowledge of the unper- turbed policy и of the main agent. Caution should be taken, in order reduce bias, and minimize the risk of exploring like an on-policy method. If we follow the information contained in without restriction we would end up in sub-optimal solution, that minimizes the value of the unperturbed policy The exploration noise is given by the following equation:
Ct.
X, = 0,
wije + (1
v)),
X,
where g a normalizing function such as g(x) - or ( (2). Xxe is Bernoulli random variable with P(X, = 1) = p. and w, max, (a (s) min, (a)ss)) J is the cross cross-entropy loss between (a(s:) and one-hot vector y that encodes the action with minimum probability in state 8t-
13

---

Under review as a conference paper at ICLR 2020
If is not known, but the action-value function Q" is known, compute s(-[s) by applying the softmax function on the action-valuee of a particular state s.
Training the attack
10.00%
WN
20.00N
Reward statisties
No attack
FGM
Uniform exploration
Gradient-hasee exploration
Mcan
103.4
134.6
163.2
173.6
C0.0056
M
Std
9.79
29.97
Min
-81
-85
Max
-115
200
-200
-200
70.00%
1.00
300
Number of
Figure 10: Mountainear environment. On the left shown the average episode loss when exploring using a gradient-basse based exploration methods vs. uniform noise exploration. On the right are shown statistics regarding episodes reward for & = 0.05, and norm E,, after training.
The gradient of the cross-entropy loss computes the direction that increases the probability of taking the action with minimum value when the policy æ is unperturbed. The random variable X,, instead, is necessary in order reduce bias induced by following this gradient, while wt quantifies how strongly an agent prefers the optimal action in state 8,. Intuitively, in case π is an optimal policy, the lower wt is, the less impact the optimal action has in that state. This guarantees that we follow the gradient of J mostly when we are in critical states. In Figure 10 we show the low-past filtered loss during the training of the attack. We clearly see an improvement in performance, although highly depends on the selection of the parameters.
We have tested this type of exploration on Mountaincar, with p 0.35, and temperature of the softmax function set to 1. The exploration noise is drawn from a Uniform distribution with parameters shown Table 3 (Mountaincar). In Figure 10 we compare gradient-based exploration vs. uniform noise exploration. On the left, we show the low-pass filtered loss during the training of the attack, with æ = 0.05 and & norm. On the right of Figure 10 we compare episodes reward statistics after training. Results in the table were averaged across 5 different seeds, with 100 episodes for each seed, for a total of 500 episodes for each method. We see an improvement in performance, although it highly depends on the selection of the parameters.
A.2 PROOF OF PROPOSITION 5.1
Proof. Let
g" (s,aa) = Eg ((ls))) (s"(s') and (s,a) = (( (1 (s)).
(8)
We will now proceed by bounding the quantity I(V* - (s))
I(V* - (s)] (asa(1 + (s, a)] Bunnro((l) (r + 7g (0 (s,
(9)
E5-0(11)) (1rrs ng (s, E Ea + xg (s.
(10)
where in the last line made of the following equality E E Eg Ea (-1)) [s] fl] Now, by making use of [[ < E[[z]] we get:
I(V" (s) Es-0(1=)) (a-r(ie)(((ss)) 2g (s. E 7g T0
a)]
(11)
S Eand())) Banr( 7g - 7 ro
(s,
(12)
14

---

Under review as a conference paper at ICLR 2020
At this point consider random variable X that assume values only in bounded set K. Let Xoc be a bound on the absolute value of X. Then, for any two distributions f1. f2 whose support is K we have 5 [X] 1 [X]] I 20 Ilf f2llTv:
1 1 [X]] = Σ (x) Σas(s))
Σ [((()))(((() = 2X 11-//22/TV xEK
øEK
ØEK
(13)
By using the previous inequality we can bound the reward difference:
1- (aort)))) (1)x((1)(r(ss000 ()]s, S 2 -
(14)
I (
(15)
where (6) = max [1((8) (+)]lrv. The inequality becomes
I(V - (s)] S (x) R (l)))) (o
(a) (s.
(16)
Now, let (8, a) = ( (s, a) + A (s,
(s, a). and observe that (8, a) is uniformly bounded by V V bo. and A (s, by V yloollos lloo thus
E (1)))
Bu [g (s, a) Ea l n00 (s,
(17)
E3 (110)
BaN (s, a)] - Ea (8) 1g (8, - A(s
(18)
<E (1ss) ( + -|V (Tool (s) + -IV - Vrod
(19)
Hence, we can deduce the following inequality:
I(V - (x)00)()) S 2a (8) 27 ( + +7IVV= - Vxod 1oos
(20)
= 20 (8) + * - |1
(21)
By taking the term y| |xoo|| supremum over s E S we obtain the result:
so on the left hand side, dividing by 1 - y, and finally taking the
V - 1oo 2 || (R + (R+vV=). 1-7 7
(22)
Assume now the main agent's policy is smooth, such that ||=(() (s') S Ld(s, s'), Rzo- We can now bound the term Bz=Ø(-le) [E) [[) a)] - EE (5) [r(s, a)ll as follows:
a)] Ea (r(8, a)]l < 1) [2 |2) ((s) (+)]lv]],
(23)
S 22LRE (. ()).
(24)
2LRd(ss,
(25)
where d(ss) = Eg-o(- [d(s, 5)] denotes the average weighted distance between elements of X and 8, according to the distribution (-[ss). Equivalently we also have
Ezddl a) Ea (s, a]] Ee to a)] s E Esdd(()s)
2V ((()) +())llTV
(26)
+ +1 11
(27)
$ 2LV3(d(s) + -|V - vTod
(28)
from which we obtain
I(V" - [s)) < 2LR((s) +>2Ld( + ynoo
(29)
2Ld(( ( R マ + - vTo0 100,
(30)
and
- - | < 2sup Ld(ss)
L8
(R + 7 V 22 1-
(R + +V=)
(31)
sES
1 7
15

---

Under review as a conference paper at ICLR 2020
A.2.1 ALGORITHMS
Attack training algorithm. In this section we describe the DDPG algorithms discussed in Section 4.3. We will denote the projection layer and the output before the projection layer respectively by In and d.,t- Algorithm refers to the black-box attack, whilst Algorithm 2 makes use of if it is known. Apart from the projection layer, and the change in the gradient step in the white-box case, both algorithms follow the same logic the usual DDPG algorithm, explained in Lillicrap et al. (2016), Moreover, projections have a regularization term À in order to prevent division by
Algorithm Training adversarial attack
1: procedure ATTACK TRANIING(T Tiasiingg NB)
2: Initialize critic and actor network Qe: ow with weights and é.
3: Initialize target critic and actor networks Q0-, P0- with weights 0 t- 0, t &..
4: Initialize replay buffer D.
5:
for episode 1:M do Initialize environment and observe state so, and initial time 0. 7: repeat
6:
8:
Select a perturbation and add exploration noise & +- 8, + (n et). Feed perturbation & to main agent and observe reward and state Fr, S1+1. Append transition (St, St, FI, St+ 1) to D.
9:
10:
11:
if = 0 mod Tiininng then Sample random minibatch ~ D of NB elements.
12:
13:
Set target for the i-th element of B to y: F, + 7Q0 (s141,00 (81+1)).
14:
Update critic by minimizing loss 1 Σ (y/= - Q0(s;, 3;)) Ng Update actor using sampled policy gradient
15:
Ng
3)
NB
16:
Slowly update the target networks:
0 - (1- T0,
+ -) TΦ.
17:
end if
18:
1 + 1 ++1.
until episode terminal.
20:
end for
end procedure
A.2.2 GRADIENT-BASED ATTACKS.
Gradient- based attacks can be adapted to MDPs by solving the optimization problem these methods are trying to solve. There are two main techniques that were developed, one by Huang et al. Huang et al. (2017), and the other one from Pattanaik et al. Pattanaik al. (2018): the former tries to minimize the probability of the optimal action ax, whilst the latter maximizes the probability of taking the worst action, a., defined the action that attains the minimum value for the unperturbed policy и. Pseudo-code for the former is given Algorithm 3, while for the latter is in Algorithm 4.
A.3 EXPERIMENTAL SETUP
Hardware and software setup. All experiments were executed on a stationary desktop computer, featuring Intel Xeon Silver 40 CPU, 48GB of RAM and GeForce GTX 1080 graphical card. Ubuntu 18.04 was installed on the computer, together with Tensorflow 1.13.1 and CUDA 10.0.
Deep-learning framework. We set up experiments within the Keras-RL framework Plappert (2016), together with OpenAl Gym Brockman et al. (2016) as interface to the test environments used in this paper.
16

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Training adversarial attack 2
1: procedure ATTACKTRAININGZA) RAINING2( Tiainingg Nn)
2: Initialize critic network Qo and actor tw with weights 0 and ø.
3: Initialize target critic and actor networks Q Per with weights 0- + 0, +0 t 0.
4: Initialize replay buffer 2.
5:
for episode 1:M do Initialize environment and observe state 80- 7: Set initial time f <- 0. repeat
6:
8:
9:
Select perturbation and add exploration noise 5, +- 8 + In (Zg,t ex). Feed perturbation & to main agent and observe reward and state Fr 84+1. Append transition (st, ag, Ft, St+1) to D, where a, = (T:))
10:
12:
if't = mod Tuaining then Sample random minibatch Set target for the i-th element of B to y/ Fi 7Q0- (8141, T (8i+1))).
D of NB elements.
14:
Update critic by minimizing loss No Update actor using sampled policy gradient
1 Σ (ys + Qo(s;, a;))?
NB V)
V.J=
NB
o(x))
Slowly update the target networks:
0- --- +70,
- (1 -r) TQ.
18:
end if
19:
1 <- +++1
20:
until episode is terminal.
21:
end for
end procedure
Algorithm 3 Adversarial Attack Adaptation of Huang et al. (2017).
I: procedure ATTACCKROCEDUREI(,,,,.AA
2: a' + arg max, (a)s))
bi Best action in state
3: 368
4: for all o' E X do
Loop trough the e-neighbor states Evaluate if a² not optimal in
if Va
a(a'ls) then
6:
8 s+4
end if 8: end for
7:
return
10:
end procedure
Algorithm 4 Adversarial Attack Adaptation of Pattanaik et al. (2018).
1: procedure ATTAACKPOCEDURR(6, m, x5)
2: 9"' + Q (x arg max, (als))
3:
4: for all s' X do
Loop trough the e-neighbor states
q + Q" (s, arg max N (alss)))
6:
if < q- then
Evaluate if optimal action in s' is worse in
7:
8:
end if 10: end for
11: return
12:
end procedure
17

---

Under review as a conference paper at ICLR 2020
All the adversarial methods used in this paper, including Gradient attacks, and DRQN, have been implemented from scratch using the Keras-RL framework. All the code and experiments can be found at the following url https//bbi.ly2JJA7CCC.
Neural network architectures.
different neural network architecture, shown in table The network architecture was not optimised, but was chosen large enough also to have a better comparison with Gradient methods since they depend on the dimensionality of the network, and may fail for small-sized networks. For all environments we set the frame skip to 4. For the adversary,
Depending the environment
Neural network
MountainCar DQN
MountainCar
Lunarl ander
Cartpole DQN FC(256, ReLU) FC(266, ReLU) FC(2)
structure
Actor
Q-Critic
Actor
Q-Critic
Layer
FC(512, ReLU)
FC/400, ReLU)
FC(400, RcLU)
FC(400, ReLU)
FC(400, RcLU)
Layer
FC(256, ReLU)
FC ReLU)
FC(300, RcLU)
FC(300, ReLU)
FC(300, ReLU)
Layer
FC(64, ReLU)
FC(1, Tanh)
RC(1)
FC(2, Tanh)
FC(1)
Layer 4
FC(3)
Table 1: Neural network structure for the policies of the main agent. FC stands for Fully Connected layer. Inside the parentheses is indicated how many units were used, and the activation function for that layer.
instead, we used different actor-critic architecture, although we have not tried to optimise it. The model can be seen figure 11. In the figure (S) denotes the dimensionality of the state space. The neural network structure for the game of Pong is the same the one used in Huang et al. (2017), for both the adversary and the main agent.
Actor Network
Critic Network
State
(Perturbed observation, State)
Fully Connected Layer 512 units
Fully Counected Layer 512 units
ReLU
ReLU
Fully Connected Layer 256 units
Fully Connected Layer 256 units
ReLU
ReLU
Fully Connected Layer 128 units
Fully Connected Layer 128 units
BeLU
ReLU
Fully Connected Layer dim (S) units
Fully Connected Layer unit
Tanh
Fxplomation Notse
Q value
Projection layer
Perturbed obscrvation
Figure 11: Diagram of the adversarial agent perturbing observations of the environment. On the right is shown the environment model for the malicious agent. The equivalent MDP is denoted by M.
Data collection and preprocessing, Every data point is an average across 10 different seeds, where for each seed we have tested an attack against all of the main agent's policies, and each test considers 30 episodes (10 in the case of FGSM for Continuous Mountaincar and
In Table 2
18

---

Under review as a conference paper at ICLR 2020
is shown the number of policies for the main agent and the number of episodes collected. Density plots were preprocessed using the function eeabor..deplo. with Gaussian kernel and bandwidth 0.05 (0.15 for LunarLander). Gradient-bast exploration data preprocessed using first order acausal Butterworth filter with critical frequency wn 0.1.
Algorithm/Enviroonment
MountainCar
MountainCar-Continuuur
Cartpole
LunarLanndr
DQN DRQN
3 (1800)
3 (1800)
(600)
(600)
DDPG
3 (1200)
3 (1200)
Table 2:
Number of policies for the main agent for each Algorithm/Ennironmeen pair. In parenthesis shown the total number of episodes taken.
Settings for gradient-attacks. For gradient-attack, we have always used the same norm as the one used by the optimal attack. For the Softmax layer we have always used a temperature of = 1. We compute the gradient using the method from Pattanaik al. Pattanaik et al. (2018), which has proven to be better than the one proposed by Huang et Huang et al. (2017). In order to speed-up the computation we do not sample the attack magnitude from random distribution, but fix it to the maximum value &.
Training procedure and parameters. All the training parameters for the main agent shown in table 3. Parameters were chosen based on previous experience, so that the policy trains in a fixed number of training steps. Policies were trained until they were within 5% of the optimal value referenced in the OpenAl Gym documentation Brockman et al. (2016), across an average of the last 100 episodes. For the game of Pong use same as Huang al. (2017)
MountainCar
MountainCar
Lunarl lander
Cartpole
Pong
Action space Algorithm used
Discrete DQN/DRQN
Continuous DDPG
Continuous
Discrete DON/DRQN
Discrete
DDPG
DQN
Reference value
50.0
90.0
200.0
195.0
20
Number of steps
4 10
2 2-10"
3-10
22 10
4 10"
Number of warmup steps
100
10a
3 3-10
10a
5 10*
Replay memory size
4 10
2- 10
15.
22 10
5 105
Discount factor
0.99
0.99
0.99
0.99
0.99
Leaminng rate
10-3
(10-4 10 10-8)
(10-4, 1,100
10-4
25 10-4
Optimizer
Adam
Adam
Adam
Adam
Adam
Gradient clipping
Not set
1.0
1.0
1.0
Not set
Exploration style
egreedy
Gaussian noise
Gaussian noise
e-greddy
e-greddy
Initial exploration rate/ Std (DDPG)
0.95
2
1 10-3
0.95
0.95
Final exploration rate/ Std (DDPG)
0.1
10-3
0.1
0.05
Exploration steps
28 10
14 10
21 10
11 11-10
35 10
Batch size
32
64
192
256
32
Target model update
10a
10-3
10-3
200
10*
Initial random steps Training
0
U
Initial random steps Testing Frame-skip
10
10
50
Table 3: Training settings for the policies of the main agent. When using DDPG we sometimes used different parameters for the actor and the critic. In that case in the table shown 2 parameters, the first one for the actor, the second one for the critic.
For the adversary, used similar parameters, shown in Table 4.
19

---

Under review as a conference paper at ICLR 2020
MountainCar Continuous
ParametersFnnvinnmment
MountainCar
Lunarl Lndder
Caripole
Pong
Algorithm used
DDPG
DDPG
DDPG
DDPG
DON+Features
Number of steps
4 10+
6-10+
5 104
4 10
2 2-10P
Number of warmup steps
4 10a
3-10"
5- 10
4 4-10²
10"
Replay memory size
15 10a
3 3-10*
15 15-10a
15
50 10*
Discount factor
0.99
0.99
0.99
0.99
0.99
Leaming rate
(10 10-$)
(10-1, 10-$)
(10-4, 10-3)
(10-3, 10-2)
10-4
Optimizer
Adam
Adam
Adam
Adam
Adam
Gradient clipping
1.0
1.0
1.0
1.0
Not set
Exploration style
Uniform noise
Uniform noise
Uniform noise
Uniform noise
e-greedy
Initial exploration settings
[-0.5, 0.5]
2
10.5.0.5.
[-0.3, 0.3]
0.95
Final exploration settings
10-3,10 1,10- 9|
10-3, _10-a]
10-2 -0000
10-3,10 10
0.05
Exploration steps
28 10
48 10a
10a
28 10
15 15-10*
Batch
164
32
256
32
Target model update
10-3
10-3
10-3
10-2
10a
Initial rundom steps Training
0
0
Initial random steps Testing
10
10
10
10
10
Frame-skip
4
4
4
4
Projection Regularization term
10-6
10-6
10
10-6
Table 4: Training settings for the attacker.
20


Reviews:
[{"review_id": "SklxEU8CtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors of this paper propose a novel adversarial attack for deep reinforcement learning. Different from the classical attacks, e.g., FGSM, they explicitly minimize the reward collect by the agent in a form of Markov decision process. Experiment results demonstrate that the proposed approach can damage the well-performed policy with a much bigger performance drop than gradient-based attacks.\n\nPaper strength:\n\n1.\tThe paper is well-organized and easy to follow.\n2.\tModel the adversary of reinforcement learning (RL) system as another MDP and solve it with RL is novel and interesting. The proposed attacking diagram can be devised in a pure black-box setting and also can be incorporated with white-box attacks. \n3.\tWith such a strong attack, the authors derive an upper bound on the impact of attacks and shed light on new research studying the robustness of deep RL approach.\n\nPaper weakness:\n1.\tThe author should give more details about how you use a gradient-based exploration to guide the adversary. From my point of view, I think the black-box attack is more practical and interesting. I would like to see the clearer comparison of optimal attack in a pure black-box setting with gradient-based attacks.\n2.\tThough FGM is not as efficient as the proposed optimal attack, they are simpler than a learning-based approach. Please describe the details and cost of training the attack agency, e.g., the hyper-parameter, number of training iterations.\n3.\tWhile the conclusion of smooth policies is more resilience for adversaries is interesting, I would like to see the evaluation results of such a novel finding. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgxbHDkcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates adversarial attacks on learned (fixed) policies. In particular, they devise optimal attacks in the sense that e.g. the agent’s objective should be minimised by the attacker. It is assumed that the attacker can manipulate the observed state of the agent at each time step during testing in a restricted way such that perturbations are small in the state or feature space. The paper derives attacker value functions when the attacker’s goal ist to minimise the original agent’s average cumulative reward and shows how to maximise them using gradient methods. In order to show when attacks are more likely to be successful, a theorem is presented and proved that shows that the resilience of a policy against attacks depends on its smoothness. Experiments show that a) attacks that are trained for optimality w.r.t. to minimising the average reward of the agent outperform a baseline method that only optimises a related criterion. b) Agent policies that are trained with DRQN, yielding a recurrent controller, outperform non recursive ones. \n\nOverall, I think this paper can be a good contribution to the subfield of adversarial attacks on MDP policies. In my view, the derivations and overall structure and results of the paper are sound. However, I would have liked to get a better understanding and motivation for the investigated problem setting, such as projected applications and state or features spaces that could be manipulated in the proposed way.\n\nPros:\nPaper is well written and generally easy to follow.\nThe derivations look sound and are well motivated. Proposition 5.1. can help in understanding how to train resilient agent policies. \nExperiments confirm that optimising for optimal attacks does indeed find empirically better attack strategies. \n\n\nCons: \nNo real application was presented. While applications are imaginable, a real application would have been beneficial. In particular, getting an idea of how real pertubations may look like in a realistic domain. The same holds for the assumption that the state (or features) can be manipulated at each time step, but only slightly. \nThe implications of Proposition 5.1. were never tested, as far as I understand. You only test DRQN vs DQN, which validates your POMDP assertions, but other comparisons are missing.\nThe tested Open AI Gym environments are very basic, with the exception of Pong. This reiterates my application argument.\n\nNotation/Writing: \nOn page 5, last paragraph you should use a different letter for J in order to reduce notational confusion.  \nSection 6 is not on the same standard as the other sections in terms of writing, e.g. 'We have obtained from 3 policies‘ , some stray ’the’, some plural vs. singular issues. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xNwye0YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the design of adversarial policies (where the action of the adversarial agent corresponds to a perturbation in the state perceived by the primary agent). In particular it focuses on the problem of learning so-called optimal adversarial policies, using reinforcement learning.\n\nI am perplexed by this paper for a few reasons:\n1)\tWhat is the real motivation for this work?  The intro argues “casting optimal attacks is crucial when assessing the robustness of RL policies, since ideally, the agent should learn and apply policies that resist *any* possible attack”.   If the goal is to have agents that are robust to *any* attacks, then they cannot be robust just to so-called optimal attacks.  And so what is really the use of learning so-called optimal attacks?\n2)\tThe notion itself of “optimal” attack is not clear.  The paper does not properly discuss this.  It quickly proposes one possible definition (p.4): “the adversary wishes to minimize the agent’s average cumulative reward”.  This is indeed an interesting setting, and happens to have been studied extensively in game-theoretic multi-agent systems, but the paper does not make much connection with that literature (apart from a brief mention at bottom of p.2 / top of p.3), so it’s not clear what is new here compared to this.   It’s also not discussed whether it would ever be worthwhile considering other notions of optimality for the adversary, and what would be the properties of those.\n\nSo overall, while I find the general area of this work to be potentially interesting, the current framing is not well motivated enough, and not sufficiently differentiated from other work in robust MDPs and multi-agent RL to make a strong contribution yet.\n\nMore minor comments:\n-\tP.3: “very different setting where the adversary has a direct impact on the system” => Clarify what are the implications of this in terms of framework, theory, algorithm.\n-\tP.4: You assume a valid Euclidean distance for the perturbed state.  Is this valid in most MDP benchmarks?  How is this implemented for the domains in the experiments?  What is the action space considered? Do you always assume a continuous action space for the attacker?\n-\tP.5: “we can simply not maintain distributions over actions” -> Why not?  Given the definition of perturbation, this seems feasible.\n-\tP.5:  Eqn 4 is defined for a very specific adversarial reward function. Did you consider others?  Is the gradient always easy to derive?\n-\tP.6: Eqn (5) & (6): What is “R” here?\n-\tP.7: Figure 1, top right plot. Seems here that the loss is above 0 for small \\epsilon.  Is this surprising?  Actually improving the policy?\n-\tP.7: What happens if you consider even greater \\epsilon?  I assume the loss is greater.  But then the perturbation would be more detectable?  How do you think about balancing those 2 requirements of adversarial attacks?  How should we formalize detectability in this setting?\n-\tFig.2: Bottom plots are too small to read.\n-\tSec.6:  Can you compare to multi-agent baselines, e.g. Morimoto & Doya 2005.\n-\tP.8: “We also show that Lipschitz policies have desirable robustness properties.” Can you be more specific about where this is shown formally?  Or are you extrapolating from the fact that discrete mountain car suffers more loss than continuous mountain car?  I would suggest making that claim more carefully.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of devising optimal attacks in deep RL to minimize the main agent average reward. In the white-box attack setting, optimal attacks amounts to solving a Markov Decision Process, while in black-box attacks, optimal attacks can be trained using RL techniques. Empirical efficiency of the attacks was demonstrated. It has valuable contributions on studying the adversarial robustness on deep RL. However, the current motivation and setup needs to be made clearer, and so is not being accepted at this time. We hope for these comments to help improve a future version.
Decision: Reject