Toward Evaluating Robustness of Deep Reinforcement Learning with Continuous Control
Authors: Tsui-Wei Weng
Krishnamurthy (Dj) Dvijotham*
Jonathan Uesato*
Kai Xiao*
Sven Gowal*
Robert Stanforth*
Pushmeet Kohli
Venue:  (2020)
Keywords: deep learning
reinforcement learning
robustness
adversarial examples

Abstract: Deep reinforcement learning has achieved great success in many previously difficult reinforcement learning tasks, yet recent studies show that deep RL agents are also unavoidably susceptible to adversarial perturbations, similar to deep neural networks in classification tasks. Prior works mostly focus on model-free adversarial attacks and agents with discrete actions. In this work, we study the problem of continuous control agents in deep RL with adversarial attacks and propose the first two-step algorithm based on learned model dynamics. Extensive experiments on various MuJoCo domains (Cartpole, Fish, Walker, Humanoid) demonstrate that our proposed framework is much more effective and efficient than model-free based attacks baselines in degrading agent performance as well as driving agents to unsafe states. 

Published as a conference paper at ICLR 2020
TOWARD EVALUATING ROBUSTNESS OF DEEP REIN- FORCEMENT LEARNING WITH CONTINUOUS CONTROL
Tsui-Wei Weng' Krishnamurthy (Dj) Dvijotham Jonathan Uesato A Kai Xiao Sven Gowal Robert Stanforth 2 Pushmmet Kohli 'MI. 'DeepMind
twwengemit. edu, {dvij, jueeatto/goooggeeeo com, kaix@mit. {agowal, stanforth, pushmeet @google. com
ABSTRACT
Deep reinforcement learning has achieved great success in many previously diffi- cult reinforcement learning tasks, yet recent studies show that deep RI. agents are also unavoidably susceptible to adversarial perturbations, similar to deep neural networks in classification tasks. Prior works mostly focus model-free adver- sarial attacks and agents with discrete actions. this work, we study the problem of continuous control agents in deep RL with adversarial attacks and propose the first two-step algorithm based on learned model dynamics. Extensive experiments various MuloCo domains (Cartpole, Fish, Walker, Humanoid) demonstrate that our proposed framework is much more effective and efficient than model-free at- tacks baselines in degrading agent performance well as driving agents unsafe states.
INTRODUCTION
Deep reinforcement learning (RL) has revolutionized the fields of Al and machine learning over the last decade. The introduction of deep learning has achieved unprecedented success in solving many problems that were intractable in the field of RL, such playing Atari games from pixels and performing robotic control tasks (Mnih et al., 2015 Lilicrap et al., 2015) Tassa et al., 2018). Un- fortunately, similar to the case of deep neural network classifiers with adversarial examples, recent studies show that deep RL agents are also vulnerable to adversarial attacks.
A commonly-used threat model allows the adversary to manipulate the agent's observations every time step, where the goal of the adversary is to decrease the agent's total accumulated reward. As a pioneering work in this field, (Huang et al. (017) show that by leveraging the FGSM attack on each time frame, an agent's 's average reward can be significantly decreased with small input adversarial perturbations in five Atari games. (Lin al. 2017) further improve the efficiency of the attack in (Huang et al., (017)) by leveraging heuristics of detecting a good time to attack and luring agents bad states with sample- based Monte-Cari planning on a trained generative video prediction model.
Since the agents have discrete actions in Atari games (Huang et al. 2017 Lin et al. 2017). the prob- lem of attacking Atari agents often reduces to the problem of finding adversarial examples on image classifiers, also pointed out in (Huang et al., (017), where the adversaries intend to craft the input perturbations that would drive agent's 's new action to deviate from its nominal action. However, for agents with continuous actions, the above strategies can not be directly applied. Recently, (Uesato) et al. 2018) studied the problem of adversarial testing for continuous control domains in similar but slightly different setting. Their goal was to efficiently and effectively find catastrophic failure given trained agent and to predict its failure probability. The key to success in (Uesato et al.] 2018) is the availability of agent training history. However, such information may not always be accessible to the users, analysts, and adversaries.
Besides, although may not be surprising that adversarial attacks exist for the deep RL agents as adversarial attacks have been shown to be possible for neural network models in various supervised
Work done during summer internship DeepMind, UK.
Equal contributions.

---

Published as a conference paper at ICLR 2020
learning tasks. However, the vulnerability of RL agents not be easily discovered by existing baselines which are model-free and build upon random searches and heuristics - this is also verified by our extensive experiments
various domains (e.g. walker, humanoid, cartpole, and fish), where the agents still achieve close to their original best rewards even with baseline attacks at every time step. Hence it is important and necessary to have a systematic methodology to design non-trivial ad- versarial attacks, which can efficiently and effectively discover the vulnerabilities of deep RL agents this is indeed the motivation of this work.
This paper takes a first step toward this direction by proposing the first sample-efficient model-based adversarial attack. Specifically, we study the robustness of deep RL agents in more challenging set- ting where the agent has continuous actions and its training history is not available. We consider the threat models where the adversary is allowed to manipulate an agent's observations or actions with small perturbations, and we propose two-step algorithmic framework to find efficient adversarial attacks based on learned dynamics models. Experimental results show that our proposed model- based attack can successfully degrade agent performance and is also more effective and efficient than model-free attacks baselines.
The contributions of this paper are the following:
To the best of our knowledge, we propose the first model-based attack on deep RL agents with continuous actions. Our proposed attack algorithm is a general two-step algorithm and can be directly applied to the two commonly-uses threat models (observation manipulation and action manipulation).
We study the efficiency and effectiveness of our proposed model-based attack with model- free attack baselines based on random searches and heuristics. We show that our model- based attack can degrade agent performance in numerous MuJoCo domains by up to 4x in terms of total reward and up to 4.6x in terms of distance to unsafe states (smaller means stronger attacks) compared to the model-free baselines.
Our proposed model- -aaeel attack also outperform all the baselines by large margin in weaker adversary setting where the adversary cannot attack at every time step. In addition, ablation study the effect of planning length in our proposed technique suggests that method can still be effective even when the learned dynamics model is not very accurate.
2 BACKGROUND
Adversarial attacks in reinforcement learning. Compared to the rich literature of adversarial examples in image classifications (Szegedy et al. (013) and other applications (including natural language processing (Tia Liang 2017), speech (Carlini & Wagner] 2018), etc), there relatively little prior work studying adversarial examples in deep RL. One of the first several works in this field are (Huang et al. 2017) and (Lin et al. 2017). where both works focus on deep RL agent in Atari games with pixels-based inputs and discrete actions. In addition, both works assume the agent to be attacked has accurate policy and the problem of finding adversarial perturbation of visual input reduces to the same problem of finding adversarial examples
image classifiers. Hence, (Huang let al. 2017) applied FGSM (Goodfellow et al.) 2015) to find adversarial perturbations and (Lin al. 2017) further improved the efficiency of the attack by heuristies of observing a good timing to attack - when there is a large gap in agents action preference between most-likely and least- likely action. In similar direction, (Uesato et al. (018)) study the problem of adversarial testing by leveraging rejection sampling and the agent training histories. With the availability of training histories, (Uesato et al., 2018) successfully uncover bad initial states with much fewer samples compared to conventional Monte-Carke sampling techniques. Recent work by (Gleave et al. 2201)) consider an alternative setting where the agent is attacked by another agent (known as adversarial policy), which is different from the two threat models considered in this paper. Finally, besides adversarial attacks in deep RL, a recent work (Wang et al., 2019) study verification of deep RL agent under attacks, which is beyond the scope of this paper.
Learning dynamics models. Model-basee RL methods first acquire predictive model of the environment dynamics, and then use that model to make decisions (Atkeson & Santamaria 1997). These model-based methods tend to be more sample efficient than their model-free counterparts,
2

---

Published as a conference paper at ICLR 2020
+ås
Agent
Agent
Observation S;
Action a;
Observations: Rewardr:
Action ap
Reward n
1i+1
Si+1
Environment
Environment
Ti+1
T++1
+āa,
Attack observations of agent.
(b) Attack actions of agent.
Figure 1: Two commonly-used threat models.
and the learned dynamics models be useful across different tasks. Various works have focused on the most effective ways to learn and utilize dynamics models for planning in RL (Kurutach et al., 2018 Chua al. 2018 Chiappa et al. 2017 Fu et al. 2016).
PROPOSED FRAMEWORK
In this section, first describe the problem setup and the two threat models considered in this paper. Next, we present an algorithmic framework to rigorously design adversarial attacks on deep RL agents with continuous actions.
3.1 PROBLEM SETUP AND FORMULATION
Let 8; E RN and a; E R be the observation vector and action vector at time step i, and let RN + RM be the deterministic policy (agent). Let f R. x RM RNN be the dynamics model of the system (environment) which takes current state-action pair (s, a,) as inputs and outputs the next state 8-+1- We are now in the role an adversary, and as an adversary, our goal to drive the agent to the (un-safe) target states Surget within the budget constraints.
We can formulate this goal into two optimization problems, as we will illustrate shortly below. Within this formalism, we can consider two threat models:
Threat model (i): Observation manipulation. For the threat model of observation manipulation, an adversary is allowed to manipulate the observation s; that the agent perceived within budget:
|| xc S,
Lg S 8; As; S Us:
where As; E N is the crafted perturbation and U, E INN, Lg E N are the observation limits.
Threat model (ii): Action manipulation. For the threat model of action manipulation, adver- sary can craft Aay e R such that
A S,
Lg S a, Aa; S Ua.
where Uo E M Lo E R are the limits of agent's actions.
Our formulations. Given an initial state so and a pre-trained policy n, our (adversary) objective is minimize the total distance of each state 8; to the pre-defined target state Surget up the unrolled (planning) steps T. This can be written as the following optimization problems in Equations 3 and for the Threat model (i) and (ii) respectively:
T
min As:
Σ i=1
d( Suareet)
(3)
s.t.
= T(2 + Asi), 8141 f(si, Constraint (1), i E Er,
T
min
Σ d(si. Suager)
Anc
1
s.t.
0, = T (st), Si+1 = f(3,, a; + Aaj), Constraint (2), i E Er.
3

---

Published as a conference paper at ICLR 2020
A common choice of d(x, 3) is the squared & distance || v11 and f is the learned dynamics model of the system, and is the unrolled (planning) length using the dynamics models.
3.2 OUR ALGORITIMM
In this section, we propose a two-step algorithm to solve Equations and The core of our pro- posal consists of two important steps: learn dynamics model f of the environment and deploy optimization technique solve Equations [3] and 4 We first discuss the details of each factor, and then present the full algorithm by the end of this section.
Step 1: learn a good dynamics model f. Ideally, if f is the exact (perfect) dynamics model of the enviromment and assuming we have optimization oracle to solve Equations [ and 4 then the solutions are indeed the optimal adversarial perturbations that give the minimal total loss with e-budget constraints. Thus, learning good dynamics model can conceptually help developing strong attack. Depending on the environment, different forms of f can be applied. For example, the environment of concerned is close a linear system, then we could let f(s a) As+ where A and B are unknown matrices to be learned from the sample trajectories (st, a, S(+1) 1) pairs. For more complex environment, we could decide we still want to use a simple linear model (the next state prediction may be far deviate from the true next state and thus the learned dynamical model is less useful) or instead switch to a non-linear model, e.g. neural networks, which usually has better prediction power but may require more training samples. For either case, the model parameters A, B or neural network parameters can be learned via standard supervised learning with the sample trajectories pairs (s,, 0g 81+1).
Step 2: solve Equations 3 aadd 4 Once we learned a dynamical model f. the next immediate task solve Equation aadd to compute the adversarial perturbations of observations/action.. When the planning (unrolled) length T > 1. Equation 3 usually can not be directly solved by off-the- shelf convex optimization toolbox since the deel RL policy is usually non-linear and non-convex neural network. Fortunately, we can incorporate the two equality constraints of Equation 3 into the objective and with the remaining c-budget constraint (Equation [T) Equation 3 can be solved via projected gradient descent (PGD) [] Similarly, Equation 4 can be solved via PGD to get Aa,- We note that, similar to the nestep model predictive control, our algorithm could use a much larger planning (unrolled) length 7 when solving Equations 3 and 4 and then only apply the first (5 7) adversarial perturbations on the agent over n time steps. Besides, with the PGD framework, f is not limited feed-forward neural networks. Our proposed attack is summarized in Algorithm for Step 1, and Algorithm 3 for Step 2.
Algorithm Collect.trajjecton
1: Input: pre-trained policy n, MaxSampleSize nt,, environment env
2: Output: a set of trajectory pairs S
á 4: env.reset()
3:
5: while k < n, do
6: (kk + (*k)
7: Sk+1 < env.stp((as)
8:
SJ ag 5k41)}
9: k+ +1
10:
end while
11: Return S
EXPERIMENTS
In this section, we conduct experiments on standard reinforcement learning environment for con- tinuous control (Tssa et al. 2018). We demonstrate results on 4 different environments in Mu-
Alternativel, standard optimal control methods such as Linear Quadratic Regulator (I.QR) and iterative L Linear Quadratic Regulator (i-LQQ) can also be applied to solve Equations and approxmmaelly.
4

---

Published as a conference paper at ICLR 2020
Algorithm 2 learn ddnaaiissss
1: Input: pre-trained policy т, MaxSampleSize n,,, environment env, trainable parameters W
2: Output: learned dynamical model [(s, a; W)
3: Sugent Colloooolttaaaaeee trajectories(t..c......... env)
4: Srandom t- Colllect.rajecctriiee//moooo www.ocycppppooo.oooooooooooocoo T2, env)
5: f(s, a; W) <- pnterviiee.eemmiigg.gggooonniii
U Srandom W)
6: Return f(8, as W)
Algorithm baaede__bsed_atte
1: Input:
pre-trained policy x, learned dynamical model [(s, a; W), threat model, maximum perturbation magnitude e, unroll length 7, apply perturbation length (S T)
2: Output: a sequence of perturbation 61,---, 6n
if threat model is observation manipulation (Eq. T then
4: Solve Eq. 3 with parameters f. e. () via PGD to get Ó1,...,8r 8T
5: else if threat model action manipulation (Eq. 2 then
6: Solve Eq. [ with parameters (I, f,,, c, T) via PGD to get 81 61....,.r
7:
end if
8: Return δ1 Ó1,...,, 6n
JoCo Tassa et al. (2018) and corresponding tasks: stand/walk and agent (Barth-Maron et al. 2018) with default Gaussian noise (2 0.3I) on the action and the score of the agents without attacks is summarized in Appendix A.3] The organization is follows: we first evaluate the effectiveness of our proposed model-based attack and three model-free baselines in terms of both loss and reward. Next, we conduct ablation study on the key parameter of our algorithm the planning length T, evaluate our algorithm on a weaker attack setting and also discuss the efficiency of our proposed attack in terms of sample complexity.
Fish-upright, Walker- For the deep RL agent, we train a state-of-the-art D4PG
Evaluations. We conduct experiments for 10 different runs, where the environment is reset to different initial states in different runs. For each run, we attack the agent for episode with 1000 time steps (the default time intervals is usually 10 ms) and we compute the total loss and total return reward. The total loss calculates the total distance of current state to the unsafe states and the total return reward measures the true accumulative reward from the environment based on agent's 's action. Hence, the attack algorithm is stronger if the total return reward and the total loss smaller.
Baselines. We compare our algorithm with the following model-free attack baselines with random searches and heuristies:
rand-U: generate m randomly perturbed trajectories from Uniform distribution with inter- val [-c, c] and return the trajectory with the smallest loss (or reward),
rand-B: generate randomly perturbed trajectories from Bernoulli distribution with prob-
ability 1/2 and interval [-c, c], and return the trajectory with the smallest loss (or reward),
flip: generate perturbations by flipping agent's observations/actions within the budget in foo norm.
For rand-U and rand-B, they are similar Monte-Carka sampling methods, where we generate m sample trajectories from random noises and report the loss/reward of the best trajectory (with minimum loss or reward among all the trajectories). We set m 1000 throughout the experiments. More details see Appendix A.2
Our algorithm. A 4-layer feed-forward neural network with 1000 hidden neurons per layer is trained as the dynamics model f respectively for the domains of Cartpole, Fish, Walker and Hu- manoid. We use standard & loss (without regularization) to learn a dynamics model f. Instead of using recurrent neural network to represent f. we found that the 1-step prediction for dynamics with the 4-layer feed-fon forward network is already good for the MuJoCo domains we are studying. Specif- ically, for the Cartpole and Fish, found that 1000 episodes (1e6 training points) sufficient
5

---

Published as a conference paper at ICLR 2020
Walker.walk
t 160
320
t= 480
t= 640
800
960
Rand U
Rand B
Flip
Ours
Makes Walker fall down
Figure 2:
Video frames of best attacks in each baseline among 10 runs for the Walker. example. Only our proposed attack can constantly make the Walker fall down (since we minimizing its head height to be zero).
to train good dynamice model (the mean square error for both training and test losses are at the order of 10-5 for Cartpole and 10-2 for Fish), while for the more complicated domain like Walker and Humanoid, more training points (5e6) are required to achieve a low test MSE error (at the order of 10 and 10t for Walker and Humanoid respectively). Consequently, we use larger planning (unrolled) length for Cartpole and Fish (e.g. T 10, 20), while smaller T (e.g. 3 or 5) is used for Walker and Humanoid. Meanwhile, we focus on applying projected gradient descent (PGD) to solve Equation 3 and 4 We use Adam the optimizer with optimization steps equal to 30 and report the best result for each run from a combination of 6 learning rates, unroll length (Ti and steps of applying PGD solution with Tü
4.1 RESULTS
For observation manipulation, we report the results on Walker, Humanoid and Cartpole domains with tasks (stand, walk, balance, swingup) respectively. The unsafe states Stargt for Walker and Humanoid are set to be zero head height, targeting the situation falling down. For Cartpole, the unsafe states set to have 180" pole angle, corresponding to the cartpole not swinging up and nor balanced. For the Fish domain, the unsafe states for the upright task target the pose swimming fish to be not upright, e.g. zero projection the 2-axis.
The full results of both two threat models on observation manipulation and action manipulation are shown in Table Tp. and c, d respectively. Since the loss is defined the distance to the target (unsafe) state, the lower the loss, the stronger the attack. is clear that our proposed attack achieves much lower loss Table [ & than the other three model-free baselines, and the averaged ratio is also listed in D & d. Notably, over the 10 runs, our proposed attack always outperforms baselines for the threat model of observation perturbation and the Cartpole domain for the threat model of action perturbation, while still superior to the baselines despite losing two times to the flip baseline on the Fish domain.
To have a better sense on the numbers, we give some quick examples below. For instance, as shown in Table a and b, we show that the average total loss of walker head height is almost unaffected for the three baselines - if the walker successfully stand or walk, its head height usually has to be greater than 1.2 at every time step, which 1440 for one episode while our attack can successfully lower the walker head height by achieving average of total loss of 258(468), which is roughly 0.51(0.68) per time step for the stand (walk) task. Similarly, for the humanoid results, a successful humanoid usually has head height greater than 1.4, equivalently total loss of 1960 for episode, and Table Ta shows that the d4pg agent is robust to the perturbations generated from the three model- free baselines while being vulnerable to proposed attack. Indeed, shown in Figure the

---

Published as a conference paper at ICLR 2020
Compare three model-free attack baselines (rand-U, rand-B, flip) and our algorithm (Ours) in 4 different domains and tasks. We report the following statistics over 10 different runs: mean, standard deviation, averaged ratio, and best attack (number times having smallest loss over 10 different runs). Results show that our attack outperforms all the model-free attack baselines for observation manipulation threat model by large margin for statistics. Our proposed attack is also superior on the action manipulation threat model and win over most of the evaluation metrics.
Table 1:
(a) Observation manipulation: mean and standard deviation (in parenthesis)
Total loss
rand-U
rand-B
flip
Ours
stand
1462 (70)
1126 (86)
1458 (24)
258 (55)
Walker
walk
1517 (22)
1231 (31)
1601 (18)
466 (42)
stand
1986 (28)
1808 (189)
1997 (5)
516 (318)
Humanoid
walk
1935 (22)
1921 (31)
1982 (9)
1457 (146)
balance
4000 (0.02)
3999 (0.04)
3989 (2)
2101 (64)
Cartpole
swingup
3530 (1) 3525 (1) 3516 (1)
2032 (172)
(b) Observation manipulation: averaged ratio and rank-1
Total loss (avg ratio)
Ours/rand-U
Ours/rand-B
Ours/flip
best attack
stand
0.18
0.23
0.18
Ours: 10/10, others: 0/10
Walker
walk
0.31
0.38
0.29
Ours: 10/10, others: 0/10
stand
0.26
0.29
0.26
Ours: 10/10, others: 0/10
Humanoid
walk
0.75
0.76
0.74
Ours: 10/10, others: 0/10
balance
0.53
0.53
0.53
Ours: 10/10, others: 0/10
Cartpole
swingup
0.58
0.58
0.58
Ours: 10/10, others: 0/10
(c) Action manipulation: mean and standard deviation (in parenthesis)
Total loss
rand-U
rand-B
flip
Ours
balance
4000 (0.03)
3999 (0.08)
3046 (1005)
1917 (102)
Cartpole
swingup
3571 (1)
3487 (7)
1433 (4)
1388 (50)
Fish
upright
935 (27)
936 (24) 907 (22)
824 (84)
(d) Action manipulation: averaged ratio rank-1
Total loss (avg ratio)
Ours/rand-U
Ours/rand-B
Ours/flip
best attack
balance
0.48
0.48
0.63
Ours: 10/10, others: 0/10
Cartpole
swingup
0.39
0.40
0.97
Ours: 10/10, others: 0/10
Fish
upright
0.88
0.88
0.91
Ours: 8/10, flip: 2/10
7

---

Published as a conference paper at ICLR 2020
Table 2: Compare three attack baselines (rand-U, rand-B, flip) and our algorithm (Ours) in three different domains and tasks. Performance statistics of 10 different runs are reported.
(a) The mean and standard deviation (in parenthesis) over 10 different runs
Total reward
rand-U
rand-B
flip
Ours
stand
937 (41)
744 (48)
993 (8)
235 (38)
Walker
walk
941 (23)
796 (21)
981 (9)
225 (50)
stand
927 (21)
809 (85)
959 (5)
193 (114)
Humanoid
walk
934 (22)
913 (21)
966 (6)
608 (66)
balance
995 (0. 17)
986 (0.16)
985 (3)
385 (6)
Cartpole
swingup
873 (0.75)
851 (2)
852 (0.29)
353 (61)
(b) Average ratio and number of times our algorithm being the best attack over 10 runs.
Total reward (avg ratio)
Ours/rand-U
Ours/rand-B
Ours/tlip
best attack
stand
0,25
0.32
0.24
Ours: 10/10, others: 0/10
Walker
walk
0.24
0.28
0.23
Ours: 10/10, others: 0/10
stand
0.21
0.24
0.20
Ours: 10/10, others: 0/10
Humanoid
walk
0.65
0.67
0.63
Ours: 10/10,
others: 0/10
balance
0.39
0.39
0.39
Ours: 10/10, others: 0/10
Cartpole
swingup
0.41
0.42
0.42
Ours: 10/10, others: 0/10
walker and humanoid falls down quickly (head height close to zero) under our specially-designed attack while remaining unaffected for all the other baselines.
4.2 DISCUSSION
Evaluating on the total reward. Often times, the reward function is a complicated function and its exact definition is often unavailable. Learning the reward function is also an active research field, which not in the coverage of this paper. Nevertheless, as long as we have some knowledge of unsafe states (which is often the case in practice), then we can define unsafe states that are related to low reward and thus performing attacks based on unsafe states (i.e. minimizing the total loss of distance to unsafe states) would naturally translate to decreasing the total reward of agent. As demonstrated in Table 2, the results have the same trend of the total loss result in Table [ where proposed attack significantly outperforms all the other three basclines. In particular, our method can lower the average total reward up to 4.96 x compared to the baselines result, while the baseline results close to the perfect total reward of 1000.
Evaluating the effect of planning length. To investigate model effect over time, we perform ab- lation studies on the planning/unroll length T' of our proposed model-based attack in three examples: balance (II) walker. and (III) walker.stand.
(I) cartpole. carlpole.balance
(I) Cartpole balance. Our learned models are very accurate (test MSE error on the order of 10-6). We observed that the prediction error of our learned model compared the true model (the MuJoCo simulator) is around 10% for 100 steps. Hence, we can choose T to be very large (e.g. 20-100) and our experiments show that the result of = 100 slightly better, see Appendix [A.4]
(II) Walker walk. This task is much more complicated than (I), and our learned model is less accurate (test MSE is 0.447). For 10 steps, the prediction error of our learned model compared to the true model is already more than 100%, and hence using small T for planning would be more reasonable. Table 3h shows that T = indeed gives the best attack results (decreases the loss by 3.2x and decreases the reward by 3.6x compared to the best baseline (randB)) and the attack becomes less powerful as T increases. Nevertheless, even with = 10, our proposed technique still outperforms the best baseline (randB) by 1.4 both in the total loss and total reward.
8

---

Published as a conference paper at ICLR 2020
Table 3: Ablation study on the planning length T. Compare attack baselines (rand-U, rand-B, flip) and our algorithm (Ours) and report performance statistics of 10 different runs.
(a) domain: Walker, task: walk (observation perturbation)
Walker.walk
Total loss
Total reward
mean
std
med
min
max
mean
std
med
min
max
Ours, T
468
79
489
286
567
222
45
227
135
300
Ours, T
604
31
611
535
643
353
51
362
253
441
Ours, T ===
761
65
771
617
837
483
60
496
348
540
Ours,
881
886
753
975
568
48
579
469
623
Ours,
874
891
723
1002
583
58
604
483
647
Ours, T = 20
937
62
950
804
993
634
41
638
559
687
rand-U
1517
22
1522
1461
1542
941
23
945
885
965
rand-B
1231
31
1234
1189
1272
796
21
796
766
824
flip
1601
18
1604
1562
1619
981
9
984
961
991
(b) domain: Walker, task: stand (observation perturbation)
Walker.stand
Total loss
Total reward
mean
std
med
min
max
mean
std
med
min
max
Ours, T
322
84
319
202
453
257
67
265
163
366
Ours, T
279
55
264
223
391
246
40
232
200
322
Ours, T
163
53
154
93
246
193
27
188
154
238
Ours, T
84
46
67
42
165
153
24
142
132
194
Ours,
101
40
82
57
157
164
23
152
143
201
Ours, T =20
117
98
68
193
170
161
149
207
rand-U
1462
70
1454
1341
1561
938
932
866
999
rand-B
1126
86
1130
973
1244
744
48
744
664
809
flip
1458
24
1451
1428
1501
993
8
997
979
999
Table 4: Less frequency attack. Report statistics of 10 different runs with different initial states in the walker domain with task stand.
Total loss
Total reward
mean
std
med
min
max
mean
std
med
min
max
Ours
934
152
886
769
1187
648
95
622
559
799
rand-U
1511
35
1502
1468
1558
970
20
964
947
999
rand-B
1431
77
1430
1282
1541
924
41
923
840
981
flip
1532
15
1537
1496
1546
996
5
999
984
1000
(III) Walker stand. The learned model is slightly more accurate than the (II) (test MSE is 0.089) in this task. Interestingly. Table 3p show that with the more accurate walker.stand model (compared to the walker.walk model). T 10 gives the best avg total loss& reward, which 13.4x and 4.9x smaller than the best baseline rand-B, Note that even with T' = 1, the worst choice among all our reported 7', the result is still 3.5x and 2.9x better than the best baselines, demonstrating the effectiveness of our proposed approach.
The main takeaway from these experiments that when the model accurate we can use larger T in our proposed attack; while when the model is less accurate, smaller T' is more effective (as in the Walker. walk example). However, even under the most unfavorable hyperparameters, our proposed attack still outperforms all the baselines by large margin.
Evaluating on the effectiveness attack. We study the setting where attackers are less powerful they can only attack every time steps instead of every transition. Table 4soow that our proposed

---

Published as a conference paper at ICLR 2020
attack is indeed much stronger than the baselines even when the attackers power is limited to attack every 2 time steps: (1) compared to the best results among three baselines, our attack gives 1.53x smaller avg total loss (2) the mean reward of all the baselines is close to perfect reward, while our attacks can achieve x.43x smaller average total reward compared to the best baseline.
Evaluating on the efficiency of attack. We also study the efficiency of the attack in terms of sample complexity, i.e. how many episodes do we need to perform an effective attack? Here we adopt the convention in control suite (Tassa et al. 2018)) where one episode corresponds 1000 time steps (samples) and learn the neural network dynamical model with different number of episodes.
Figure [ in Appendix A.1 plots the total head height loss of the walker (task stand) for base- lines and our method with dynamical model trained with three different number of samples: (5e5, le6, 5e6], or equivalently {500, 5000) episodes. We note that the sweep of hyper pa- rameters is the same for all the three models, and the only difference is the number of training samples. The results show that for the baselines rand-U and flip, the total losses are roughly at the order of 1400-1500, while stronger baseline rand-B still has total losses of 900-1200. However, if we solve Eq. equation 3 with f trained by 5e5 or le6 samples, the total losses can be decreased to the order of 400-700 and are already winning over the three baselines by significant margin. Same as our expectation, if we use more samples (c.g. 5e6, which is 5-10 times more), to learn a more accurate dynamics model, then it is beneficial our attack method the total losses can be further decreased by more than 2 and are at the order of 50-250 over 10 different runs. See Appendix A. for more details.
Here we also give a comparison between our model-based attack to existing works (Uesato et al. 2018 Gleave al.) 2019) on the sample complexity. In (Uesato et al.) 2018), 3e5 episodes of training data is used to learn the adversarial value function, which is roughly 1000 more data than even our strongest adversary (with 5e3 episodes). Similarly, (Gleave et al., 2019) use roughly 2e4 episodes to train an adversary via deep RL, which is roughly 4x more data than ours")
5 CONCLUSIONS AND FUTURE WORKS
In this paper, study the problem of adversarial attacks in deep RL with continuous control for two commonly-used threat models. We proposed the first model-based attack algorithm and showed that our formulation be easily solved by off-the-shed gradient-basee based solvers. Extensive experiments on MuJoCo domains show that our proposed algorithm outperforms all model-free based attack baselines by large margin. We hope discovery of the vulnerability of deep RL agent can bring more safety awareness to researchers when they design algorithms to train deep RL agents.
There are several interesting future directions be investigated based this work, including learning reward functions to facilitate a more effective attack, extending our current approach to develop effective black-box attacks, and incorporating our proposed attack algorithm to adversarial training of the deep RL agents. In particular, we think there are three important challenges that need be addressed study adversarial training of RL agents along with our proposed attacks: (1) The adversary and model need to be jointly updated. How do we balance these two updates, and make sure the adversary is well-trained each point in training? (2) How to avoid cycles in the training process due to the agent overfitting to the current adversary? (3) How to ensure the adversary doesn't t overly prevent exploration/balance unperturbed vs. robust performance?
ACKNOWLEDGEMENT
The authors thank Chongli Qin, Po-Sen Huang, Taylan Cemgil, Daniel J. Mankowitz, Nir Levine, Alistair Muldal, Gabriel Barth-Maron, Matthew W. Hoffman, Yuval Tassa, Tom Erez and Jost Tobias Springenberg for useful discussions and suggestions.
II is only qualitative comparison in the sample complexity regimes since the applications are not the same. This is in-line with the theoretical perspective that model-hased approaches are expected to be more sample-efficient than the model-free counterparts
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Abhas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint FrXiv:/806.0692 2018.
Christopher G Atkeson and Juan Carlos Santamaria. A comparison of direct and model- rein- forcement learning. International Conference on Robotics and Automation, 1997.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1844.08617 2018.
Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-to- text. In 2018 IEEE Security and Privacy Workshops (SPW). pp. 1-7. IEEE, 2018.
Silvia Chiappa, Sbastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. International Conference on Learning Representations (ICLR), 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Neural Information Process- ing Systems (NIPS), 2018.
Justin Fu, Sergey Levine, and Pieter Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. Intelligent Robots and Systems (IROS), 2016.
Adam Gleave Michael Dennis, Neel Kant, Cody Wild, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv: 1905. 10615, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks neural network policies. arXiv preprint arXiv: 2017.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods Natural Language Processing (EMNLP), Outstanding paper award, 2017.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint a8Xiv: 802.. 2018.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint www.iv::509.02900 2015.
Yen-Chen Lin, Zhang- Meang-Weiiln Hong, Yuan-Hong Liao, Meng-Li Shih, Ming- Yu Liu, and Min Sun. Tac- tics of adversarial attack deep reinforcement learning agents. arXiv preprint 10r.xv:703.06777. 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(750)::529, 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:/312.6111 1312.6199, 2013.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv:1801.00690 2018
Jonathan Uesato, Ananya Kumar, Csaba Szepesvari, Tom Frez, Avraham Ruderman, Keith Ander- son, Nicolas Heess, Pushmeet Kohli, et al. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. arXiv preprint arXiv:18/2.01647, 2018.
Yuh- Shhh-Shyng Wang, Tsui- Wei Weng, and Luca Daniel. Verification of neural network control policy under persistent adversarial perturbation. arXiv preprint arXiv: 1908 06353, 2019.
11

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 MORE ILLUSTRATION ON FIGURE
The meaning of Fig 3 is to show how the accuracy of the learned models affects our proposed technique:
1. we first learned models with different number of samples: 5e5, le6, 5e6 and found that with more training samples (e.g. 5e6, equivalently 5000 episodes), we are able to learn a more accurate model than the one with 5e5 training samples;
2. we plot the attack results total loss for our technique with learned models (denoted as PGD, num train) as well as the baselines (randU, randB, Flip) 10 different runs (initial- izations).
We show with the more accurate learned model (5e6 training samples), we are able to achieve a stronger attack (the total losses at the order of 50-200 over 10 different runs) than the less accurate learned model (e.g. 5e5 training samples). However, even with a less accurate learned model, the total losses on the order of 400-700, which already outperforms the best baselines by margin of 1.3-2 times. This result in Fig also suggest that very accurate model isn' It necessarily needed in our proposed method achieve effective attack. Of course, if the learned model is more accurate, then we are able to degrade agent's performance even more.
Walker.stand, total head height loss
2000
randU
randB
Flip
1500
PGD, num__raai 5e5
PGD, num train: 1e6
PGD, num num_train train: 5e6
1000
0
k i_t episode
Figure 3: Compare sample size on the Walkersstnd in 10 different initialization in the environment. The x-axis is the kth initialization and the y-axis is the total loss of corresponding initialization.
A.2 MORE DETAILS ON BASELINE IMPLEMENTATIONS
For the baselines (rand-U and rand-B), the adversary generates 1000 trajectories with random noise directly and we report the best loss/reward at the end of each episode. The detailed steps are listed below:
Step 1:
The perturbations are generated from a uniform distribution or bernoulli distribution within the range [-eps, eps] for each trajectory, and we record the total reward and total loss for each trajectory from the true environment (the MuloCo simulator)
Step 2: Take the best (lowest) total reward/loss among 1000 trajectories and report in Table 1 and 2.
12

---

Published as a conference paper at ICLR 2020
We note that here assume the baseline adversary has an "unfair advantage" since they have access to the true reward (and then take the best attack result among 1000 trials), whereas our techniques do not have access this information. Without this advantage, the baseline adversaries (rand-B, rand-U) may be weaker they use their learned model to find the best attack sequence. In any case, Table and 2 demonstrate that proposed attack can successfully uncover vulnerabilities of deep RL agents while the baselines cannot.
For the baseline flip, we add the perturbation (with the opposite sign and magnitude e) the original state/action and project the perturbed state/action are within its limits.
A.3 SCORE OF LEARNED POLICY WITHOUT ATTACKS
use default total timesteps 1000, and the maximum total reward 1000. We report the total reward of the d4pg agents used in this paper below. The agents are well-trained and have total reward close to 1000, which outperforms agents trained by other learning algorithms on the same tasks (e.g. DDPG, A3C in Sec (Tassa et al. 2018); PPO in Sec 5 (Abdolmaleki et al., ) and thus the agents in this paper can be regarded as state-of-the tfae--ffthhe-rt RL agents for these continuous control domain tasks. The attack results in Table and in our manuscript are hence suggested to be representative.
Domain
Task
Total reward
Walker
stand
994
Walker
walk
987
Humanoid
stand
972
Humanoid
walk
967
Cartpole
balance
1000
Cartpole
swingup
883
Fish
upright
962
A.4 ADDITIONAL EXPERIMENTS ON ABLATION STUDY
Table 5: Cartpole balance (action perturbation)
Total loss
mean
std
med
min
max
Ours, T 20
2173
51
2189
2087
2239
Ours, T 100
1951
113
1924
1851
2192
rand-U
4000
0
4000
4000
4000
rand-B
3999
0
3999
3999
3999
flip
3046
1005
3074
2060
3999
13


Reviews:
[{"review_id": "SkgbcLA1cS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an adversarial attack for perturbing\nthe actions or observations of an agent acting near-optimally\nin an MDP so that the policy performs poorly.\nI think understanding the sensitivity of a policy to\nslight perturbations in the actions it takes or the\nobservations that it receives is important for having\nrobust learned policies and controllers.\nThis paper presents an empirical step in the direction\nof showing that such attacks are possible, but in the context\nof the other adversarial attacks that are possible, this is\nnot surprising alone and would be much stronger with\nother contributions.\nI think an exciting direction of new work could be to\ncontinue formalizing these vulnerabilities and\nlooking at ways of adding robustness across many\nother domains.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eTSCxTtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "  *Synopsis*:\n  This paper looks at a new framework for adversarial attacks on deep reinforcement learning agents under continuous action spaces. They propose a model based approach which adds noise to either the observation or actions of the agent to push the agent to predefined target states. They then report results against several model-free/unlearned baselines on MuJoCo tasks using a policy learned through D4PG.\n\n  Main contributions:\n  - Adversarial attacks for Deep RL in continuous action spaces.\n\n  *Review*\n  The paper is well written, and has some interesting discussion/insight into attacking deep RL agents in continuous actions spaces. I think the authors are headed in the right direction, but compared to prior work in adversarial attacks for deep RL agents (i.e. the Huang and Lin) I have a few concerns that I feel the authors need to better explain/motivate in their paper. I am recommending this paper be rejected based on the following concerns. I am willing to raise my score if some of these are addressed by the authors in subsequent revisions\n\n  1. This algorithm requires the pre-trained policy to plan attacks (which may be a high bar for such an adversarial attack). It would be a nice addition to include similar results with \"black-box\" adversarial attacks, as mentioned in the Huang. \n\n  2. Another issue, addressed in the Lin paper, is this attack seems to require perturbation on every time step in a proposed trajectory. As mentioned by Lin, this is probably unrealistic and would cause the attacker to be detected. It would be another nice contribution to include variants that don't require perturbations on each transition.\n\n  3. Another unfortunate requirement is a learned model (or a way to simulate trajectories). From the Model Based RL literature, we know learning such a model is quite difficult and often unrealistic given our current approaches. While this is problematic, I think the paper could systematically test this looking at what happens as the model becomes less accurate over time. This could provide some nice results showing an accurate model isn't necessarily needed and anneal concerns over having to learn such a model.\n\n  4. It is unclear if the baselines measured against are meaningful in this setting, and I'm also a bit unclear how they are generated/implemented. Specifically, the random trajectories require you to return the generated trajectory with the smallest loss/reward. It is unclear how the adversary knows this information. Is it known through a model or some other simulation? Also the flip baseline could use a bit more explanation. I think these details can be safely placed in the appendix, but should appear somewhere in the final version.\n\n  5. I'm not sure the comparison to sample efficiency to the Gleave or Uesato papers are meaningful. For Gleave, the threat model explored is much different where they do not have access to the agent's observation or action streams and instead learn policies to affect the other agent in game scenarios. This is very different. Also, the Uesato is not adversarially attacking the agent, but attempting to find failure cases for the agent, which I again feel is very different from what you are trying to accomplish. I would remove this discussion and the claim at the end of the conclusion.\n\n\n  Other suggestions:\n\n  S1. It would be helpful to include the score of the learned policy without any attacks, to see how well the baselines are performing (this will help readers understand if these are reasonable/meaningful baselines).\n\n  S2. I'm unclear what figure three is adding to the paper, and am actually uncertain what the y-axis means. I don't think this is a wise use of the 9th page, and this plot could probably be relegated to the appendix.\n  \n  S3. As in prior work, it would be useful to see how well this line of attack works for multiple learning algorithms. Some potential candidates could be: PPO, TRPO, SAC, etc... \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByljlkVFFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposed a new adversarial attack method based on model-based RL. Unlike existing adversarial attack methods on deep RL, the authors first approximate the dynamics models and then generate the adversarial samples by minimizing the total distance of each state to the pre-defined target state (i.e. planning). Using Cartpole, Fish, Walker, and Humanoid, the authors showed that the proposed method can pool the agents more effectively. \n\nDetailed comments:\n\nThe proposed idea (i.e. designing an adversarial attack based on model-based RL) is interesting but it would be better if the authors can provide evaluations such as adversarial training and ablation studies for the proposed method (see the suggestion & question). So, I'd like to recommend \"weak accept\"\n\nSuggestion & question:\n\nCould the authors apply adversarial training based on the proposed methods? I wonder whether RL agents can be robust after adversarial training. \n\nInstead of utilizing a pre-defined target state $s_{target}$, we can also approximate a reward function and generate adversarial samples by minimizing the total rewards. It would be interesting if the authors can consider this case. \n\nCould the authors report an ablation study on the effects of T?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers adversarial attacks in continuous action model-based deep reinforcement learning. An optimisation-based approach is presented, and evaluated on Mujoco tasks.

There were two main concerns from the reviewers. The first was that the approach requires strong assumptions, but in the rebuttal some relaxations were demonstrated (e.g., not attacking every step). Additionally, there were issues raised with the choice of baselines, but in the discussion the reviewers did not agree on any other reasonable baselines to use.

This is a novel and interesting contribution nonetheless, which could open the field to much additional discussion, and so should be accepted.
Decision: Accept (Poster)