Illusory Adversarial Attacks on Sequential Decision-Makers and Countermeasures
Authors: Tim Franzmeyer
Stephen Marcus McAleer
Joao F. Henriques
Philip Torr
Jakob Nicolaus Foerster
Adel Bibi
Christian Schroeder de Witt
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
adversarial attacks

Abstract: Autonomous decision-making agents deployed in the real world need to be robust against possible adversarial attacks on sensory inputs. Existing work on adversarial attacks focuses on the notion of perceptual invariance popular in computer vision. We observe that such attacks can often be detected by victim agents, since they result in action-observation sequences that are not consistent with the dynamics of the environment. Furthermore, real-world agents, such as physical robots, commonly operate under human supervisors who are not susceptible to such attacks. We propose to instead focus on attacks that are statistically undetectable. Specifically, we propose illusory attacks, a novel class of adversarial attack that is consistent with the environment dynamics. We introduce a novel algorithm that can learn illusory attacks end-to-end. We empirically verify that our algorithm generates attacks that, in contrast to current methods, are undetectable to both AI
agents with an environment dynamics model, as well as to humans. Furthermore, we show that existing robustification approaches are relatively ineffective against illusory attacks. Our findings highlight the need to ensure that real-world AI, and human-AI, systems are designed to make it difficult to corrupt sensory observations in ways that are consistent with the environment dynamics.

Under review as a conference paper at ICLR 2023
ILLUSORY ADVERSARIAL ATTACKS ON SEQUENTIAL DECISION-MAKERS AND COUNTERMEASURES
Anonymous authors
Paper under double-blind review
ABSTRACT
Autonomous decision-making agents deployed in the real world need to be robust against possible adversarial attacks on sensory inputs. Existing work on adversarial attacks focuses on the notion of perceptual invariance popular in computer vision. We observe that such attacks can often be detected by victim agents, since they result in oostrvntobsnnatiinn sequences that are not consistent with the dynamics of the environment. Furthermore, real-world agents, such as physical robots, commonly operate under human supervisors who are not susceptible to such attacks. We propose to instead focus on attacks that are statistically undetectable. Specifically, we propose illusory attacks, a novel class of adversarial attack that is consistent with the environment dynamics. We introduce novel algorithm that can learn illusory attacks end-to-end. We empirically verify that our algorithm generates attacks that, in contrast to current methods, undetectable to both AI agents with an environment dynamics model, as well as to humans. Furthermore, we show that existing robustification approaches illusory attacks. Our findings highlight the need to ensure that real-world AI, and human-AI, systems are designed to make it difficult to corrupt sensory observations in ways that are consistent with the environment dynamics.
relatively ineffective against
INTRODUCTION
Deep reinforcement learning algorithms (Mnih et al. 2015; Schulman et al.) 2017:: Haarnoja et al. 2018 Salimans et al. 2017, DQN, PPO. SAC, ES) have found applications across a number of sequential decision- making problems, ranging from simulated and real- robotics (Todorov cet al. 2012: Andrychowicz et al. (2200) to arcade games (Mnih et al., 2015). It has recently been found, however, that deep neural network control policies conditioning on high-dimensional sensory input are prone to adversarial attacks the input observations, which poses threats security and safety-critical applications (Kos & Song 2017 Huang et al.) 2017) and thus motivates research into robust learning algorithms (Zhang al. 2020).
Existing frameworks of attacks on sequential decision-makers are largely inspired by pioneering work perceptually invariant attacks in supervised computer vision settings (Ilahi al. 2021) Unlike supervised settings, however, sequential serision-making settings involve temporally- environment interactions which give rise temporally-cone correllled sequences of observations. this paper, we argue that their failure to take temporal consistency considerations into account renders existing Abssevvation-spaa adversarial attacks ineffective many settings of practical interest.
Al agentt often have access to an approximate or exact world model (Sutton 2022 Ha & Schmidhuber 2018). addition, humans have the ability to perform "intuitive physics" (Hamrick et al.) (0116). using robust but qualitative internal models of the world (Battaglia et al., 2013). This makes it possible to use one's understanding of the world to detect a large range of existing adversarial attacks, by spotting inconsistencies in observation sequences. Existing observation space adversarial attacks (Ilahi et al.) 2021; Chen et al.) 2019 Qiaoben et al. 2021 Sun et al.) 2020) ignore these facts, As a result, these attacks produce observation trajectories that are inconsistent with the dynamics of the unattacked environment.
The consequences of this are twofold. First, state-of-the-aa adversarial attacks can be trivially detected by victim agents with access to even low-accuracy world models, i.c,, models of the environment dynamics. Second, in cases where Al agents are supervised by humans, humans ele can

---

Under review as a conference paper at ICLR 2023
action
action
MNP attack
SA-MDP
attack
Illusory
attack
Figure 1: Illustration of different classes of adversarial attacks 6-cell Gridworld environment. Victim agents need to reach the green target as quickly as possible without traversing the orange lava. The adversary replaces the original victim observation (blue triangle) with adversarial observation (red triangle). Note that in all scenarios, the victim ends up in the lava, upon which the episode terminates. However, the observations under the MNP and SA-MDP attacks (see Section 3 are not consistent with the forward actions taken by the agent, i.c. the red arrow jumps between cells (top row), respectively incorrectly stays in the same position (middle row). contrast, the observations under the proposed illusory attack (bottom row) are consistent with the environment dynamics.
be able to detect these adversarial attacks. In other security contexts such as computer networks, real-world adversarial attacks typically attempt to evade detection an effort to avoid triggering security escalations, and this must be taken into account by the defender. While this insight has been exploited in the cybersecurity community (Provos) 2001 Claffy & Dainotti] 2012, Cazorla al.. 2018), undetectable adversarial attacks on sequential decision-makers and their defences have not yet been systematically explored in the community.
In this paper, we introduce illusory attacks, novel class of adversarial attacks sequential decision- makers that result in observation space attacks that are consistent with environment dynamics. We show that illusory attacks can succeed where existing attacks do not, and, in particular, can successfully fool humans. Illusory attacks therefore pose specific safety and security threat to human-AI interaction and human-in-ther -e-ttt--hoooppp settings (Schirner al. 2013 Zerilli al. 2019) as they may can be undetectable even to attentive human supervisors.
Illusory attacks seek remain undetected and must hence attack the vietim by replacing its perceived reality by an internally coherent alternative one. We show that perfect (statistically undetectable) illusory attacks exist variety of environments. We then present the W-illusory attack framework (see Figure for illustration), which introduces a world-model consistency optimisation objective that encourages the resultant victim's 's action-observatton histories be consistent with the environment dynamics. We show that W-illusory attacks can be efficiently learnt, and are undetectable by humans, unlike MNP (Kumar et al., 202T) or SA-MDP attacks (Zhang al. 202I).
We empirically demonstrate that existing victim robustification methods are largely ineffective against illusory attacks. This leads us to suggest that the existence of reality feedback, i.e., observation channels that are hardened against adversarial interference, may can play decisive role in certifying the safety of real-world AI, and human-Al, systems.
Our work makes the following contributions:
We formalise perfect illusory attacks, a novel framework for undetectable adversarial attacks (see Section 4.2) We give examples in common benchmark environments in Section 5.3
We introduce W-illusory attacks, a computationally feasible learning algorithm for adversarial attacks that generate victim action-observation sequences that are consistent with the unperturbed environment dynamics (see Section 4.5).
We show that, compared to state-of-the-ard adversarial attacks, W-illusory attacks are significantly less likely to be detected by Al agents (Section 5.2). as well as by humans (Section[5.4
We demonstrate that victim robustification against W-illusory attacks is challenging unless the environment admits reality feedback (see Section 5.5
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Adversarial attacks literature originates in non-sequential decision-making applications such as image classification (Szegedy et al., 2013), where the goal is to find perturbations å for given classifier such that J yields different predictions for x and x + ō, despite the difference between ær and x: + 5 being imperceptible to humans. To enforce the imperceptibility requirement, such works use simple minimum-norm perturbations constraints as a proxy (Goodfellow et al.) 2014).
This line of work has been extended to adversarial attacks on sequential decision-makinn maainn
agents (Oikarinen et al. 202 Kumar et al. 2022]] Cohen
R02InCohen et al. 2019 Everett et al. 2021 Moosavi-
Dezfooli et al. [2017] Chaubey et al.) 2020; Wu et al. 2021 Iaahi et al. 2021 Chen et al. 2019]]
Qiaoben et al. [202]] Sun et al., [020 Zhang et al. 2020). largely building upon the minimum noom
perturbation (MNP) framework. In the MNP framework, the adversary can modify the victim's 's
observations up to fixed step- or episode-wise perturbation budget. While many adversarial attack frameworks require white-box access to the victim's 's policy, /Zhang et al.] (2020) and [Sun et al. (2021) use reinforcement learning to learn adversarial policies, thus require only black box access to the victim's 's policy. Assuming different setting, Hussenot et al. (2019) introduce a class of adversaries for which a unique mask is precomputed and added to the agents' observation at every time step. Our framework differs from these previous works both in that it takes into account the temporal correlation of sequences of observations, and in that it focuses on learning adversarial attacks that are undetectable.
Work towards robust agents in sequential decision environments uses randomized smoothing Ku mar et al. 2021 Wu et af.] [202T) test-time hardening by computing confidence bounds (Everett al. 202 D, training with adversarial loss functions (Oikarinen et al. 2021a). and co-training with adversarial agents (Zhang et al. 202I). We compare against, and build upon, this line work.
Another extensive body of work focuses on detecting adversarial attacks. Lin et al. (017) develop an action- cittionconditioned frame module that allows agents to detect adversarial attacks by comparing both the module's e's action distribution with the realised action distribution. (Tekgul et al. (2021) detect adversaries by evaluating the feasibility of past action sequences. Compared to these works, our proposed methods hinge detectability based on stronger notions of statistical indistinguishabbiityy Further, it focuses on learning undetectable attacks, rather than detecting adversarial attacks.
There exists only a limited body of literature on undetectable (or "stealthy"") adversarial attacks et al. 2019 Sun et al. 2020) on sequential-decto settial--eessiaaaiiinn agents, none of which achieves statistical indistinguishabilyy nor takes into account lemporally-extendee observation dependencies; [Huang &
Zhu 2019) consider stealthy attacks on reward signals used for RL. In cybersecurity, similar ideas
have been explored for sensor attacks on linear control systems
(Mo & Sinopoli] 2010 Pasqualetti
et al. 2015
3 BACKGROUND
MDP and POMDP. A Markov decision process (Bellman 1958 MDP) is a tuple (S, A. p.
where S is a discrete continuous state space, A is discrete or continuous action space. p : S x Ax - [o. 1] is the state transition probability, r: S x A R > [o, is a probabilistic reward function, and y [0, 1] is scalar discount factor. Á Markov decision process proceeds by sampling
initial state 8p p(0) E S, upon which the agent may-take takes an action dg ~ T(0)) E A triggering a state transition 81 p(sn, a1) and returning reward ro r(11, ao; sn). Here, S A [0 is a stochastic agent policy.
POMDP) is latent MDP A. together with a stochastic observation function Z : S + [0, 1]. In a POMDP, the agent cannot observe states s, directly, but has to act based on observations 2, Z(s)).
partially-observabee MDP Aström 1965
We further define state- action-observatton trajectory of length n as + (20, ag. 7ny (zg, ag. ru...., 2,), and the distribution over trajectories generated by a policy acting in (PO)MDP & as (T) We assume that agents do not have access to the reward signal at test- time. We denote that two distributions (T), (T) are identical up to reward by writing
Py (T) (7) _ (T).
3

---

Under review as a conference paper at ICLR 2023
Observation-space adversarial attacks on sequential decision making agents. Observation- space adversarial attacks (Kumar et al. 2021) consider the scenario where an adversary can manipu- late the observation given victim agent at test-time, thus introducing partial observability to the victim's s test-time environment. The victim's 's observation at time step is then given as ž, = 2; + et instead of 2,, where 62 IR is the perturbation introduced by the adversary. When modeling the adversarial perturbations as generated by state-conditioncl adversarial agent with policy Tado we can write ã (5)). In many scenarios, the size of the perturbation bounded by budget B > O; for simplicity, we consider zero-sum adversarial attacks, where the adversary minimizes the expected return of the victim. This yields the following definition of an optimal state-oondiiooned observation-spuce adversary:
T-1 Σ
Tadv arg min ET,
Era
s.
1ell S B,
(I)
where a2 ~ To 12x,,, 2e ~ Tado (8r), 8841 p(-1st, at).
This concept be formalized defined as a tuple of the victim MDP (8, A. p.r. r, together with an adversary v P(S) and
state-adversary MDP SA-MDP (Zang al. 2020). which is
mapping B + S, limiting supp v(|ss) E B(s).
Robustifying against adversarial attacks with co-training. The optimal policy n; in an SA-MDP thereby defines the optimal robust policy a victim under the presence of optimal adversary v*. To find such optimally robust victim policies, Zhang et al. (2021) ATLA) introduce co-training approach to SA-MDPs, where victim and adversary are trained turn while keeping the other's policy fixed.
4 METHODS
In this paper, tectable. Our setting differs from traditional observation-space attacks (see Section 3 in three ways. First, assume that the victim has access world model m., Second, we-anoumme perturbation budgets may unconstrained we consider unconstrained perturbation budgets. Third, assume that the victim has costly contingency option which it can safely execute once it has detected the presence of test-time adversary.
concerned with finding adversarial attack policies that unde-
While seemingly unusual, we argue that the above three assumptions are realistic in many settings: Often victims can learn accurate world models from unperturbed train-time samples, or rely on the experience of human supervisor for anomaly detection. addition, in some settings the constraint on small perturbations seems arbitrary, e.g. when an attacker is replacing the entire video feed of surveillance camera. Lastly, contingency options Al agents are a common feature of real-world systems, such as e.g. self-driving cars that can hand-over to human controller, security level escalation procedures in cyber-physical systems (Züren 202T)
4.1 DETECTABILITY OF ADVERSARIAL ATTACKS
An adversarial attack be detected if the altacked MDP g', which includes the adversary with Tade, can be statistically distinguished from the train-time MDP E.
Definition 4.1 (Statistical indistinguishabiltty.. Let E == AS p.r. r,y) be a MDP with horizon T. Let P be partially obbeevvbbl discrete-time stochastic control process, with observation space
Z = S, action space A, and horizon T. Let क S A (S A)T + [0, 1] be sampling
policy that may possibly conditions on the whole action-observation history. Then P is statistically
indistinguishable from E under if and only if P... Pr.e PE.E= _ PP.T.
4.2 UNDETECTABLE ADVERSARIAL ATTACKS
We define perfect illusory attacks, i.c. adversarial attacks that are undetectable, as those where and E' are statistically indistinguishable. Importantly, since in our setting reward signals cannot be
4

---

Under review as a conference paper at ICLR 2023
observed at test-time, indistinguishabiiity need only hold for observation transitions. For simplicity, we only consider zero-sum adversarial attacks, i.c those that try to minimise the expected return of the victim.
Definition 4.2 (Perfect illusory attack). A perfect illusory attack on an environment for given victim policy is an adversarial policy Ta such that & and E' are statistically indistigguishable under Fy where E' is the environment included by aa attacking E.
Note that the definition of perfect illusory attacks makes no reference the adversary's performance (victim reward). We prove that non-trivial (i.e. those actually changing the observation) perfect illusory attacks do exist some, but not all, victim policy/environmen pairs (see Appendix (711). We provide further examples of perfect illusory attacks in Section
Definition 4.3 (Optimal illusory attack). An illusory attack with Wado (E, m.)) is optimal if and only if it is the perfect illusory attack with the highest expected adversarial return, i.e.
T Σ 1=
πI addv arg min By)P((E .T.)
rt
Tude
s.t. PE... PE PE
Per
where 72 is the victim reward.
As the constraints in Equation 2 are difficult to optimise exactly, we instead consider a relaxed weighted objective
TT-1 Σ 0
ado
arg min (T
Tt XD (Pe,n,,, Per,..)
Tade
where À is a hyper- srper-parameter that determines the weighing of the two objectives, and D some distance measure between distributions, such the KL-divergence.
Adversaries that trade off between remaining undetected and minimizing the victim reward. In most environments, adversary will need to trade-off its expected adversarial return with the desire to avoid detection. In practice, non-perfect illusory attacks can be effective, for example when victims only have access to only n test-time samples from gx. In fact, n can usually be actively upper-bounded by the adversary's preferences engage not. In addition, the victim's world model may not be exaet be imperfect, particularly if learnt, or if relying on human experience. We define illusory attacks under such relaxations Appendix 7.2
attacks may not have to be perfeettet in-order to be effective
4.3 WORLD MODEL-CONSISTENN ADVERSARIAL ATTACKS.
Analytically optimising the distance D (Pe,s,, .e... is usually intractable in all but the smallest environments. To arrive at tractable optimisation objective, we instead consider a two-step test that allows to determine whether Pe. Per
Theorem 4.4 (Testing MDP equivalence). To determine whether a discrete-time stochastic control process M is equivalent to given Markov process is sufficient test both whether M has the Markov property, and whether M's A's transition probabilities match those of & Shi et al.) 2020).
We further assume from now that the victim has access to an exact, approximate, world model m,,(2/2 12, a) of &, which estimates the probability of the next observations given the current observa- tion and action taken in E. We also assume the victim has an estimate but in E'
a) of the same quantity
We now define W-illusory attacks, a relaxation of optimal illusory attacks (see Definition 4.3), for which the victim's world model estimate matches the estimate of the transition probabilities.
5

---

Under review as a conference paper at ICLR 2023
Definition 4.5 (W-illusory attacks). A W-illusory attack on & is adversarial attack that is consistent with the victim's model of the observation trannsitinnnn probabilities my. Formally,
T-1
T7 adv = arg min Esp
Σn 1=
m.((122; P Vzt, 01 ~ (E
We note that world-model consistency alone does nor imply the Markov property, as P does not account for long term correlations. So importantly, W-illusory attacks can in general change the distribution of trajectories.
4.4 LEARNING WORLD MODEL CONSISTENT ADVERSARIAL ATTACKS.
We now define the IllusoryMDDP (I-MDP) used to learn illusory adversarial attacks. This is the MDP the adversary trains
Definition 4.6 (lllusory MDP (Z-MDP)), Given MDP & := (8,A,p,r,)) with episode horizon T and victim policy an Illusory MDP (I-MDP) & := I(E,,) x) consists of a tuple ( (S.A.D.F.7)
where = S S A T-1
3(3 āt, 8t) = (ael act)p(se4110) 0z), Vl,
(5)
and
F(Fe4115441, a, 5,) = (aal@aa (xr
st), Vt.
(6)
As we assume zero-sum adversarial settings, r++1 = fi+1- Note that the victim's 's policy conditions on its action-observation history, and not just on its last observation, as contains sequence detector. As result, the Z-MDP differs from the SA-MDP (Zhang et al.) (021 in that the state & : {St, acc,, acr), on which the adversarial policy wade is conditioned, is given by the state MDP E, as well as the victim's 'ss action-obserratton history.
Definition 4.7 (Learning W-illusory attacks). Given fixed victim policy Tu. W-illusory attack on can be learnt by optimising
TT -1
max EIT Ta
Σ
rt 2XD (ma (a;-1, (a())-1)) na (8t, at T (a(a-11)))
Here, we choose the Læ-distance, i.c. |m (a,-1, (a;-)))) ael2 for deterministic environ- ments, where m, is the adversary's world model. For stochastic environments, possible choice would be DKL, however, alternatively, one could simply employ the -iiiiumm-ikklhhood objective Mor-Maako temporal correlations the adversarial policy can be suppressed by restricting the horizon of the adversarial policy's 's observation space Š, which could be detected by the victim (Shi et al. 2020).
log ma (d (, (1.
4.5 REALITY FEEDBACK
So far we have assumed that the adversary can corrupt all parts of the victim's observation. However, in practice, the AI agent may-be bealeee- could receive limited amount of unperturbed environment ("reality") feedback through robust channels which it may, in principle, the presence of an adversary.
to act optimally in
Definition 4.8 (Reality feedback). We defin reality feedback Ç as a part of the victim's observation in Ex that cannot be corrupted by the adversary, i.c. we assume that Z : Zo Zc. where the adversary has access to zd E Zo but not 28 E Zc.

---

Under review as a conference paper at ICLR 2023
Figure 2: In CartPole, the agent aims to balance the brown pole by adjusting the position of the black cart. In the perfect illusory attack depicted above, the agents observations (left) appear unper- turbed while the true system fails (right).
Figure In Pendulum, the agent must apply torque to stabilize the pendulum upright.
EXPERIMENTAL EVALUATION
In Section 5.2 we first provide empirical evidence that state-of-the-arr attacks on sequential decision- makers can be detected by AI victims with learnt world model. In Section 5.3 we then provide examples of both perfect illusory attacks (see Definition 4.2), as well as Iearned W-illusory attacks (see Definition 4.5). In Section 5.4 give evidence that humans efficiently detect state-of-the. art adversarial attacks, but may can be unable to detect W-illusory attacks. Lastly, we investigate the effectiveness of state-of-the-art adversarial robustification algorithms against W-illusory attacks (Section 5.5). We find that effective defences against illusory attacks may can require the victim to have access unperturbed feedback channels (Section 5.5
5.1 EXPERIMENTAL SETUP
We compare illusory attacks against two adversarial attack benchmarks, (Kumar et al. 2021 MNP) and (Zhang et al. 202T SA-MDP) (see Section 3). Together, we consider these attacks to be repre- sentative of existing state-of-ther approaches. Both MNP and SA-MDP attacks require choosing perturbation budgets, which we here fix at B E {0.05, 0.2} relative to normalised observation vectors. To allow for fair comparisons, where necessary we also impose these budgets illusory attacks. We consider two choices of vietim policy: naive policies learnt through train-time return maximisation in the unattacked environment, and ATLA policies learnt through co-training (Zhang et al. 2021 ATLA). We evaluate all methods on three simulated environments. Gridworld (see Figure Ip involves navigating to a "good" grid cell while avoiding "bad" grid cell. The OpenAI gym (Brockman et al. 2016) environments CartPole and Pendulum (see Figures 2 and 3 feature continuous control tasks that require balaneing a pole with discrete actions, and swinging up pendulum with continuous torques. In accordance with Section 4.4, we assume that the adversary has access to an exact model of the environment dynamics m,- To ensure reproducibility, averaged over five runs with independent random seeds, and report full results with standard deviations Table 7.5.1 in the Appendix. We release our code at annoymiized
generated
results
5.2 DETECTING STATE-OF-THER-ARY ADVERSARIAL ATTACKS WITH LEARNED WORLD MODELS
We demonstrate that existing state-of theoofftt-atttt art adversarial attacks can be detected using learned world models. To this end, we use train-time trajectory rollouts to train a small neural network based world model r. for each environment (see Appendix 7.3). We then evaluate each m., on episodes from holdout set of test-time trajectories from both the respective attacked and unattacked environment and under both naive and ATLA victim policies (see Section [5.Tp. To account for the neural network models' finite precision, regard transitions to be consistent with m. if they lie within the 5 confidence bound, which we computed on a held-out set of train-time transitions.
We find that both MNP and SA-MDP attacks frequently generate world-model inconsistent transitions, which allows for > 93% of attacked episodes to be detected with high accuracy, given the false- positive rate of m. unattacked environments of < 1%. Figure I gives graphical illustration of attacks on Gridworld. We provide videos of all attacks in the supplementary material (see Appendix 7.5.3).
7

---

Under review as a conference paper at ICLR 2023
Detection accuracy
Relative adversary score under different defences
Attack
Budget /
naive
ATLA
none
smoothing
ATLAA
ATL
Pendulum
SA-MDP (Zhang et al. Illusory (ours)
2021
97%
93%
100 %
87.4 %
97.9 %
100 %
0.2
0%
0%
86.2
71.7
76.7 %
75.3
SA-MDP (hhng et al.
2021
96%
95%
100 %
60.6 %
95.9 %
100 %
0.05
Illusory (ours)
1%
2%
83.9
58.5 %
83.5 %
90.4
CartPole
MNP Kumar et al. 2021
100%
98.1 18.1% %
97.6 %
SA-MDP (Zhang et al. 2021
0.2
99%
96%
99.9 %
93.9 %
100 %
99.9 %
Illusory (ours)
2%
3%
100
99.9 %
99.8
MNP Kumar et al. 2021
96%
3.1 %-
65.2 %
SA-MDP (Zhang al. 2021
0.05
94%
100%
100
76.9
97 %
98.5
Illusory (ours)
100
97.5 %
92.6
HalfChectah (MuloCo)
SA-MDP (Zhang et al. 2021
100%
100%
85.4%
tbd
tbd %
thd
0.05
Illusory (ours)
0%
0%
32.1%
tbd
tbd %
tbd
SA-MDP Zhang et al. 2021
100%
100%
100
tbd %
tbd %
tbd %
0.2
Illusory (ours)
0%
0%
48.1%
tbd
tbd %
tbd
Hopper (MuJoCo)
SA-MDP (Zhang et al. Mussryy (ours)
2021
100%
100%
100%
tbd %
tbd %
tbd
0.05
0%
0%
48.8%
tbd
thd %
thd
SA-MDP (Zhang et al. 2021
100%
100%
100%
tbd % tbd
tbd %
tbd %
0.2
Illusory (ours)
0%
0%
53.4%
tbd %
thd
Table I: Empirical detection accuracy and victim scores for different adversarial attacks and possible defences on Pendulum (top two rows), and CartPole (bottom three rows). Illusory attacks are able to achieve almost perfect undetectability while only sacrificing small amount of adversarial performance.
5.3 LEARNING UNDETECTABLE ADVERSARIAL ATTACKS
We first show concrete examples of non-trivial perfect illusory attacks as defined in Section 4.2 We then empirically investigate W-illusory attacks according to Definition 4.5
Perfect illusory attacks. We find that Pendulum and CartPole admit perfect, i.e. entirely unde- tectable, illusory attacks (see Definition 4.2) that significantly impact the victim's performance independently of its policy. For Cartpole, such adversarial policies can be constructed by starting off with a coordinate transform 20 = Sg. and then using the adversary's 's world model to generate all subsequent observations (see Figure Z}. Constructing the adversary's policy proceeds equivalently for Pendulum, as both have symmetric initial state distributions.
The existenee strong uttucky for two populur benehmark environments may astonish, as effective perfect illusory attacks do not always exist (see- Appendix I.D. This may suggest that robotic control environments of practical interest could be
undetectable adverssaria attacks::
W-illusory attacks. We now demonstrate how illusory attacks can be learnt rather than analytically constructed. We employ Algorithm I (see Appendix (7.5), which implements W-illusory adversarial attacks. W-illusory adversarial attacks trade off between the objectives of detectability and adversarial performance. In all of our environments, we choose our detectability objective to be proportional to the -orm of the distance between the correct next observation, i.e. the next observation expected by the victim, and the adversarial observation generated by the adversary (cf. Definition 4.7). We find the weighting parameter
10 using a grid search (ablation in Appendix 7.5.2).
Note that while Algorithm I could, in principle, discover the exact perfect illusory attacks presented above for both Cartpole and Pendulum, we do find this be the case practice. We suggest that this is due the convergence to sub-optimal local equilibria.
8

---

Under review as a conference paper at ICLR 2023
Environment
both
Pendulum
CartPole
P[false no attack)
34.2 + 11.4
31.5 + 10.5
37.0 + 12.3
Faaas W - illusory (Ours))
32,4 1: 10.8
37.0 1: 12.3
27.7 1 9.3
Pifalse SA-MDP)
81.4 + 27.2
96.3 + 32.1
66.7 + 22.2
Pifalse MNP)
83.3 + 27.8
83.3 + 27.8
Table 2: Results for the study with human participants. The answer "false" signifies that the participants did not believe that a given video displayed the ground truth environment. Note that participants are not more significantly skilled at detecting illusory attacks over non-attacks.
Detection. We now compare and contrast W-illussory attacks, MNP attacks and SA-MDP attacks for perturbation budgets 3 = 0.05, 0.2. We observe that illusory attacks are much less likely to be detected by the learned world model m. (see section 5.2 than MNP and SA-MDP attacks (see Table D.. This finding is independent of the perturbation budget B.
ATLA victims are based on a recurrent neural network policy, enabling them, in principle, to learn an implicit model of the environment dynamics during train-time. Co-training with such world model-aware victims, could, in theory, result in world-model consistent adversarial attacks. However, we empirically find this not to be the case: As shown in Table attacks on ATLA agents are detected just as likely as those on naive agents. This underlines the need for explicit world model consistency constraints during adversarial training (see Definition 4.7 We provide videos of all types of attacks for different seeds in the supplementary material (see Appendix 7.5.3).
Victim performance under different attacks. We find that the average reward achieved by the victim (Table [1 4th column) is generally lower for 3 = 0.2, and also generally lower for SA-MDP attacks than for W-illusory attacks. This is to be expected, as W-illusory attacks trade off between minimizing the reward of the victim and generating consistent observation sequences, while an SA-MDP attack solely minimizes the victim reward. Given that illusory attacks are significantly more constrained than SA-MDP attacks, we consider it surprising that undetectability be traded in for only minor decreases in adversarial performance.
5.4 FOOLING HUMANS WITH ILLUSORY ATTACKS
We perform controlled experiment with n 10 human participants in order te-eellect evidence for our eluim that while humans ean efficiently deteet state of the adversaria attaeks, they--ennee detect )4' Hlltasorr--ttteee to investigate whether humans unfamiliar with adver- sarial attacks are able to detect W-illusory attacks. Participants were first shown example videos of both CartPole and Pendulum trajectories, exposing them to environment-speciftt dynamics. Participants were then asked to classify a random mixture of videos of test-time trajectories from both unattacked and attacked CartPole and Pendulum environments; further details in Appendix (7.6
We found that participants were able to visually classify MNP and SA-MDP attacks from videos with high accuracy, similarly for unattacked trajectories. However, humans struggled to correctly classify W-illusory attacks more accurately than non-attacks (see Table 2 attacked. We establish the statistical significance of these results using a 2-test statistic (see Section 7.6). which suggests that humans may be unable to detect such illusory attacks.
5.5 ROBUSTIFYING AGAINST ILLUSORY ATTACKS
we now investigate how victims casn be robustified. We first consider two standard defences against adversarial attacks, randomized smoothing (Kumar et al. (221)) and adversarial pretraining (ATLA, (Zhang et al.) 2021 We also add an ablation of ATLA that co-trains the victim with W-illusory adversaries.
Having established the effectiveness of illusory attacks in Sections 5.3 and 5.4
For a given budget 3. we define the relative adversary score as the percentage deduction in the expected return of the unattacked victim, normalized by the deduction achieved by the best-performing adversary. As indicated by blank spots, MNP attacks cannot be applied to Pendulum (continuous action space), nor ATL. based agents (I.STM policy).

---

Under review as a conference paper at ICLR 2023
As summarised in Table 1 find that defences generally more effective restoring victim performance at 3 = 0.05 For 3 = 0.2, all defences result in only minor improvements. With only minor variations, these trends hold across all three robustification methods we deployed. Note that the victim performance achieved by applying randomized smoothing for / = 0.2 CartPole is equivalent to that of a random policy. Overall, findings suggest that none of the robustification methods studied are particularly effective against illusory attacks.
Exploiting reality feedback. We conclude our empirical investigations by exploring the importance of reality feedback in victim robustification (see Section 4.5) We establish two reality feedback scenarios for CartPole: one where the cart observation is unperturbed and one where the obser- vation of the pole is unperturbed. We find that robustifying the victim agent through pre-training may--ee-essnntite to enable enables victim agents to learn how to effectively use reality feedback. Our results further suggest that having access informative reality feedback channels can allow for significant robustification. See Appendix 7.7 for further details.
CONCLUSION AND FUTURE WORK
In this paper, we introduce perfect illusory attacks, a novel class of adversarial attacks on sequential decision-makers that is undetectable. Importantly, show that W-illusory attacks, unlike state-of- the-art adversarial attacks, can fool both Al victim agents with access to a model of the environment dynamics, well as human supervisors. Our results imply that robustification against illusory attacks is mostly ineffective unless the victim has access to reality feedback, making the provisioning of such hardened feedback channels imperative to designers of safe real-world autonomous or Human-in-the-looe decision-making systems.
Future work may investigate illusory attacks in large-scale and partially observable victim envi- ronments, Similarly, evaluations on real-world settings would be beneficial, e.g. using sim-to-rew methodology. Our work further does investigate effect constraining the number of test-time interactions between adversary and victim, or the effect of world model imperfections. Future work must also develop novel approaches and algorithms for victim robustification against illusory attacks. In particular, such methods should be able to maximise the utility of existing reality feedback channels through test-time information gathering behaviour. Lastly, we believe that a conceptual framework unifying both budgeted and illusory attacks could be of value to the community.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 2020.
Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45):18327- 18332, November 2013. doi: 10. 10.1073/pnas htt.ppa........222. 1306572110. Publisher: Proceedings of the National Academy of Sciences.
Richard Bellman. Dynamic programming and stochastic control processes. Information and Control, 1(3):228-23 239, September 1958. ISSN 0019-9958. doi: (2.0016/50019-9558588888800
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Lorena Cazorla, Cristina Alcaraz, and Javier Lopez. Cyber Stealth Attacks in Critical Information Infrastructures. IEEE Systems Jourmal, 12(2):1778 17001999779 June 2018. ISSN 1937-9234. doi: 10.1109//SYST.22155.25522 24876644 Conference Name: IEEE Systems Journal.
Ashutosh Chaubey, Nikhil Agrawal, Kavya Barnwal, Keerat K. Guliani, and Pramod Mehta. Universal Adversarial Perturbations: Survey. Technical report, May 2020.
Tong Chen, Jiqiang Liu, Yingxiao Xiang, Wenjia Niu, Endong Tong, and Zhen Han. Adversarial attack and defense in reinforcement learning-froe fromm AI security view. Cybersecurity, 2019.
Kc Claffy and Alberto Dainotti. Analysis of an Internet-wide Stealth Scan from Botnet. 2012.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. International Conference on Machine Learning. PMLR, 2019.
Michael Everett, Bjorn Lutjens, and Jonathan P. How. Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learning. IEEE Transactions Neural Networks and Learning Systems, 2021.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6672, 2014.
David Ha and Jürgen Schmidhuber. World Models. March 2018. doi: 10.5281/2enodo.1200 arXiv:1803.0 10122 [cs, stat].
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learming. PMLR, 2018.
Jessica B Hamrick, Peter W Battaglia, Thomas Griffiths, and Joshua Tenenbaum. Inferring mass in complex scenes by mental simulation. Cognition, 157:61-76, 2016.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284 2017.
Yunhan Huang and Quanyan Zhu. Deceptive reinforcement learning under adversarial manipulations on cost signals. In International Conference on Decision and Game Theory for Security, pp- 217-237. Springer, 2019.
Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Copycat: Taking control of neural policies with constant attacks. arXiv preprint arXiv:1905.12288. 2019.
Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and countermeasures for adversarial attacks deep reinforcement learning. IEEE Transactions on Artificial Intelligence, 3(2):90-109, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.66 14/2.6980, 2014.
11

---

Under review as a conference paper at ICLR 2023
Jemej Kos and Dawn Song. Delving into adversarial attacks deep policies. arXiv preprint arXiv:170S.06452. 2017.
Aounon Kumar, Alexander Levine, and Soheil Feizi. Policy Smoothing for Provably Robust Rein- forcement Learning. Technical report, arXiv, 2021.
Shasha Li, Ajaya Neupane, Sujoy Paul, Chengyu Song, Srikanth V. Krishnamurthy, Amit K. Roy Chowdhury, and Ananthram Swami. Adversarial Perturbations Against Real-Time Video Classifi- cation Systems. In Proceedings 2019 Network and Distributed System Security Symposium, 2019. doi: 10. 10.144222ndss.20993
180,00880,000458 [cs, stat].
Yen-Chen Lin, Ming- Yu Liu, Min Sun, and Jia-Bin Huang. Detecting adversarial attacks on neural network policies with visual foresight. arXiv preprint arXiv: 10.000011 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning, nature, 2015.
Yilin Mo and Bruno Sinopoli. False data injection attacks in control systems. In Preprints of the 1st workshop on Secure Control Systems, volume 2010.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal ad- versarial perturbations. Technical Report arXiv:1610.08400, arXiv, March 2017. arXiv:1610.08401 [es, stat] type: article.
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui- Wei Weng. Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Processing Systems, 34, 2021a.
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui- Wei Weng. Robust Deep Reinforcement Learning through Adversarial Loss. In Advances in Neural Information Processing Systems, volume 34, pp.
Curran Associates, Inc., 2021b.
Fabio Pasqualetti. Florian Dorfler, and Francesco Bullo. Control-theoretic methods for cyberphysical security: Geometric principles for optimal cross-layer resilient control systems. IEEE Control Systems Magazine, (5((1):10-113)) 2015.
Niels Provos. Defending against statistical steganalysis. In Proceedings of the 10th conference USENIX Security Symposium Volume 10, SSYM'01, USA, August 2001. USENIX Association.
You Qiaoben, Chengyang Ying. Xinning Zhou, Hang Su, Jun Zhu, and Bo Zhang. Understanding Adversarial Attacks on Observations in Deep Reinforcement Learning. Technical report, arXiv, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. baable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint 1 2017.
Gunar Schirner, Deniz Erdogmus, Kaushik Chowdhury, and Taskin Padir. The Future of Human-in-
the-Loop tebb--Physical Systems. Computer, 46(1):36 45, January 2013. ISSN 1558-0814, doi: 10.90 Conference Name: Computer.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. Technical report, arXiv, August 2017.
Chengchun Shi, Runzhe Wan, Rui Song, Wenbin Lu, and Ling Leng. Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Making, February 2020. 1111v:2002.01751 [cs, stat].
Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, and Yang Liu. Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning, May 2020. arXiv:2005.07009 [cs].
12

---

Under review as a conference paper at ICLR 2023
Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl. In International Conference on Learning Representations, 2021. 2021.
Richard S. Sutton. The Quest for Common Model of the Intelligent Decision Maker, June 2022. arXiv:2202. 13252 [cs].
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks, arXiv preprint arXiv:1312.6199 2013.
Buse GA Tekgul, Shelly Wang, Samuel Marchal, and Asokan. Real-time attacks against deep reinforcement learning policies. arXiv preprint arXiv:2106.08746, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model- control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, October 2012.
Fan Wu, Linyi Li, Zijian Huang, Y. Vorobeychik, Ding Zhao, and Bo Li. CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing. ArXiv, 2021.
John Zerilli, Alistair Knott, James Maclaurin, and Colin Gavaghan. Algorithmic Decision-Making and the Control Problem. Minds and Machines, 29(4):555-577 -788, December 2019. ISSN 1572-8641. doi: 10.1007/s 022--019-99513-7
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations. In Advances in Neural Information Processing Systems, volume 33, pp. 21024-21037. Curran Associates, Inc., 2020.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust Reinforcement Learning on State Observations with Learned Optimal Adversary, January 2021. arXiv:2101.08452 [cs, stat].
Arno van Züren. IT security escalation management must be part of every contingency plan, August 2021.
Karl Johan Aström. Optimal Control of Markov Processes with Incomplete State Information I. Journal of Mathematical Analysis and Applications, 1965.
13

---

Under review as a conference paper at ICLR 2023
7 APPENDIX
7.1 PROOF ON THE EXISTENCE OF PERFECT ILLUSORY ATTACKS
3
3
4
2
4
(a)
(b)
Figure 4: An environment for which perfect illusory attacks do exist (left), and one for which they do not exist (right).
Proof that perfect illusory attacks exist. We consider an example MDP (see Figure 4A) where a victim starts in node 1 or 2 each with probability 1 and can go up, down, or right in both states I and The episode terminates immediately with a return the agent receives reward of +1 if it reaches state 3 within maximum of 2 steps. The optimal victim policy is therefore to take paths 1 + if starting in state 1, and take one of the two possible paths 2 + 3 otherwise. The victim observes the labelled state graph, as well its current state label. Clearly, choosing Tado (1) = and (2) = constitutes a perfect illusory attack in this environment.
should the victim reach state 4. Otherwise,
Proof that perfect illusory attacks don't It always exist. To show that some environments do not admit perfect illusory attacks, consider the modificd environment in Figure Ap. Here, clearly a timestep- conditioned victim policy that takes the action sequence right) independently of observations cannot be perfectly attacked.
7.2 FINITE-SAMPLE ILLUSORY ATTACKS
Definition 7.1 Saimle Illusory Attacks). A finite-sample illusory attack is adversarial attack that is not detectable by the victim within given test-time interaction budget n. Formally, an n-sample illusory attack on an environment with victim policy is an adversarial policy wadv for which $D, #6 such that simultaneously
D (P 1()(P))))(( < v, ~ P P TT) and D (P(TNE)))P(E.)) 2 6, VT, ~ P
(E',TT Ty. dv).
Note that this definition is symmetric between adversary and victim. In practice, the victim will choose a D, 5 such that its average test-time performance is maximised. As the victim is, however, not able to exactly infer Eg n. [R] as does not have access to the adversary's policy, the adversary in turn cannot exactly deduce how exactly the victim chooses D, 8. Therefore, in practice, the adversary might wish to choose an isotropic D, such as DKL, and construct policy n, that minimises this metric as far as possible given desired performance-detecaabllty trade-off.
7.3 WORLDMODEL TO DETECT ADVERSARIAL ATTACKS
Setup. We assume that the victim is trained in the unperturbed MDP Munperturbedd for k episodes of length n. During training, the agent records the observed environment transition tuples f; (s;, Gg 81+1) and stores these in a set Dump = {t ii=k(x=-1 At test time, the victim executes the learned policy and is attacked by adversary agent, which replaces the true observation 2 by the adversarial observation é,
14

---

Under review as a conference paper at ICLR 2023
Learning dynamies model. We use the the set of unperturbed transitions Dunp to learn an approximate dynamics model T( 11)))), We implement I as a Mulli-LayerPerreetron with two hidden layers of size 10, and train the model with an 12 loss for 10 epochs, using a learning rate of 0.001 and the ADAM (Kingma & Bay 2014) optimizer. We then evaluate the model on held-out set of the training observations and compute the 5o confidence bound on the lo distance, which we refer to as c. We use then use c as a threshold to detect adversarial attacks. The high confidence ensures that the false positive rate remains low.
Note that an adversary with access to my could attempt to learn universal perturbations (Moosavi Dezfooli et al. 2017) that simultancously attack both n. and mi.. However, adversaries without access to m. may be unable to exploit its imperfections and instead need to assume is perfect and hence justifying our comparison to solely policy- based MNP/SA-MDP attacks.
Using a dynamics model to detect adversarial attacks. We then evaluate the accuracy in detecting adversarial attacks for all scenarios listed in Table ! More specifically, at every time step. the victim uses the learned world model mh to predict the next observation given the previous observation and action taken. If the lo distance between the predicted observation and the actual (adversarial) observation is larger than the threshold the victim classifies the episode as adversarially attacked. Results for this detection mechanism are given Table
In the six-cell gridworld environment depicted in Figure 1. we similarly evaluated detection accuracy, but only considered naive victim agents. We found that 100% of adversarial attacks could be detected in this environment, while 0% of illusory attacks were detected. We show the resulting sequences of observations and true states in Figure]
7.4 PERFECT ILLUSORY ATTACKS
We implemented perfect illusory attacks in Pendulum and CartPole by setting the first observation 20 to the negative of the true first state, i.c. 20 = S0. We compute all subsequent observations 21, ... 2T with the worldmodel ma- As the distribution over initial states is symmetric in both environments, this approach of generating illusory attacks satisfies the conditions of perfect illusory attack (see Definition 4.2p. We provide videos of the generated perfect illusory attacks in the supplementary material in the respective folder. We found that the average reward achieved by the victim under a perfect illusory attack in CartPole was 41.2 + 6.8, while was -628.32 60.1 Pendulum. Figure illustrates a perfect illusory attack in CartPole.
7.5 LEARNING W-ILLUSORY ATTACKS WITH REINFORCEMENT LEARNING
We next describe the algorithm used to learn W-illusory adversarial attacks and the training procedures used compute the results Table I
Setup. We use the CartPole and Pendulum gym (Brockman et al.) 2016) environments. The world model is implemented using the given physics computed for a new environment step. We normalise observations in both environments by the maximum absolute observation. We train the victim with PPO (Schulman et al. 2017) and use the implementation PPO given in Raffin et al. (202T). while not making any changes to the given hyperparameters. In both environments victim for million environment steps. We implement the ATLA (Zhang et al. 202T) victim by co-training with adversary agent, and follow the original implementation of the authors P We implement the ablation of ATLA (Zhang et al. (202I) that trains the victim with an illusory adversary by merely replacing the ATLA adversary with an illusory adversary, which was implemented as stated in algorithm 7.3 For co-training, we alternate between training the victim and the ad adversary agent every 400 environment steps. This parameter has likewise been found in small evaluation study, and was chosen as it yields non-oscillating behaviour. We further investigated different ratios between training steps of the adversary and training steps of the victim, but found that a ratio of one, i.e. equal training of both, yields the most stable and results in co-training.
train the naive
We implement the illusory adversary agent with SAC (Haarnoja et al.) 2018), where we likewise use the implementation given in Raffin et al. 2222I). We initially ran small study and investigated four
https:://githbb.ccm
1
15

---

Under review as a conference paper at ICLR 2023
different algorithms possible implementations for the adversary agent, where found that SAC yields best performance and training stability.
We train all adversarial attacks for three million environment steps. We implemented randomized smoothing as a standard defence against adversarial attacks on RL agents, as introduced in Kumar al. 20221). We use the author's original implementation In contrast to the authors, compute the budget with respect to the normalised observations, but otherwise follow the per-step budget definition in Kumar al. (2021).
Algorithm W-illusory adversarial training
Input:
illusory reward weighing parameter X, victim Initialize nI " parametrised be neural network with parameters s/. while iteration < N do
environment enee adversary worldmodel Max
policy iterations N
50 env()
20
1 addo (80)
Initial perturbation only depends on initial system state > Adversarial observation is clipped to permissible range
ao = T (clip((c=)
51, , done- = env((a))
1 -
adde
No illusory reward at first step
while done=False do
2t Tedo (8t, 21 a(-1)
Adversary conditions past information
04 = To [=(cpp((za)
8t41, dome env(ar)
[+1 --ii1 X || (2-1
Illusory reward added
end while
update π adv
> Adversary policy updated using past experience
end while
7.5.1 FULL RESULTS TABLE FOR CONTRASTING OF DIFFERENT ATTACKS
Table 3: Full results table top row Pendulum, bottom row CartPole
Artacks detected
Vietim reward under different defences
budget
naive
ATLA
Rone
smoothing
ATLA abl
SA-MDP Zhang et 2021
Illusory (ours)
0.2
97%
93%
-1387.0
69.9 -1170.1
119.0
-1188.3 70.4
-1354.6 = 107.1 -1020.4
-1029.4 106.7
-1428.3 91.5
SA-MDP Zhang et Illusory (OUN)
2021
0.05
96%
95%
-797.2
-408.4 + 146.6
-757.2 + 109.3
-722.2 +: 30.8
115.8
340.7
none
0%
0%
-189.4 1. 13.4
228.2 1. 43.8
220.9 58.8
MNP Kumar el al.
2021
100%
18.3 1. 20.8
20.8 1 8.7
SA-MDP Zliing 2021
0.2
99%
96%
9.3 1::
39.01 = 10.7
9.2 0.1
9.7 0.6
Illusory (ouns)
0.6
MNP Kumar et al 2021
96%
485.0 485.0
180.3 33.6
SA-MEP 2hiing el 2021
9.4
122.5
24.2
Illusory (ours)
0%
2%
9.3 1 0.1
165.4 46.3
21.4 1 6.0
45.4 56.5
none
500.0
Table 7.5.1 shows all results for all scenarios evaluated; we report mean and standard deviation. Note that each scenario was evaluated for five random seeds; Table [ in the main paper is derived from Table 7.5.
7.5.2 ABLATION: INFLUENCE OF LAMBDA ON RESULTS
We investigated the influence of the À parameter in small study. Generally, the À parameter determines the trade off of the illusory adversary between generating consistent observation seguences, and minimizing the victim's 's reward. We considered À E {1, 10, 100), with B = 0.2. Table shows that smaller weighing parameters À generally lead to lower victim rewards, which is in line with the fact that for À 0, the objective of the adversary solely to minimise the victim reward.
httpps://openrevieev.t wtwp:rrwwwwweeeeeeeeeeeeooomeeee d- 8NHrJ
16

---

Under review as a conference paper at ICLR 2023
Table Reward achieved by the victim across different weighing parameter X.
Weighing parameter A
Environment
100
CartPole
9.0 0.3
9.0
22.2
Pendulum
1382.87 + 158.3
11700.1 + 67.5
616.1 120.0
Table 5: Results from our study with human participants.
Environment
both
Pendulum
CariPole
P(false
attack)
34.2 i: 11.4
31.5 = 10.5
37.0 12.3
P P(false
SA-MDP)
81.4 + 27.2
96.3 + 32.1
66.7 + 22.2
P[Jalse
W illusory)
32.4 10.8
37.0 i 12.3
27.7 1 9.3
P false
MNP)
83.3 27.8
83.3 27.8
P(false)
54.2 18.1
54.9 + 18.3
53.7 17.9
7.5.3 VIDEOS OF ALL ADVERSARIAL ATTACKS
We provide videos for different seeds for all adversarial attacks in the supplementary material. The folders named respectively. Note that the left side shows the environment as observed by the victim agent, while the right side shows the true environment state. Please further note that the episode in CartPole terminates once the angle has crossed a specific threshold, which is why the episodes might appear to be shorter than otherwise expected. All videos were generated for a budget 3 0.2.
7.6
HUMAN STUDY
We performed controlled experiment with = 10 human participants. All participants were graduate-level university students, while none had prior knowledge about the objective the study. Participants participated voluntarily; minutes. Participants were handed a slide show which contained all relevant information. This slide show is included in the supplementary material in the respective folder. We further add the sheet with ground truth labels for all video sequences.
estimate the time needed per participant
around 15
After consenting to participate, participants were provided with the slide show and online sheet to indicate their answers. The study was self-paced and done by the participants without further supervision. We further advised participants not do the study if they would not feel comfortable at any point in time.
The first 18 videos in the slide show are 100-frame long sequences of the Pendulum environment, iee six each for either no adversarial attack, a W-illusory attack or an SA-MDP attack (Zhang et al. 202T). The subsequent 24 10-frame long sequences of the CartPole environment include the same attacks as for Pendulum, plus an additional six sequences under an MNP attack (Kumar et al. (021). Note that MNP attacks cannot be applied to the Pendulum environment, as it has continuous action space. For all attacks, the budget / was 0.2. Within both groups, the order of the videos shown was randomised. Subjects had to choose for each video between correct and false, where false indicates that subjects find that the system observed in the respective video is not the same as the one observed in the introductory video. Table includes all empirically estimated probabilities of classifying segment from given class as false, split up also by the different environments CartPole and Pendulum.
Result analysis. We find that subjects reported an unexpectedly high proportion (one third) of unattacked sequences as false. Subjects classified sequences which were under an MNP or SA- MDP (Zhang et al., 2020) attack as not from the same system with a probability higher than 80%, while categorizing illusory attacks as false with probability of only 32.4%. We computed the double-sided z-statistics and were able to reject both the hypothesis that p( I f'alse SA-MDP) = p( p(false) none) and the hypothesis that p(false MNP) = p(false none) for 0.05, while the hypothesis that p(,alls W-illusory) = p(false none) cannot be rejected. We conclude that subjects were able to
17

---

Under review as a conference paper at ICLR 2023
Table 6: Reward achieved by victim for different reality feedback scenarios.
Victim agent
reality feedback
naive
ATLA abl.
pole
9.84 + 0.1
182.44 + 36.9
cart 8.83 0.3
15.54 6.6
distinguish SA-MDP and MNP attacks from unattacked sequences, while being unable distinguish illusory attacks from unattacked sequences.
7.7 REALITY FEEDBACK
Setup. We evaluate the importance of realism feedback the CartPole environment by investigating two possible scenarios. Note that the observation in CartPole is given four-dimensional vector of the pole angle and angular velocity, as well as cart position and velocity, the first test scenario, the victim correctly observes the pole, while the adversary can perturb the observation of the cart; the second scenario vice versa. We investigate two test cases for each scenario: First, attacking a naive victim, and second, attacking agent pretrained with co-training.
Results and discussion. Table shows that the reward achieved by the victim is generally higher when pretrained with co-training. We hypothesize that this pretraining enables the agent to learn how to utilize the reality feedback effectively. The achieved victim performance when reality feedback contains information about the pole more than 10 times larger than when containing information on the carl instead. This seems intuitive, as the observation of the pole appears much more useful for the task of stabilizing the pole, and underlines the importance of equipping agents with strong reality feedback channels.
18


Reviews:
[{"review_id": "pleBNgrIN3", "reviewer": "Reviewer_G6kZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies adversarial attacks on sequential decision-making policies, with a focus on *statistically undetectable* attacks. The authors assume exact knowledge of the world model, and introduce a novel class of adversarial attacks called illusory attacks, which are consistant with the world dynamics and thus more stealthy. This paper formulates the illusory attacks, and propose a feasible learning algorithm, W-illusory attacks, to generate illusory attacks. Experiments on simple control tasks show that the proposed attack and less detectable to humans and AI agents than state-of-the-art attacks.", "summary_of_the_review": "The formulation and methods proposed by the paper are interesting. But I do have some concerns on the significance of the problem, as well as the realisticity of the key assumptions. In short, it is not clear to me whether the illusory attack and the world model assumption is practical in real-world scenarios and high-dimensional environments.", "strength_and_weaknesses": "Strengths:\n\n1. The idea of statistically consistant attacks is interesting, and to my knowledge, novel in the literature.\n2. The paper provides theoretical justifications that perfect illusory attacks exist for some but not all policy-environment pairs. \n3. Provided human study are useful for understanding the detectability of adversarial attacks.\n4. The proposed algorithm makes intuitive sense, and the empirical results on Cartpole and Pendulum do show the effectiveness of the algorithm.\n\nWeaknesses:\n\n1. I do not agree with some claims made by the paper. In particular, most existing adversarial attacks are imperceptible to humans [1,2], which is an important motivation of adversarial attacks. Most literature of adversarial RL studies the adversarial perturbations on observation, which lies in a high-dimensional space (e.g. images [2,3,4]). These perturbations are usually undetectable by humans, unless special care is taken.\n2. The assumption of an exact world model is not very realistic, especially in the senarions where adversarial attacks are concerning. In simulators, we may have an exact world model. However, it is less possible and also less dangerous that a local simulator is attacked. Adversarial examples are more critical during the interaction with real-world environments where observation may be noisy and exposed to outside attackers. Under these real-world scenarios, the access to a world model is unrealistic. Even learning a good model can be challenging in these environments.\n3. The experiments are on simple environments. Can the authors also provide results on larger scale environments like Atari games, or at least MuJoCo environments?\n4. Some related works are missing. For example, [4] proposes a stronger adversarial attack that SA-MDP [3]. Can the authors compare the proposed attack with [4]?\n\n\n\n[1] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. \"Explaining and harnessing adversarial examples.\"\n\n[2] Huang, Sandy, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. \"Adversarial attacks on neural network policies.\"\n\n[3] Zhang, Huan, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. \"Robust deep reinforcement learning against adversarial perturbations on state observations.\n\n[4] Sun, Yanchao, Ruijie Zheng, Yongyuan Liang, and Furong Huang. \"Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl.\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hUj4AQ6TQD", "reviewer": "Reviewer_YTMM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies adversarial attacks on the state observation (sensor inputs) channel of a RL agent. Different from previous work (in particular [Zhang et al., 2020]), the authors consider the stealthiness of the adversarial attacks. The contribution of the paper is that it defines the concept of detectability for adversarial attacks on state observation and proposes an algorithm that can compute and carry out such undetectable attacks. \n\n", "summary_of_the_review": "The paper studies a novel and interesting problem of adversarial attacks on state observations in RL. The adversarial attacks are crafted in a way to avoid being detected. But the problem formulation has flaws. For example, the conditions posed for statistically indistinguishability, based on the reviewer's understanding, are very strong. The reviewer hopes to see some examples to support the definition. The assumption is not unrealistic itself. But it is unrealistic that the victim will use the algorithm discussed in the paper.  The detector discusses in the paper is not a detector one usually finds in cyber-security. It does not consider statistical evidence. There are other concerns regarding the detection mechanisms discussed in previous sections.", "strength_and_weaknesses": "The problem studied in this paper novel and meaningful. That is to consider the stealthiness of adversarial attacks and the trade-off between stealthiness and effectiveness of the attacks. Stealthiness and detection are the important twins in security problems, especially security problems for sequential decision-making systems where attacks are launched not just one time but sequentially.\n\nBut the definition of indistinguishability and the detection mechanisms proposed throughout the paper are questionable. The assumptions made in Section 4 are contradicting with the RL algorithm that the author later choose as the focus of the study.\n\n1. The definition of statistical indistinguishability is a strong one. If statistical indistinguishability holds, no detection algorithms can detect such attacks. But the attacks that can avoid being detected by some detection mechanisms don't necessary need to be statistically indistinguishable. The use of statistical indistinguishability as a condition to craft adversarial attacks can lead to no attacking strategy satisfying the condition. Consider an MDP with **continuous** state space. Can the authors give an example of an MDP and an attacking strategy $\\pi_{adv}(s)\\neq 0$ for some $s\\in\\mathcal{S}$ such that $\\mathbb{P}_{\\pi,\\mathcal{E}} = \\mathbb{P}_{\\pi,\\mathcal{E}'},\\forall (\\mathcal{S}\\times\\mathcal{A})^T$? To make the example simple, we can assume T=2. Such examples can help the readers better understand how strong the definition is.\n\n2.   I agree with the authors that the assumption of the victim knowing a world model $m_v$ is not unrealistic. \"victims can learn accurate world models from unperturbed train-time samples\". However, if the victim knows the world model, what is the point of interacting with the world to observe the state? The victim can leverage a more efficient algorithm (value iteration, policy iteration + function approximation if the state space is large) instead of the algorithms discussed in the paper. \n\n3. The detection mechanism used in the experiment is included in the paragraph starting with \"Using a dynamics model to detect adversarial attacks.\" Please highlight the detection mechanism since it is an important factor for the experiments. If I understand it correctly, the detection mechanism considers observations in a single step and an attack is detected if the predicted observation and the actual observation is larger than a threshold $c$. Does this detection technique come from a reference? I have **three** concerns regarding the mechanism. 1. It does not consider the whole trajectory history to detect 2. how do you measure distance for discrete state space (how to set threshold c)? 3. if c is small, even if there is not attack, the actual observation can be different from the predicted observation given the stochasticity of the model.\n\nOther minor comments:\n* I came across a paper that studies adversarial attacks on rewards with a definition of stealthiness in it (C1). As a reader, I am curious about the difference between undetectability in attacks in reward and attacks in state observations? \n\n* In optimal control or MDP or POMDP, adversarial attacks have been investigated by many researchers (C2, C3). These attacks are sometimes called false data injection attacks or sensor attacks. The detector of attacks is usually based on statistical evidence instead of single observations.\n\n* Avoid using \"may\" in your statement. For example, in definition 4.1, a sampling policy that may conidtion on the whole history. The authors can say \"a sampling policy where T can be infinity\" or \"a sampling policy that can condition on the whole action-observation history\". In section 5 \"may be unable to detect W-illusory attacks\". Instead of using \"may\", scientific writing should specify under what conditions, humans are able to detect W-illusory attacks and under what conditions, human are not able to do so. Similar examples of using \"may\" can be found throughout the paper.\n\n\n[C1] Huang, Yunhan, and Quanyan Zhu. \"Deceptive reinforcement learning under adversarial manipulations on cost signals.\" International Conference on Decision and Game Theory for Security. Springer, Cham, 2019.   \n\n[C2] Mo, Yilin, and Bruno Sinopoli. \"False data injection attacks in control systems.\" Preprints of the 1st workshop on Secure Control Systems. Vol. 1. 2010.\n\n[C3] Pasqualetti, Fabio, Florian Dorfler, and Francesco Bullo. \"Control-theoretic methods for cyberphysical security: Geometric principles for optimal cross-layer resilient control systems.\" IEEE Control Systems Magazine 35.1 (2015): 110-127.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VA8lWVUHjcY", "reviewer": "Reviewer_58cU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary:\nThis paper emphasizes developing statistically undetectable attacks for autonomous decision-making agents. The authors develop a novel class of illusory attacks that are consistent with environment dynamics. Their results show that illusory attacks can easily fool humans, unlike the previous attacks in the literature. They compare illusory attacks with other attacks and show their performance under different defense techniques.\n\n\nQuestions and comments:\n1. In Table 2, what is the perturbation budget used for the attacks?\n2. Based on Section 3, I think in definition 4.1 and the rest of the text, it should be P_{\\varepsilon, \\pi} and not P_{\\pi, \\varepsilon} for consistency.\n3. In Section 3, \\gamma is not defined.\n4. In definition 4.3, it says the highest expected return but shows minimization in equation (2). Is this correct? If yes, why? Also, why is \"optimal illusory attack\" defined? I do not see it being used in any part of the text.\n5. I would suggest using \\tau \\sim P_{\\varepsilon, \\pi} instead of \\tau \\sim (\\varepsilon, \\pi}) in equation (3).\n6. In equations (3, 7), should the summand not be till T-1 and not T?\n7. In equation (7), it should be \\lambda instead of \\gamma, right?\n8. On page 8, it should be Table 1 and not Table 5.3 for consistency.\n9. I think it's a better idea to replace Table 1 with Table 7.5.1 in the main text since the absolute reward values are easier to compare.", "summary_of_the_review": "As mentioned in the weaknesses, I am not convinced if W-illusory attacks are undetectable. I think evaluations in more complex environments are required to strengthen the paper. The proposed attack is not robust to reality feedback. Hence, I do not feel that the contributions in this paper are very significant.", "strength_and_weaknesses": "Strengths:\n1. Focuses on developing attacks that are consistent with the dynamics of the environment.\n2. Develops a novel class of illusory attacks that are consistent with the dynamics. Introduces the concept of statistical indistinguishability for stochastic control processes.\n3. Well-structured sections. The supplementary material contains all the relevant videos related to human study.\n\nWeaknesses:\n1. I think that the human study setup is biased. The samples given in the supplementary show that it's easy to detect attacked vs. unattacked videos. All the unattacked videos labeled in the study balance the pendulum or cart pole perfectly, while all the attacked videos do not do that. Hence, it should be easy to detect attacked vs. unattacked videos for these simple environments.\n2. I think illusory vs. unattacked videos can also be figured out without much difficulty. For example, only in illusory attacks the cart pole moves very quickly. In the rest of the attacks/ unattacked videos, the cart poles do not move as fast as in illusory attack videos.\n3. Given the above weaknesses, I am not sure why the detection accuracies are low for the illusory attacks in Table 1. What is the naive detection algorithm that is used for Table 1? Also, W-illusory attacks are not \"statistically indistinguishable\", unlike perfect illusory attacks, right?\n4. I am not convinced if good W-illusory attacks should always exist. Pendulum and cart pole are very simple environments. I would suggest adding experiments with more complex environments. Not sure about transferability to other tasks.\n5. Reality feedback is a practical and obvious way to defend against illusory attacks. Hence, the attack proposed in this paper does not appear to be strong.\n\nResponse to rebuttal\n================\n\nThanks for the rebuttal; however, some of my concerns remain. Could you please try to address these?\n\n1. I am not convinced about illusory attacks not being detectable to \"attentive human supervisors\". I think a non-expert in RL can figure out attacked vs. unattacked in cases of illusory attacks. Quoting from my review -- \"For example, only in illusory attacks the cart pole moves very quickly. In the rest of the attacks/ unattacked videos, the cart poles do not move as fast as in illusory attack videos.\". I think  this requires a closer investigation.\n\n2. Unanswered question from my review -- \"Also, why is \"optimal illusory attack\" defined? I do not see it being used in any part of the text.\"\n\nEdit:\nAfter the discussion phase, I have decided to increase my scores since the rebuttal cleared most of my questions. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8jVfgTYbyC", "reviewer": "Reviewer_pRsS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies test-time attacks on reinforcement learning agents. It focuses on attacks that are statistically undetectable, and proposes novel attack models that aim to preserve consistency of trajectories with the environment dynamics. The paper develops a new optimization framework for generating such attacks and experimentally validates the effectiveness of these attacks, called illusory attacks. The experiments test the efficacy of the proposed approach in terms of: i) detectability of adversarial attacks via statistical consistency checks, ii) detectability of adversarial attacks via visual inspection (i.e., human-subject studies), and iii) susceptibility of robustly trained RL agents to adversarial attacks. The experimental results indicate that the proposed attack approach yields lower detectability rates compared to prior works. ", "summary_of_the_review": "Overall, I enjoyed reading the paper. The core idea seem quite intriguing and novel. Having said that, I believe that the paper could benefit from having more experiments (i.e., additional environments), and that it could better explain the limitations of this work and the practicality of the proposed approach. Apart from that, some parts of the paper could be polished. ", "strength_and_weaknesses": "**Strengths of the paper**:\n- To my knowledge, the attack models considered in this work has not been studied in the literature on test-time attacks against RL agents. The attack models are well motivated and complement those from prior work. Instead of focusing on $L_p$ norm-based attack models, the paper advocates models that are statistically undetectable. \n- The paper introduces a formal framework for studying these attack models, as well as an optimization problem for finding an optimal illusory attacks. The optimization problem aims to minimized the victim's return, while minimizing a distance function that measures the inconsistency between generated trajectories and the environment dynamics. \n- The paper also conducts a human-subject experiment to test the detectability of this attack model via visual inspection. This validation techniques appears to be novel when it comes to adversarial attacks on RL agents.  \n\n---\n**Weaknesses of the paper**: \n- The experiments are primarily based on two simple environments, Pendulum and CartPole. In contrast, prior work, e.g., Zhang et al. 2021, has studies more complex environments, such as MuJoCo. More experiments would be useful in order to understand the scalability of the approach. Also, given that one the experiments involves human-subjects, IRB may be required; the paper doesn't seem to report if the study has an IRB approval.  \n- In general, it is not clear what are the computational properties of the proposed optimization framework. The I-MDP model does not scale well with the time horizon, and given that the experiments are only based on two simple environments, it is not clear how practical this approach is. Additionally, the attack optimization problem (7) seems to require the environment/world model. Some discussion on the practicality of the approach would be useful to have. \n- Some parts of the formal framework are not entirely clear and may contain typos. Firstly, I don't fully understand why rewards are not included in trajectory \\tau when one measure consistency with the true environment, nor why \\tilde S does not include rewards. It's not immediately clear to me that the victim cannot detect this attack by inspecting received rewards. Some discussion on this would be useful. Secondly, definition 4.1 uses $\\mathcal E '$ but does not specify it. Thirdly, Eq. (3), (5), (6) and (7) may not be precise. E.g., why do we minimize $-\\lambda D()$ in (3)? Moreover, transition probabilities $p(.)$ should be functions of $s$, not $\\tilde s$, but in (5) we have $p(\\tilde s_{t+1}|\\tilde s_{t}, a_{t})$. Similarly, I don't understand how we obtain (6) since rewards function $r$ should take elements from $S$ not $\\tilde S$; it is also not clear why this equality holds as we have $\\tilde r_{t+1}$ on LHS and $r_{t+1}$ on RHS. In (7), why do we have $r_{t}$ and not $\\tilde r_{t}$ and why does $\\pi_a$ depend only on $a_{t-1}$ and $\\pi_v(a_{t-1})$ but not the whole history? The notation also seems to be ambiguous. E.g., is $a_t$ the action of the attacker or the victim, and does $\\pi_v(a_{t-1})$ denote stochastic or deterministic policy?   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject