Illusory Attacks: Information-theoretic detectability matters in adversarial attacks
Authors: Tim Franzmeyer
Stephen Marcus McAleer
Joao F. Henriques
Jakob Nicolaus Foerster
Philip Torr
Adel Bibi
Christian Schroeder de Witt
Venue: ICLR 2024 spotlight (2024)
Keywords: sequential decision making
adversarial attacks
robust human-AI systems
robust mixed-autonomy systems
Primary area: reinforcement learning

Abstract: Autonomous agents deployed in the real world need to be robust against adversarial attacks on sensory inputs. 
Robustifying agent policies requires anticipating the strongest attacks possible.
We demonstrate that existing observation-space attacks on reinforcement learning agents have a common weakness: while effective, their lack of information-theoretic detectability constraints makes them \textit{detectable} using automated means or human inspection. 
Detectability is undesirable to adversaries as it may trigger security escalations.
We introduce \textit{\eattacks{}}, a novel form of adversarial attack on sequential decision-makers that is both effective and of $\epsilon-$bounded statistical detectability. 
We propose a novel dual ascent algorithm to learn such attacks end-to-end.
Compared to existing attacks, we empirically find \eattacks{} to be significantly harder to detect with automated methods, and a small study with human participants\footnote{IRB approval under reference R84123/RE001} suggests they are similarly harder to detect for humans. 
Our findings suggest the need for better anomaly detectors, as well as effective hardware- and system-level defenses. The project website can be found at https://tinyurl.com/illusory-attacks.

Published as a conference paper at ICLR 2024
ILLUSORY ATTACKS: INFORMATION-THEOREIC DE- TECTABILITY MATTERS IN ADVERSARIAL ATTACKS
Tim Franzmeyer
Stephen McAleer!
Joao F. Henriques
Jakob Foerster!
Philip Torr
Adel Bibit
Christian Schroeder de Witt
University of Oxford
Carneeiee Mellon University
ABSTRACT
Autonomous agents deployed in the real world need to be robust against adver- sarial attacks on sensory inputs. Robustifying agent policies requires anticipating the strongest attacks possible. We demonstrate that existing observation-spac at- tacks on reinforcement learning agents have common weakness: while effective, their lack of information-theorele detectability constraints makes them detectable using automated means or human inspection. Detectability is undesirable to ad- versaries as it may trigger security escalations. We introduce e-illusory attacks, a novel form of adversarial attack on sequential decision-maker fective and of e-bounded statistical detectability. We propose a novel dual ascent algorithm to learn such attacks end-to-end. Compared to existing attacks, we em- pirically find e-illusory attacks be significantly harder to detect with automated methods, and a small study with human participante suggests they similarly harder to detect for humans. Our findings suggest the need for better anomaly detectors, well as effective hardware- and system-level defenses. The project website can be found at httpt:////// http:///iinurlrlrr Aloooyyyla.......oooaoooaaaaaae attacks
that is both ef-
INTRODUCTION
The sophistication of attacks cyber-physical systems is increasing, driven in small part by the proliferation of increasingly powerful commercial cyber attack tools (NSCS) 2023). Al-driven technologies, such as virtual reality systems (Adams et al. 2018) and large-language model assis- tants (Radford et al. 2019) are opening up additional attack surfaces. Further examples are deep learning methods autonomous driving tasks (Ren al.] 2013 Shi et al. 2019 Minaee et al. 2022), deep reinforcement learning methods for robotics (Todorov et al. 2012 Andrychowicz et al., 2020 and nuclear fusion (Degrave et al. (022). While Al can be used for cyber defense, the threat from automated Al-driven cyber attacks is thought to be significant (Buchanan et al. (203) and the future balance between automated attacks and defenses hard to predict (Hoffman [2021).
Beyond its beneficial use, deep reinforcement learning has also been proposed as method for learning flexible automated attacks on Al-driven sequential decision makers (Ilahi et al.) (02T). A common approach to countering adversarial attacks is to use policy robustification (Kumar et al.) 2021 Wu et al. 2021). This approach can be effective, as visualized by the red-circled budgets in Fig. However, as show in this work, for observation -ppceee attacks with larger budgets (grey circles in Fig. D. robustification can be ineffective. The practical feasibility of large budget attacks has been highlighted in domains such as visual sensor attacks (Cao al. 2021 patch attacks), as well as botnet evasion attacks (Merkli, 2020 Schroeder de Witt et al.) 2021). This highlights the importance of a two-step defense process which the first step employs anomaly detection (Haider et al. 2023), followed by attack-mitige mitigatinii security escalations. This coincides with common cy- bersecurity practice, where intrusion detection systems allow for the implementation of mitigating contingency actions as a defense strategy (Cazorla et 2018) Therefore, effective cyber attackers are known to prioritize detection avoidance (Langner 2011 STUXNET 417 attack).
"frtim @ robots.ox..coo ackuk
'RB approval under reference R84123/RF001

---

Published as a conference paper at ICLR 2024
Adversary Performance
Examples
Victim detects attack and uses contingency option
c-0.3 Attack
Attack Budget
Robustfication is effective
Robustfication is not aftective
Classic
Illusory Attack
..... E=0.3 Illusory Attack
detector
Classic Adversarial Attack
KL divergence between tralningg distribution and attacked distribution
E=0.1
E-0.3
Figure 1: We adversary performance (reduction in the victim's 's reward) mapped against the KL divergence between the unattacked training and the attacked test distribution. Attacks with a small L2 attack budget (indicated by small circles) can be defended against using randomized smoothing, and attacks with large KL divergence can be defended against by triggering contingency options upon detection of the attack (purple shaded area). Illusory attacks (blue) can achieve significantly higher performance than classic ad adversarial attacks (black), as they allow to limit the KL divergence and thereby avoid detection.
In this paper, we study the information cntemaation-thoreticc limits of the detectability of automated attacks on cyber-physical systems. To this end, we introduce a novel observation-space illusory attack frame- work, which imposes novel cormmation-theorete detectability constraint adversarial attacks that is grounded efformation-theorttie steganalysis (Cachin 1998). Unlike existing frameworks, the illusory attack framework naturally allows attackers exploit environment stochasticity in order to generate effective attacks that are hard (e-illusory), or even impossible (perfect illusory) to detect.
penretically-groundee dual ascent algorithm and scalable estimators for learning il- lusory attacks. On a variety of RL benchmark problems, we show that illusory attacks can exhibit much better performance against victim agents equipped with state-of-the-art detectors than conven- tional attacks. Lastly, in a controlled study with human participants, we demonstrate that illusory attacks can be significantly harder detect visually than existing attacks, owing their seeming preservation of physical dynamics. Our findings suggest that software-level defenses against au- tomated attacks alone might not sufficiently effective, and that system-wide wide and hardware-level robustification may be required for adequate security protection (Wylde) (02T). We also suggest that better anomaly detectors for scquential-decision decision matiiiiiioomiiingngg agents should be developed.
We propose
Our work makes the following contributions:
We formalize the novel illusory attack framework with information-theorrticall grounded attack detectability constraints.
We propose dual ascent algorithm and scalable estimator to learn illusory attacks in high- dimensional control environments.
We show that illusory attacks can be effective against victims with distribution detectors, whereas existing attacks can be detected and hence are ineffective.
out-of-
We show that illusory attacks are significantly harder to detect by human visual inspection.
RELATED WORK
Please see Appendix A.I for additional related work.
The adversarial attack literature originates in image classification (Szegedy et al. 2013), where attacks commonly need to be visually imperceptible. Visual imperceptibility is commonly proxied by simple pixel-space minimum-norm perturbation (MNP) constraints (Goodfellow al. 2014
2

---

Published as a conference paper at ICLR 2024
Madry et al. 2023). Several defenses against MNP attacks have been proposed (Das et al. 2018
Xu et a 2018 Samangouei et al. 2023] Xie et al. 2023). Various strands of research in cyber security concern adversarial patch (AP) attacks that do not require access to all the sensor pixels,
and commonly assume that the attack target can be physically modified (Hykholt et al., 2018 Cao
al. 2021). Illusory attacks differ from both MNP and AP attacks in that they are information-
theoretically grounded and undetectable even for large budgets.
MNP attacks have been extended adversarial attacks on sequential decision-making agents (Chen et al.) 2019b; Iaahi et al. 2022]] Qiaoben et al. 202T). In the sequential MNP frame- work, the adversary can modify the victim observations up to a step- or episode- wise perturbation budget, both in white-box, as well as in black box settings. Zhang et al. (2020) and (Sn et al. (2021) use reinforcement learning to learn adversarial policies that require only black-box access to the victim policy. Work towards robust sequential-decision making uses techniques such as random- ized smoothing (Kumar et al., 2021] Wu et al. 202T), test-time hardening by computing confidence bounds (Everett et al. [002), training with adversarial loss functions (Oikarinen et al. 202I). and
co-training with adversarial agents (Zhng et al. 2021a Dennis et al. 2020 Lanier et al. 2022). We
compare against and build upon this work.
Another body of work focuses on detection and detectability of learnt adversarial attacks on sequential decision makers. Perhaps most closely related to work, RRsso & Proutiere (2022) study action-space attacks on low-dimensional stochastic control systems and consider information- theoretic detection (Basseville al.. 1993 Lai 1998; Tartakovsky al. 2014) based stochastic cquivalence between the resulting trajectories. We instead investigate nigh-dimensional observation- space attacks, and consider learned detectors, as well as humans.
Al-driven attacks on humans and human-operated infrastructure, such as social networks, are an active area of research (Tsipras et al. 2018). (Ye & Li 2020) consider data privacy and security issues in the age of personal human assistants, and Ariza et al. (023) investigate automated social engincering attacks on professional social networks using chatbots. Illusory attacks signify that such automated attacks may be learnt such as to be hard to detect, or indeed undetectable.
Within information-theoretii hypothesis testing. Bayesian optimal experimental design CChlloner & Verdinelli, 1995) studies optimisation objectives that share similarities with the illusory attack objective. Foster et al. (2019) introduce several classes of fast EIG estimators by building on ideas from amortized variational inference. [Shen & Huan (2022) use deep reinforcement learning for sequential Bayesian experiment design.
3 BACKGROUND AND NOTATION
We denote probability distribution over set X' P(X)), and an unnamed probability distribution as P(.). The empty set is denoted by 0, the indicator function by 1, and the Dirac delta function restrict theoretical
by 6(-). Kleene closures denoted by (-)". For ease of exposition, treatment to probability distributions of finite support where not otherwise indicated.
3.1 MDP AND POMDP.
A Markov decision process (MDP) Bellman 1958) is tuple (S.A, T, where S is the finite" non-empty state space, the finite non-empty action space, p S P(S) is the probabilistic state-transinn tate-trrnsition function, and r x A ~ (((R)) is lower-bounded reward function. Starting from a state 8 E S time f, an action a E A taken by the agent policy ग + P(A) effects a transition
to state $,41 ~ p(- (a) and the emission of a reward rt+1 ~ r(-1 a,). The initial system state at time = 0 is drawn as 80 ~ p(-10). For simplicity, we consider episodes of infinite horizon and hence introduce discount factor < 7 < 1. In a partially observable MDP (Aström) 1965 Kaelbling et al. 1998, POMDP) (S, A, Ω O,p,r, (2,r,7) 7), the agent does not directly observe the system state 82 but instead receives an observation o; ~ (-1ss) where
S + P(2) an observation function and Ω is finite non-empty observation space. In line with standard literature (Monahan
For conciseness, we restrict our exposition to finite state, action and observation spaces. Results carry over to continuous state-action-ooservation spaces under some technical conditions that we omit for brevity (Szepesvári 2010).
3

---

Published as a conference paper at ICLR 2024
1982
disambiguate two stochastic processes that are induced by pairing POMDP with a policy
:: The core process, which is the process over state random variables {St). and the observation process induced by observation random variables {Or}. Please see Appendix A.2 for more detailed exposition on POMDPs.
3.2 OBSERVATION- SPACE ADVERSARIAL ATTACKS.
Observation-space adversarial attacks consider the scenario where an adversary manipulates the ob- servation of a victim at test-time. Much prior work falls within the SA-MDP framework (Zhang) et al. 2020). in which an adversarial agent with policy & S + P(S) generates adversarial observa- tions Ot [(x))- The perturbation is bounded by budget B S - 2 liiitiig supp E(-ss) B(s). For simplicity, we consider only zero-sum adversarial attacks, where the adversary minimizes the expected return of the victim. In case of additive perturbations, S Rd d E N and ip: _2 (Ku- mar et al. 2021), {(s)) == 8(0r). Here, := 8 + Pt. subject to a real positive per-step perturbation budget B such that ||:p [ I B². Vt.
3.3 INFORMATION HYPOTHESIS TESTING
Following (Blahut 1987 Cachin 1998), we assume two probability distributions P1 and IP over the space Q of possible measurements. Given a measurement Q E Q. we let hypothesis Ho be true if Q was generated from F1, and H₃ if Q was generated from P2 decision rule then a binary partition of Q that assigns each element q E Q to one of the two hypotheses. Let a be the type I error of accepting Hi when Hg true, and | be the Type II error of accepting Ho when H₃ is true. By the Neyman-Pearson theorem (Neyman et al. 1997). the optimal decision rule is given by assigning 9 to Ho iff the flogllikeihhodd log (g))(())22) (2)) 2 T, where T E R is chosen according to the maximum acceptable 3. For sequence of measurements Gr. this decision rule can be extended to testing whether E, log (g.)) (q:)) Z T WWlld 1945). It can further be shown (Blahut 1987) that d(a, 3) S (L 22), where ( P) Eq log Q log PP is the Kullback-Leible divergence between two probability distributions Q and P. and d(a, B) = a a(log a - log - -)) + a) log 3) is the binary relative entropy. Note that if KL(P, (P2,) O, then a = B = 2. and therefore Hn cannot be better distinguished from H1 than by random guessing. Hence Hg and H1 are -hformaaio indistinguishable if | = 0.
(1 a) (log(1
(P2))
ILLUSORY ATTACKS
4.1 THE ILLUSORY ATTACK FRAMEWORK
We introduce novel illusory attack framework in which an adversary attacks a victim acting in the environment & at test time, thus inducing a two-player zero-sum game G (Von Neumann & Morgen-
stern 1944). Our work assumes that the following facts about & are commonly known (Halpemn & Moses 1990) by both adversary and victim: At test time, the adversary performs observation-spar sample from the environment shared with an arbitrary adversary at train time, but has no certainty over which specific test-time policy the adver sary will choose. The adversary can sample from the environment shared with an arbitrary victim at train time, but has no certainty over which specific test-time policy the victim will choose. The task of the victim is to act optimally with respect to its expected test-time return, while the task of the adversary is to minimise the victim's 's expected test-time return.
attacks (see Sec. (.2) on the victim. The victim
We follow Haider et al. (022) in that we assume that the victim's 's reward signal endogenous (Barto et al. 2009). which means it depends on the victim's 's action-observation history and is not explicitly modeled at test-time, therehy exposing it to manipulation by the adversary. Additionally, environ-
ments of interest firequenly emit sparse or delayed reward signals that aggravate the task of detecting an attacker before catastrophic damage is inevitable (Sutton & Barto! 2018 Haider et al. 2023).
Assuming the victim's policy म. (0 x (A) + P(A) conducts adversary detection using cermattion-thoortiiin
optimal sequential hypothesis testing on its cetion-observattee history (see Section 3.33, the state of the adversary's MDP must contain the oction-observation history of the vic- tim. The adversary's 's policy v: S x (O x A)" + P(O) therefore conditions on both the state of the unattacked MDP, well the victim's action-observation history. This turns the victim's test-time
4

---

Published as a conference paper at ICLR 2024
MDP
Regular Adversary
0.010
00130
5g
ag
To
80
Adversary reward
Adversary reward
KL Divergence
1/3
+1
-0.082
Mlssory Adversary
2/3
Sg
1/2
1/2
C Chance Node
-0.175
e-illusory constraint for -0.000
States
1/2
Figure 2: Left: The unattacked MDP with an expected victim return of 1. Right: A regular adversarial attack and a perfect illusory attack, with an expected vitim return of 0 and f. re- spectively. The perfect illusory attack chooses observations Op such that the KL divergence between the attacked and unattacked observa- tion distribution is zero.
Figure 3: Empirical results for the 1-step MDP defined in Figure 2 The adversary's expected return increases with increasing e. At the same time, the empirical trajectory KI. constraint tightly controls the adversary policy's 's within e detectability. The purple line indicates the ad- versary's 's attack return ceiling at 0.0.
decision process into a POMDP with an infinite state space, making the game G difficult to solve with same-theoretic means (see Appendix A.2).
In the illusory attack framework, the trajectory density induced by the adversary's MDP is given by
pa(.) (o( (s) (ao (o)) I p(s)|st-1,, ( Oce, act). (1)
The trajectory density of the victim's observation process (see Sec. 3.1 in the attacked environment is given by
Pol:.D) E 80
Note that po(:, 1 1s.= reduces the trajectory density of the unattacked environment
po(:) = Po(:, 1o,= (i Teg
(3)
4.2 THE ILLUSORY OPTIMISATION OBJECTIVE
At test-tinn test-time, the adversary assumes that the victim is employing an information decision rule in order discriminate between the hypotheses that an adversary present, or (see Section 3) At each test-time step, the victim only has access to an empirical distribution Po(: v) based on its test-time samples N collected so far, which constrains the power of hypothesis test.
-otommatioonthhoretccllllll theoretically optimal
We here assume that the adversary does not know how many test-time samples the victim can col- lect, but has sampling access to the victim's test-time policy T.. Therefore, in order to degrade the victim's 's decision rule performance, the adversary aims to ensure that the KL-distance between Pv(..v) v) and (-) is smaller than a detectability threshold e. To maximise attack strength, the adver- sary would choose the highest that warrants undetectability, i.e.. renders the victim agent unable to distinguish between the observed trajectory distribution the attacked and unattacked environment.
We define cnormattion-theretiiin optimal adversarial attacks (c-illusory attacks) for given detectability threshold c. We set the direction of the KL-divergence analogously to Cachin (1998).
Definition 4.1 (e-illusory attacks), An e-illusory attack is an adversarial attack v" which minimizes the victim reward, subject to KI. (p.())
&
1² arg inf Br [Rt)s
s.t. KL (p(()))) Se.
5

---

Published as a conference paper at ICLR 2024
The e-illusory attack objective therefore aims to train adversary that reduces the victim's ex- pected cumulative return, while keeping its observed trajectory distribution e-close to the one would have observed in the unattacked environment.
We refer to illusory attacks that satisfy = 0 as perfect illusory attacks. In this case, to the victim, the presence of the adversary induces POMDP with infinite state-space (see Appendix A.2), in which the core process over MDP states (see Section[..] differs, but the observation process is statistically indistinguishable from the state-transition dynamics of the unattacked MDP. Importantly, the illusory KL constraint is distributional, the adversary can learn stochastic adversarial attack policies that are not restricted to the identity function.
Definition 4.2 (Perfect illusory attacks). A perfect illusory attack any undetectable non-trivial adversarial attack v, i.c. any for which y * lo,=s, and KL (pp()))llo( = 0.
Example. We now build up some intuition over the meaning of illusory attacks by studying a simple single-step stochastic control environment (Figure 2 The environment is assigned one of two initial states with probabilities 3 and 3. respectively. In the unattacked environment (Figure left), the victim can observe the initial state so. while under an adversarial attack, observes og (see right side). Given its observation, the victim chooses between two actions, upon which the environment terminates and a scalar reward issued. The reward conditions on the initial state and the victim's 's action. Without undetectability constraints, the optimal observation-space attack always generates observations fooling the victim over the initial state (Regular Adversary in Figure 2). however, changing the victim's 's observed initial state distribution. This makes this attack detectable. In contrast, perfect illusory attack only fools the victim half of the time when in the second initial state, and always when in the first initial state, this does not change the victim's 's observed initial state distribution. Note that attack undetectability comes at the cost of a higher expected victim return of for the perfect illusory attack, compared to 0 return under the regular adversarial attack.
4.3 ASCENT FORMULATION
To solve the e-illusory attack objective (see Def. 4.1pp
we propose the following dual-ascent algo- rithm (Boyd & Vandenberghe 2004) with learning rate hyper-parameter a E R4:
Mk+1 arg inl E, E,,o, [Ra] - AR-1 (p.()))
e]
(5)
AR+1 max (Ax + at KL (p)))))),( - c),
This algorithm alternates between policy updates and updates. As the KL-constraint is violated, adapts, thus modifying the influence on the KL-constraint in the policy update objective. Note that An has to be initialized heuristically.
4.4 ESTIMATING THE KL-OBIECTIVE
Accurately estimating the KL objective in Def. 4.1 is, in general, a computationally complex prob- lem due to its nested form and the large support of p-(+) and po(:, (see also Appendix A.3). We write
KL (p. (p.())loo(.>)) Brap,() [log e (pp)))
[.)
(6)
where H [px(-)] is the entropy, and H [po()), po(:, v)] is the cross-entropy (Murphy 2012 p. 953).
We now explicitly construct an estimator for the cross-entropy term. Let A A= IT (or loc,, aes). Then, p.(.)) A A-AAPooe (oo)
IT_ Pt (or41lo..))) az). and
po(-lp) A E ((ool5o) E u(or1s1, on, ao) E pl02l0, 022, (22) x(T-2)
NNoe that the e-illusory attack objective differs from standard constrained MDP
Altman 2021
CMDP) problem in that the illusory constraint cannot be expressed as discounted sum over state-transition costs (Achiam et al. 2017 CPO), but instead depends on trajectory densities.

---

Published as a conference paper at ICLR 2024
Pondulam
CartPole
HarChesaal
Adwrtary
E-ilasory ortack (nurs Porfect Lanry atrock (ourt) SA-MPP atbads
MNP atoad"
day score Adwrraa score asjattd for dirtection
Figure 4: We display normalised adversary scores, indicating the reduction in the victim's 's reward, on the y-axis. Each plot shows results in different environments, with different adversarial attacks on the x-axis. We show both the raw adversary score, as well as the adversary score adjusted for detection rates of different adversarial attacks (see Figure (5.. While the SA-MDP and MNP benchmark attacks achieve higher unadjusted scores, their high detection rates result in significantly lower adjusted scores.
Constructing unbiased estimator of (-)) is known to be non-trivial (Shalev al. 2022). How- ever, we note that the victim (and adversary) have access to a large number of samples from pa (.).. and, in the case of the adversary, (- 1/). In this work, we employ simple, but highly scalable estimator. Jensen's 's inequality (Jensen 1906 yields
H (px(-), (o(: 2)] p.)))) [log Ee..... [B]] S -Bp.. log B]
where B = =v(00 so) II. p 0c1, act). This yields the upper-bound Monte-Cate estimator
A (p = Σ (16(())))))))))))))))))))))))(
log log okn
where (oe, az iid. po(-). and ot iidd po, si20 iid p.
EMPIRICAL EVALUATION OF ILLUSORY ATTACKS
We illustrate illusory attacks in simple stochastic MDP (see Fig. 2 where show that opti- mization algorithm allows to precisely control the KL distance between the trajectory distributions of the attacked and unattacked environment. We then conduct an extensive evaluation of illusory attacks in standard high-dimensional RL benchmark environments (Zhang et al. 2021b: Kumar al. (021). We first empirically demonstrate the ineffectiveness of state-of-the-us art robustification methods for large perturbation budgets B (see Sec. B.2). However, we show that state-of-the-un out- of-distribution detectors can readily detect such attacks, rendering them ineffective. contrast, we show that c-illusory attacks with large perturbation budgets can be effective, yet undetectable. This demonstrates that c-illusory attacks can be more performant than existing attacks against victims with state-of-the-arr anomaly detectors. In an IRB-approved study, we demonstrate that humans, efficiently detect state-of-the-art observation-space adversarial attacks on simple control environ- ments, but considerably less likely to detect c-illusory attacks (Section5.00. We provide videos on the project web page at
Experimental setup. We consider the simple stochastic MDP explained in Figure and the four standard benchmark environments CartPole, Pendulum, Hopper and HalfCheetah (see Figure 6 in the Appendix), which have continuous state spaces whose dimensionalities range from well as continuous and discrete action spaces. The mean and standard deviations of both detection and performance results are estimated from 200 independent episodes per each of 5 random seeds. Victim policies are pre-traiiee trained in unattacked environments, and frozen during adversary training. We assume the adversary has access to the unattacked environment's 's state-transition function p.
17, as
Precisely controlling trajectory KL divergence. Using exact implementation of Equation 5 we learn e-illusory attacks for the single-step MDP environment pictured in Figure 2 As can be seen in Figure the measured KL(p, (-) (-)_ll((()_))) at convergence is bounded tightly by until it hits the divergence value for the unconstrained adversarial attack at ca. = 0.11. The adversary's 's return increases with increasing until it reaches the return of the unconstrained attack = 0.0.

---

Published as a conference paper at ICLR 2024
Detoction Rate in Porcen:
Automated
Human
100
Algorithm I e-illusory training (dual ascent)
75-
75-
Input: env, state transition function P. We
N, a, 6, estimator Dxr. (see Sec. 5.0.1
50
Init Ps.
25
25
for episode in I to do env.reset)
va)(8);; (T(
E-Illsory attack (ouns
desection Human detection"
SA MDP
Onews done env.stepla) y -r
((0)
attack**
while not done do
(s)ss;; a (o)
Figure 5: Different adversarial attacks are shown on the x-axis, with detection rates the y-axis. We see that both the automated detec- well as human subjects are able to detect SA-MDP and MNP attacks, while e-illusory at- tacks are less likely to be detected.
Snew: r, done env.step(a)
p =-r- (110 aodell
end while
Update uo using (8, o, ratr ,8nm)) À max(0, a(DKL -)).
end for
Effectiveness of state-of-the-art robustification methods under large-budget attacks. We first investigate the effectiveness of different robustification methods against variety of adversarial at- tacks, considering randomized smoothing (Kumar et al. 2021) and adversarial pretraining (ATLA, (Zhang et al., 2021a)). for budgets B E {0.02, 0.2}. We compare the performance improvement under adversarial attacks of each method relative to the performance without robustification. For an attack budget B 0.05, we find that randomized smoothing results in an average improvement of 61%, while adversarial pretraining results in 10% performance improvement. However, for the large attack budget B 0.2, both only result in average performance improvements of 15% and 8%, respectively (see Appendix A.5 for details).
5.0.1 COMPARATIVE EVALUATION OF ILLUSORY ATTACKS
Setup. For all four evaluation environments, we implement perfect illusory attacks (see Def. 2 by first constructing an attacked initial state distribution p(-10) that exploits environment-specitee symmetries. We then sample the initial attacked observations oo from the attacked initial state distribution and generate subsequent transitions using the unattacked state transition function ((-o)-1,0--) where 4,-1 is the action taken at the last time step (see Appendix A.6 for details). In contrast to perfect illusory attacks, e-illusory attacks are learned end-to-end using reinforcement learning. For this, we use a practical variant of the illusory dual ascent objective and estimate the KL-Distance accordance with the single-samph estimate of the MC-estimate defined in Eq. g (see Algorithm 1] and Appendix A.7). We estimate DxL. in Algorithm 1, i.e. the penalty term used update the dual parameter X, the sliding window average of the DxL. estimate defined in Equation 6) using single-sample estimate (see Eq.1).. We equip the victim agent with the state-of- the-art -il-of-distribution detector introduced by Haider al. (2023), which is trained on trajectories of the unattacked environment. This detector provides anomaly scores which we use to establish CUSUM (Page 1954 decision rule tuned achieve false positive rate of 3%. We adjust the e- illusory treshold to the empirical sensitivity of the detector on each environment. We consider attack budgets (see Sec. 3.2 B 0.05 and B 0.2, but focus on B 0.2 in this analysis (see Appendix for all results); to ensure fair comparison, we also apply the attack budget to e-illusory attacks.
Adversary performance against victim agents with automated detectors. We investigate the adversaries' relative performance in comparison to state-of-the-a adversarial attacks, specifically SA-MDP attacks (Zhang et al. (2221a) and MNP attacks (Kumar al. ] 2021). MNP attacks can only be implemented in CartPole which has discrete action space. We define the scalar adversary score as the resultant reduction in the victim's 's return, normalized relative to both the highest adversarial return in each class, as well the victim's 's expected return in the unattacked environment. We simulate contingency actions by setting the adversary's 's return to zero across episodes classified as attacked. This reflects middle ground across different scenarios in which adversary detection could trigger victim contingency options ranging from no action to test-time termination, major security escalation, and adversary persecution. As detailed in Figure 5 the detector detects MNP and SA-
8

---

Published as a conference paper at ICLR 2024
MDP attacks with probability close to 100%. In contrast, the detector classifies c-illusory attacks as adversarially attacked with very low probability. In coherence, the full columns in Figure ssow show that e-illusory attacks result in the highest average adversary scores when adjusting for detection. In contrast, detection-adjusted adversary scores for state-of-the-ar attacks are close to zero, which is expected due their high empirical detectability (see Fig. 5)
Detection of adversarial attacks by human inspection. We we perform controlled study with n 10 human participants investigate whether humans unfamiliar with adversarial attacks can detect adversarial attacks in simple and soyy-to-unerrtannd environments. We found CartPole and Pendulum, in contrast to Hopper and HalfCheetah, to be immediately accessible to participants with- out the need for additional training. Participants were first shown unattacked introduction video for both CantPole and Pendulum, exposing them to environment-sppecffc dynamics. Participants were then shown a random set of videos containing both videos of unattacked and attacked trajec- tories. For each video, participants were asked to indicate whether they believed that the video was unsuspicious, with the prompt 'the system shown in the video was [not] the same the one from the introduction video'. This phrasing was chosen so that participants would not be primed on the concept of illusory attacks (see details Appendix (A.)..
We found that participants classified MNP and SA-MDP attacks as suspicious with high accuracy (see Human detection in Figure 5 In contrast, participants were almost equally likely classify videos of unattacked and e-illusory attacked trajectories as unsuspicious. In fact, at a confidence level of 95%, the hypothesis 'participants are equally likely to classify an unattacked sequence as attacked as to classify an e-illusory attacked sequence as attacked' cannot be rejected. Our findings suggest that humans are unable to detect e-illusory attacks from short observation sequences in our simple environments. See Appendix A.8 for full results and the corresponding z-test statistic.
6 CONCLUSION AND FUTURE WORK
This paper introduces a novel class of observation adversarial attacks, illusory attacks, which admit an coormationtthooeticcll grounded notion of statistical detectability. We show the effee- tiveness and scalability our approach against both humans, and Al agents with access to state-of- the-art anomaly detectors across a variety of benchmarks.
We expect the potential positive impact of our work to outweigh the potential negative consequences as our work contributes to the design of secure syber-physice systems. However, it should be acknowledged assume the availability of contingency options for victim agents, which may not always hold true in real-world scenarios. Moreover, our experimental investigations are confined to simulated environments, necessitating further exploration in more intricate real-world domains.
Future research should conduct comprehensive theoretical analysis of the Nash equilibria within the two-player zero-sum game introduced by the illusory attack framework. Furthermore, efforts are required to develop more effective defenses against adversarial attacks applicable to real-world en- vironments, including (1) improved detection mechanisms, (2) robustified policies that incorporate detectors, and (3) improved methods to harden observation channels against adversarial attacks. An equally significant aspect of detection is gaining deeper understanding of the human capability to perceive and identify (illusory) adversarial attacks. We ultimately aim to demonstrate the viability of illusory attacks and the corresponding defense strategies in real-worle settings, particularly in mixed-autonome scenarios.
We are committed to promoting reproducibility and transparency in our re- search. To facilitate the reproducibility of our results, release the code on out project page https: nyurl w lusooy We provide detailed overviews for all steps of the experiments conducted in the Appendix, where we also link to the publicly available Code reposito- ries that our work uses.
Reproducibility.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Mazen Abdelfattah, Kaiwen Yuan, Z. Jane Wang, and Rabab Ward. Towards Universal Physical Attacks On Cascaded Camera-Lidar 3d Ohject Detection Models. In 202/ IEEE International Conference on Image Processing (ICIP), pp. 3592-3596, September 2021. doi: 10.1 /ttp:///P4299..2221.95909 ISSN: 2381- 8549.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained Policy Optimization. 2017.
Devon Adams, Alseny Bah, Catherine Barwulor, Nureli Musaby, Kadeem Pitkin, and Elissa M. Red- miles. Ethics Emerging: the Story Privacy Security Perceptions in Virtual Reality. pp. 427-442, 2018. ISBN 978- 978-1-939133-10-6 URL https: / /www. usenix. www.:e/www....ooooooeeeee///oo.a
iresentatien//e ida//dams adams
Eitan Altman. Constrained Markov Decision Processes. Routledge, New York, December 2021. ISBN 978-1- 315- 315-14022-3. doi: 10.1201/9781315144222
OpenAI: Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manip- ulation. The International Journal of Robotics Research, 2020.
Maurício Ariza, Antonio João Gonçalves de Azambuja, Jéferson Campos Nobre, and Lisandro Zambenedetti Granville. Automated Social Engincering Attacks using ChatBots Professional Social Networks. In Anais Workshop de Gerência Operução Redes Serviços (WGRS), pp. 43-56. SBC, May 2023. doi:
w0.5753/wgns.2233.PP URL bttpp::/. //sol.
sbc wacc.orgg. org.
http/indexxccc htpp://wwxppppppppppppeppppppppp wwwppppppppppppppppppppt articcetl wttp:////taccccic..rceeecc
24669
ISSN: 2595-2722.
A. Barto, Richard L. Lewis, and Satinder Singh. Where Rewards Come From. 2009. URL https: 7/ www.nteeetttttee.em
wwwpeeeeppppppeeeeeeeeeeeeeeeeee Daaeere-----oe Rewa rdw Come Fron Barto Lewis/ 2
Michele Basseville, Igor Nikiforov, et al. Detection of abrupt changes: theory and application, volume 104. prentice Hall Englewood Cliff', 1993.
Richard Bellman. Dynamic programming and stochastic control processes. Information and Control, 1(3): 228-239. September 1958. ISSN 0019-9958. doi: 00016//0019-9958888888888888
Richard E. Blahut. Principles and practice of information theory. Addison-Wesley Longman Publishing Co., Inc., 1987. ISBN 978-0-201-10709-3
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational Inference: Review for Statisticians. Jour- nal of the American Statistical Association, 112(518):859 877, April 2017. ISSN 0162 1459, doi: 1080/ 01621459.2011
URL https: /oi org 00 1880 001 621 21211459.211110 201..128777 Pub-
lisher: Taylor & Francis _eprint: http:///tai..om////0000000111112
Stephen Boyd and Lieven Vandenherghe. Convex Optimization. Cambridge University Press, March 2004. ISBN 978-1-107-39400-1 Google-Books-IDD IU7AAAAAQBAI
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeitch Zaremba. Openai gym, 2016.
Ben Buchanan, John Bansemer, Dakota Cary, Jack Lucas, and Mussah Musser. Automat-
ing Cyber Attacks, 2023.
URL sttpet///seet..ooqeetteo hwtp:///dwpppppp.d..d..oooaaao
automat ing- cyber
Christian Cachin. An information--heoreticc model for steganography. In David Aucsmith (ed.), Information Hiding, Lecture Notes Computer Science, pp. 306-318. Springer, 1998. ISBN 978-3-540-49380-8, doi: 10. 10.1000/3-540 50000773-54-49380-00 0340049330-8.22 -.211.
Yulong Cao, Chaowei Xiao, Benjamin Cyr, Yimeng Zhou, Won Park, Sara Rampazzi, Qi Alfred Chen, Kevin Fu, Morley Mao. Adversarial Sensor Attack on LiDAR-based Perception Autonomous Driving. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS New York. NY, USA, November 2019. Association for Computing Machinery, ISBN doi: 10.1 00.1145/3195555333398
'19. pp. 978- 1 14503-6747-9.
URL httts:: //d" .acm. wwt.o//g////o.o.////oo 1145/
9 9535. 3339815
10

---

Published as a conference paper at ICLR 2024
Yulong Cao, Ningfei Wang, Chaowei Xiao, Dawei Yang. Jin Fang, Ruigang Yang. Qi Alfred Chen, Mingyan Liu, and Bo Li. Invisible for both Camera and LiDAR: Security Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical- World Attacks. pp. 176-194, IEEE Computer Society, May 2021.
ISBN 978-1-72811 934-5, doi: 10.1109/SP40001 202100 URL http:ttt:: /www. http:///wwwccopp.r./cc// org/
csdl
I Lorena Cazorla, Cristina Alcaraz, and Javier Lopez. Cyber Stealth Attacks in Critical Information Infrastruc- tures. IEEE Systems Journal, 12(2):1788-1922 June 2018. ISSN 1937-9234. doi: 10.1109/SSSS..201..
2487684, Conference Name: IEEE Systems Journal.
Kathryn Chaloner and Isabella Verdinelli. Bayesian Experimental Design: A Review. Statistical Science, 10
(3):273-308 August 1995. ISSN 0883-42313 2168-8745, doi: 10. 10.12199//1177009999. URL ltt tts
7/pro jecteuc lid. http:///wwaaaa.aaaa.aa..a.aaeaaa
Bayesian- Exper iment - Des 2t wttpv/ewww...14//2...
full
Pub-
lisher: Institute of Mathematical Statistics.
Shang-Tise Chen, Cory Comelius, Jason Martin, and Duen Homg (Polo) Chau. ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detector. Michele Berlingerio. Francesco Bonchi, Thomas Gärtner, Neil Hurley, and Georgiana Ifrim (eds.), Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, pp. 52-68, Cham, 2019a. Springer International Publishing. ISBN 978- 3-030- 1092--0. doi: 10.1007/978 3-030- 10925-7.4.
Tong Chen, Jiqiang Liu, Yingxiao Xiang, Wenjia Niu, Endong Tong, and Zhen Han. Adversarial attack and defense reinforcement learning-fron security view. Cybersecurity, 2019b.
Nilaksh Das, Madhuri Shanbhogue, Shang- Tse Chen, Fred Hohman, Siwei Li, Chen, Michael Kounavis, and Duen Homg Chau. SHIELD: Fast, Practical Defense and Vaccination for Deep Learning using
JPEG Compression. In Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, KDD '18 pp. 196-204, New York, NY, USA, July 2018. Association for Computing Machinery. ISBN 978-1-4503-5555-00 doi: 10.1445/2198199.221990..
URL https:
7/al. acl.aam.com wttp.ee..../......111/////22/..2
3219910
Jonas Degrave, Federico Felici. Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414 4194444411144011 2022.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances neural information processing systems, 33: 33:13049-13061, 2020.
Arnaud Doucet, Nando de Freitas, and Neil Gordon. Introduction to Sequential Monte Carlo Meth- ods. Arnaud Doucet, Nando de Freitas, and Neil Gordon (eds.), Sequential Monte Carlo Methods in Practice, Statistics for Engineering and Information Science, pp. 3-14. Springer, New York, NY, 2001. ISBN 978-1-4757-34377 doi: 118-1-4157-447--2
URL /tttps://doi.org/11.... 1CC7/
Ranjie Duan, Xingjun Ma, Yisen Wang. James Bailey, A. K. Qin, and Yun Yang. Adversarial Camouflage: Hiding Aoysical-World Attacks With Natural Styles. pp. 997-1005. IEEE Computer Society, June 2020.
ISBN 978-1-72817-168-5, doi: 10. 10011009CVRR22200200000000 00108 URL /htp:s:tt///tt /wwww//wwww.... computer.
Michael Everett, Bjorn Lutjens, and Jonathan P. How. Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement I Learning. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Kevin Eykholt, Ivan Evtimov. Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Ta- dayoshi Kohno, and Dawn Song. Robust Physical World Attacks Deep Learning Visual Classification. In 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1625-1634, June 2018. doi: 10. 10..1099CPRR.201.0000
ISSN: 2575-7075.
Mattie Fellows, Brandon Kaplowitz, Christian Schroeder de Witt, and Shimon Whiteson. Bayesian Exploration Networks, September 2023. URL http::/aaattt http://aaxvvv... org/ abs /2/s 13049 aIXiv:2308.1000 [cs].
Adam Foster, Martin Jankowiak, Eli Bingham, Paul Horsfall, Yee whye Tch, Tom Rainforth, and Noah Good- Variational Bayesian Optimal Experimental Design. 2019.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamesssing adversarial examples. arXiv preprint arXiv:1412.6677 2014.
11

---

Published as a conference paper at ICLR 2024
Steffen Grünewälder, Guy Lever, Luca Baldassarre, Massimilano Pontil, and Arthur Gretton. Modelling tran- sition dynamics mdps with rkhs embeddings. Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12, pp. 1603-1610, Madison, WI, USA, 2012. Om- nipress. ISBN 1455012381111,
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off Off-policy maximum en- tropy deep reinforcement learning with stochastic actor. In International conference on machine learning. PMLR, 2018.
Tom Haider, Karsten Roscher, Felippe Schmoeller da Roza, and Stephan Günnemann. Out-of-distribution detection for reinforcement learning agents with probabilistic dynamics models. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 851 151-859, 2023.
Joseph Y. Halpern and Yoram Moses. Knowledge and common knowledge in a distributed environment. doi: 10.1145779147.79161. URI.
Journal of the ACM, 37(3):549 587, July 1990. ISSN 0004-5411.
https: //dp: acm. wwt/
0111/115599777 79161
Wyatt Hoffman. Al and the Future of Cyber Competition, 2021. URL
cdu/ cct/ppbblicc wwwplcultatt... Lon/ a.- awd-thh----e- future-uturr--ore cyber wttee--ommpctciii
Yu-Chih Tuan Hu, Chn-Cheng Chen, Bo-Han Kung. Kai-Lung Hua, and Daniel Stanley Tan. Naturalistic Physical Adversarial Patch for Object Detectors. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7828-7831 Octoher 2021. doi: 10.1
007755 ISSN: 2380-7504.
Lifeng Huang, Chengying Gao, Yuyin Zhou, Cihang Xie, Alan L. Yuille, Changging Zou, and Ning Liu. Uni- versal Physical Camouflage Attacks Object Detectors. pp. 717-726. IEEE Computer Society, June 2020. ISBN 978-1-72817-168- doi: 00.100//CVP4422002202000 2020 00080, URL https::////// /wwww computer.
Yunhan Huang and Quanyan Zhu. Deceptive reinforcement learning under adversarial manipulations cost signals. In International Conference Decision and Game Theory for Security, pp. 217-237. Springer, 2019.
Léonard Hussenot, Matthicu Geist, and Olivier Pictquin. Copycat: Taking control neural policies with constant attacks. arXiv preprint arXiv: 12282, 2019.
Marcus Hutter. Universal Artificial Intellegence. Texts in Theoretical Computer Science An EATCS Series. doi: 10. 10.1007/b138233
Springer. Berlin, Heidelberg, 2005. ISBN 978-3-540-220 3-7---------2139-5 978-3-540-26877-2.
URL httttttp///ikk spr com 111./..000//1333
Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang. and Dusit Niyato. Challenges countermeasures for adversarial attacks on deep reinforcement learning. IEEE Transactions on Artificial Intelligence, 3(2):90-1) 1090 2021.
J. L. W. V. Jensen. Sur les fonctions convexes et les inégualités entre les valeurs Moyennes. November 1906. doi: 10.1007/b002418571.
[RL
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially ob- servable stochastic domains. Artificial Intelligence, 101(1):99-134, May 1998. ISSN 0004-3702. doi:
10.1016/50004- 110101///0000437729990000333333
Ezgi Korkmaz. Adversarial robust deep reinforcement learning requires redefining rohustness. arXiv preprint arXiv:2301.07458 2023.
Ezgi Korkmaz and Jonah Brown-Cohen. Detecting adversarial directions in deep reinforcement learning to make robust decisions. arXiv preprint arXiv:2300.05873, 2023.
Vojtëch Kovařík, Martin Schmid, Neil Burch, Michael Bowling, and Viliam Lisy. Rethinking formal models of partially observable multiagent decision making. Artificial Intelligence, 303:103645,, February 2022. ISSN 0004-3702.
science doi: 10. 10.1016 2021. 103645. URI. (ttths:pp: : www. sci encedi wwe.edeeett...o..moo.oo com/
Aounon Kumar, Alexander Levine, and Soheil Feizi. Policy Smoothing for Provably Robust Reinforcement Learning. Technical report, arXiv, 2021.
Tze Leung Lai. Information bounds and quick detection of parameter changes in stochastic systems. IEEF Transactions on Information theory, 44(7):2917-2929, 1998.
12

---

Published as a conference paper at ICLR 2024
Ralph Langner. Stuxnet: Dissecting Cyberwarfare Weapon. IEEE Security Privacy, 933):99-5,, May 2011. ISSN 1558-4046. doi: 10.1 10/ Conference Name: IEEE Security & Privacy.
John Banister Lanier, Stephen McAleer, Pierre Baldi, and Roy Fox. Feasible adversarial robust reinforcement learning for underspecified environments. arXiv preprint arXiv:2207.09597, 2022.
Jan Leike. Nonparumetric General Reinforcement Learning. PhD thesis, Australian National University, November 2016. URL http::://awww...t///////
08944 arXiv:1611 11111611100999 1.8944 [cs].
Shasha Li, Ajaya Neupane, Sujoy Paul, Chengyu Song. Srikanth Krishnamurthy, Amit K. Roy Chowdhury, and Ananthram Swami. Adversarial Perturbations Against Real-Time Video Classification Systems. In Proceedings 2019 Network and Distributed System Security Symposium, 2019.
Yen-Chen Lin, Ming- Yu Liu, Min Sun, and Jia-Bin Huang. Detecting adversarial atlacks neural network policies with visual foresight. arXiv preprint 17X0004710000844, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learming Models Resistant Adversarial Attacks. May 2023. URL httpss://openreevieww
Stephen Marcus McAleer, Gabriele Farina, Mare Lanctot, and Tuomas Sandholm. ESCHER: Eschewing Im- portance Sampling in Games by Computing a History Value Function to Estimate Regret. February 2023.
Yannick Merkli. Evaluating and Defeating Network Flow Classifiers Through Adversarial Muchine Learning. PhD thesis, ETH Zurich, 2020.
Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image Segmentation Using Deep Learning: A Survey. IEEE Transactions Puttern Analysis and Ma- chine Intelligence, 44(7)3353-3542,, July 2022. ISSN 1939-3539. doi: 10.1 12/1TAAMM....221..0200 Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
George E. Monahan. Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms. Management Science, 28(1):1-16, 1982. ISSN 0025-1909. URL httpt::////// //www jstor. htgg/staalcc2661100 Publisher: INFORMS.
Kevin P. Murphy. Machine Learning: A Probobilistic Perspective. The MIT Press, July 2012. ISBN 978-0- 262-01802-9.
Christian A Naesseth, Fredrik Lindsten, and Thomas B Schön. Nested sequential monte carlo methods. 2015.
Jerzy Neyman, Egon Sharpe Pearson, and Karl Pearson. IX. the problem of the most efficient tests of statistical hypotheses. 231(694) 1997. doi: 10.1098/rsta.1933. 1933.0009. URL /https::/
royal aeralsociiet sociityppbliisieeee.ccccc.i shiiiiihhnngg wttpo//g///oooo/oooooooo 1ww.///t//..../ 033.0009
Publisher: Royal Society.
NSCS. The threat from commercial cyher proliferation, 2023. URL
gov. /
seport/commerciall/ heppor///ccmeiiraaaaaeeeppppppee otw.:a/a.scppppp.oo....eeooeeeas
assesssment
Tuomas Oikarinen, Wang Zhang. Alexandre Megretski, Luca Daniel, and Tsui- Weng. Robust deep re- inforcement learning through adversarial loss. Advances in Neural Information Processing Systems, 34, 2021.
E. S. Page. Continuous Inspection Schemes. Biometrika, 410112) 1954. ISSN 0006-3444. doi: 10. 2307/2333009. httt/hhtts:///t/ jstor. wwt.:/oggg/stabbboooo 2333009 Publisher: [Oxford University Press, Biometrika Trust].
You Qiaoben, Chengyang Ying, Xinning Zhou, Hang Su, Jun Zhu, and Zhang. Understanding Adversarial Attacks on Observations in Deep Reinforeement Learning. Technical report, arXiv, 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Antonin Raflin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 22(268):1-8, 2021.
Tom Rainforth, Robert Comissh, Hongseok Yang, Andrew Warrington, and Frank Wood, On nesting monte carlo estimators, 2018. hwl.httt///arsivoor///bbaee//
13

---

Published as a conference paper at ICLR 2024
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015. URL https: //wprrssppp/ee//a/ nips CC wwtppppp.ppppppccctt/ httppp/pppp....aaaapaae///////// 44t
html
Alessio Russo and Alexandre Proutiere. Balancing detectability and performance of attacks on the control channel of markov decision processes. In 2022 American Control Conference (ACC), pp. 2843-2850. IEEF, 2022.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense- GAN: Protecting Classifiers Against Adver- sarial Attacks Using Generative Models. May 2023. URL http://pss///ooe..o...eeeeeeeeee wwwp://wwwwwwwwww.owooeeeee..eae d=
RkJ3ibb0-
Christian Schroeder de Witt, Yongchao Huang, Philip H. S. Torr, and Martin Strohmeier. Fixed Points in Cyher Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS, November 2021. URL. http:
//arxiv.crg/abs/ abs//11s.11111.11 12197 arXiv:2111.12197 [es].
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimiza- tion Algorithms. Technical report, arXiv, August 2017.
Yuval Shalev, Amichai Painsky, and Trad Ben-Gal. Neural Joint Entropy Estimation. IEFE Transactions on Neural Networks and Learning Systems, pp. 1-13, 2022. ISSN 2162-2388. doi: 10.1109/NNNNS 3204919. URLittps:: / http//eeeepppppooe -iccppoorss leee. org/ http://occenttn//o 990 Conference Name: IEEE
Transactions Neural Networks and Learning Systems.
Wanggang Shen and Xun Huan. Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Palicy Gradient Reinforcement Learning, March 2022. URL
15335 12Xiv:2110.1533 [cs, stat].
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. PointRNN: 3D Object Proposal Generation and Detection From Point Cloud. In 2019 JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 770-779, Long Beach, CA. USA. June 2019. IEEE. ISBN 978-1-72813 12-272293333388 doi: 10.1109/CVPR.
2019.00086. URL https: http::::///eeexpoore... lese. wrt. 89540807
Samuel Sokota, Ryan D'Orazio, Zico Kolter, Nicolas Loizou, Mare Lanctot, Joannis Mitliagkas, Noam Brown, and Christian Kroer. A Unified Approach to Reinforcement Learning, Quantal Response Equilib- ria, and Two- Player Zero-Sum Games, April 2023. URL http:://axxxi wwwaaaivvvveo org/ abs abss/2200 2ttp:/206...00 05825 arXiv:2206.05825 [es].
Jiachen Sun, Yulong Cao, Qi Alfred Chen, and Z. Morley Mao. Towards robust LiDAR based perception in autonomous driving: general black-box adversarial sensor attack and countermcasures. In Proceedings of the 29th USENIX Conference on Security Symposium, SEC'20, pp. 877-894, USA, August 2020a. USENIX Association. ISBN 978-1-939113-11755.
Jianwen Sun, Tianwei Zhang, Xiaofci Xie, Lei Ma, Yan Zheng, Kangjie Chen, and Yang Liu. Stealthy and Efficient Adversarial Attacks against Deep Reinforcement I Learning, May 2020b. arXiv:2005.07099 [es].
Yanchao Sun, Ruijic Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl. International Conference on Learning Representations, 2021.
Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cam- bridge, MA, USA, 2018. ISBN 978-0-262-039224-
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312,19,,,,,99, 2013.
Csaba Szepesvári. Algorithms for Reinforcement Learning. Synthesis Lectures Artificial Intelligence and Machine Learning. Springer International Publishing, Cham, 2010. ISBN 378-3-031-004222 978-3- 031-01551-9. doi: 10010779888--001-015555 URL, httptsat////ikkkk 978
com www.m.................m 1CC7/
Alexander Tartakovsky, Igor Nikiforov, and Michele Basseville, Sequential analysis: Hypothesis testing and changepoint detection. CRC Press, 2014.
Buse GA Tekgul, Shelly Wang, Samuel Marchal, and N Asokan. Real-time attacks against deep reinforeement learning policies. arXiv preprint arXiv:2106008744 2021.
14

---

Published as a conference paper at ICLR 2024
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv: 1074, 2018.
Emanuel Todorov, Tom Erez. and Yuval Tassa. MuJoCo: A physics engine for model-based control. In 2012 International Conference on Intelligent Robots and Systems, 2012.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robust- ness May Be at Odds with Accuracy, May 2018. URL http::/arxvv..rrrt wwtp///gg////////eeeeeeeee aww/aa/........... 12152
Navv:1805.115555 (es, stat] version: 1.
James Tu, Mengye Ren. Sivabalan Manivasagam, Ming Liang, Bin Yang, Richard Du, Frank Cheng, and Raquel Urtasun. Physically Realizable Adversarial Examples for LiDAR Object Detection, pp. 13713- 13722. IEEE Computer Society, June 2020. ISBN 978- 778-1-722817-16855 doi: 10. 109.10/222P222200022222 2020. 01373. URL https: //www. computer, wttpp/pwww........eeeeooooeoooe.
71 118000m37133/m300550mm
J. Von Neumann and O. Morgenstem. Theory of games and economic behavior. Theory of games and cconomic behavior. Princcton University Press, Princeton, NJ, US, 1944. Pages: xviii, 625.
A. Wald. Sequential Tests of Statistical Hypotheses. The Annals Mathematical Statistics, 16(2):117-18)) 1945. ISSN 0003-4851. Utthttts:: htllhhttts:///ww/ htttpp::/wwww /ww.. jstor. .org/ wwc.or.org/stallee...oo Mathematical Statistics.
Publisher: Institute of
Larry Wasserman. All of Nonparametric Statistics (Springer Texts in Statistics). Springer- Berlin, Heidelberg, 2006. ISBN 0387251456.
Fan Wu, Linyi Li, Zijian Huang, Y. Vorobeychik, Ding Zhao, and Bo Li. CROP: Certifying Robust Policies for Reinforeement I Learning through Functional Smoothing. ArXiv, 2021.
Allison Wylde. Zero trust: Never trust, always verify. 202/ International Conference on Cyber Situational
Awareness, Data Analytics and Assessment (CyberSA), pp. 1-4, June 2021. doi: 10. A0.1009/ 2021.9478244. htt.:htttt:/ieeeeeee
eee or http://aaaaaatt///ccoeeeeee//e//
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating Adversarial Aidiegating Effects Through Randomization. May 2023. httphhtts:///peeeee
Weilin Xu, David Evans, and Yanjun Qi. Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks. In Proceedings 2018 Network and Distributed System Security Symposium, 2018. doi: 10.
2dss.2018. 2018.2398... htR.htt:::/ /ttttp://axxvvv arxiv org/ /abp///a///// 704 01 155 arXiv:1704.01 155 [es].
Winson Ye and Qun Li. Chatbot Security and Privacy in the Age of Personal Assistants. In 2020 IEEE/ACM Symposium on Edge Computing (SEC). pp. 388-393, November 2020. doi: 10. (1109//EEC500022220000000
URLthttps: www.epelreee........e. leee. wtt.://ee..../oo/oooooeeeeeeeeee
9355740
Huizhen Yu and Dimitri P. Bertsekas. On near Optimality of the Set of Finite-State Controllers for Average Cost POMDP. Mathematics of Operations Research, 33(1):1-11, 2008. ISSN 0364-765X. URL https:
www. jsterr .wweererrgo crgg/stable/555113....
Publisher: INFORMS.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsich. Robust Deep Reinforcement Learning against Adversarial Perturbations on State Ohservations. Advances in Neural Information Processing Syxtems. Curran Associates, Inc., 2020.
Huan Zhang, Hongge Chen, Duane Boning, Cho-Jui Hsieh. Robust Reinforcement Learning on State Observations with Learned Optimal Adversary, January 2021a.
Jindi Zhang, Yifan Zhang, Kejie Lu, Jianping Wang, Kui Wu, Xiaohua Jia, and Bin Liu. Detecting and Iden- tifying Optical Signal Attacks on Autonomous Driving Systems. IEEE Internet of Thingy Journal, 8(2): 1140-1153 January 2021b. ISSN 2327-4662. doi: 10. 109//00..0000001 1690. Conference Name: IEEE Internet of Things Journal.
Yi Zhu, Chenglin Miao, Tianhang Zheng, Foad Hajiaghajani. Lu Su, and Chunming Qiao. Can We Use Arbitrary Objects to Attack LiDAR Perception in Autonomous Driving? In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, CCS '21, pp. 1945-1960, New York, NY, USA, November 2021. Association for Computing Machinery. ISBN 978- -4503-8854-44 doi: 1145/41444444 0445//34512200 3485377 utthttpss /ttp::://ddl.cm.. pttp//oo////oooooooooooo///ooo 1ttp//....1111...0
01.55/3446112.3488833 20.
Karl Johan Aström. Optimal Control of Markov Processes with Incomplete State Information 1. Journal of Mathematical Analysis and Applications, 1965.
15

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1 ADDITIONAL RELATED WORK
Assuming different black-hox setting, Hussenot al. (2019) introduce class of adversaries for which unique mask precomputed and added the agent observation at every time step. Our framework differs from these previous works in that it preserves consistency across trajectories of observation sequences. [Kork- maz 2023) proposes adversarial attacks motivated by a notion of imperceptibility measured in policy network activation space. One major difference is that the paper focuses per-state imperceptibility, while our work focuses cnoomatiionthcorette undetectability, which hence requires focusing on whole trajectories.
AP attack targets include cameras (Eykholt al. 2018 Chen et al. 2019a Duan et
2020 Huang et al.
2020 Hu et al.) 2021). LiDAR (Sun et al.
2020a Cao el al. 2019 Zhu al. 2021 et al. 2020) and
multi-sensor fusion mechanisms (Cao et al.
2021 Abdellattah et al 2021).
Lin et al. (2017) develop an cetton-conditioond frame module that allows agents to detect adversarial attacks by comparing both the module's 's action distribution with the realised action distribution. (Tekgul et al. (2021)
detect adversaries by evaluating the feasibility of past action sequences. Li et al. (2019); [Sun et al. (2020b);
Huang & Zhu (2019); Korkmaz & Brown-Cohen (2023) focus on the deiectability of adversarial attacks but without considering notions of stochastic equivalence between observation processes.
A.2 POMDP CORRESPONDENCEE
We begin this section by defining standard POMDP notation. In a partially observable MDP Aström 1965
Kaelbling al. 1998 POMDP) (5, A, Q O p.r,y the agent does not directly observe the system state 8, but instead receives an observation O₂ ~ (-(ss) where O + P(2)) is an observation function and Ω is a finite non-empty observation space. The canonical embedding pomdp -0 + P from the set of finite MDPs 20 to the family of POMDPs Q maps Ω 1- S, and sets (ss) 8, Vs E S. In POMDP, the agent acts on policy T : HC, + P(A), growing a history h++1 = hea,0++17++1 from a set of histories H = (A where 7 := U, H' denotes the set of all finite histories. We denote histories (or sets of histories) from which reward signals have been removed as (.)..
IR)',
In line with standard literature (Monahan 1982) we distinguish between two stochastic processes that are induced by pairing a POMDP with policy π:: The core process, which is the process over state random variables {S+) }. and the observation process, which is induced by observation random variables }O:}. The frequentist agent's goal is then to find an optimal policy n²' that maximizes the total expected discounted return, iee. n'
arg sup Big ~F Ee 7 rra, where II == ( : Hi, + P(A)) is the set of all policies.
Now consider POMDP == (S' A, O², ,',,,,", with finite horizon T, state space S' := AXxAx deterministic observation function O' S' + Ω2 and stochastic state transition function p' S - P(S)). Then, for any n, H., + (AA) and v: S H." + PS2). we can define corresponding p' and O' such that the reward and observation processes cannot be distinguished by the victim. We now proceed to the formal Theorem.
Theorem A.1 (POMDP Correspondence). For any EV, there exists a corresponding POMDP & (E.) for which the victim's 's learning problem is identical.
Proof. Recall that the semantics of Σ are as follows: Fix vietim policy क Hir 1 P from the space of all possible sampling policies II. At time L = O, we sample initial state so p() The adversary then samples an observation ou ~ v(-1su) which is emitted to the victim. The victim takes an action au ~ T (-loo)), upon which the state transitions 81 ~ p(-[su, au) and the victim receives a reward rı (-1s0, (a)s, At time t >> O, the victim has accumulated history h, := 0000r1 Os, on which or ~ v(-1st, herr)) conditions.
Define p' as the following sequential stochastic process: At time = 0, first sample ~ p(-10). Then sample oo v(([ss)), and define := 00(0) (s0,00). For any > 0, first sample st p(- ae-1) 1) then og ~ v--lsge. aee,, oct) and define s/ := p' [(( Se, oe, a)-1). We finally define ((+() == ( == oe, where we indicate that o, is stored in sy by using an explicit projection operator proj.. Clearly, under any sampling policy v, the observation and reward processes induced by & and E are identical as T - co. This renders the reward and observation processes identical in both environments. Note that, as T oc, E.'s state space grows infinitely large.
In other words. Theorem A.I implies that, given enough memory (Yu & Bertsekas 2008 the adversary can be chosen such that the staeespace of E, becomes arbitrarily due to its infinite horizon. This renders the worst-case problem of finding optimal victim policy in (E,)) intractable even if the adversary's policy
is known (Hutter 2005
Leike 2016). The underlying game G, therefore, assumes an infinite state space,
16

---

Published as a conference paper at ICLR 2024
Figure 6: Benchmark environments used for empirical evaluation, from left to right. In CartPole, the agent has to balance a pole by moving the black cart. In Pendulum, the agent has to apply torque action balance the pendulum upright. In Hopper and HalfCheetah, the agent has choose high-dimensional control inputs such that the agent moves towards the right of the image.
preventing recent progress in solving finite-horizon extensive-form games (Kovařík ct al. 2022 McAleer et al. 2023 Sokota al. 2023 from being leveraged in characterizing its Nash equilibria. We now give proof of construction.
A.3 ON THE DIFFICULTY OF ESTIMATING THE ILLUSORY OBJECTIVE
We note that estimating the illusory objective is, in general, difficult. Even when choosing a nonparametric kernel with optimal bandwidth, the risk of conditional density estimators increases O(N و with support
dimensionality [Wasserman] 2006 Grünewälder et al. 2012 Fellows et 2023). This is aggravated by
KL-estimation being a nested estimation problem (Rainforth al. 20188..
While the estimator bias may be further reduced by using a more sophisticated nested estimation method such as multi-level MC estimator (Naesseth et al. (015)) and by performing improved estimates for pu(., using variational inference (Blei 2017 VI), sequential Monte-Carlo (Doucet al. 2001 SMC), these
methods come with increased computational complexity.
A.4 DETECTOR AND DECISION RULE USED EXPERIMENTS
2023 using the implementation provided by authors As detector provides anomaly scores at every time step but does not provide a decision rule for classifying a distribution as attacked, we implement a CUSUM (Page 1954) decision rule based on the observed anomaly scores observed at test time and the mcan anomaly score for held-out test set of unattacked episodes. We train the detector unperturbed environment interactions, using the configuration provided by the authors. We then tune the CUSUM decision rule such that per-episade false positive rate 3% achieved. We assess the accuracy of detecting adversarial attacks across all scenarios presented in Table A.7.1
We implement the out-of-distributie detector proposed Haider
A.5 ROBUSTIFICATION
implement ATLA (Zhagg et al. 202 victim by co-training it an adversary agent, and follow the original implementation of the authors I We implemented randomized smoothing as a standard defense against adversarial attacks on RL agents, as introduced in Kumar et al. (2021). We use the author's 's original implementation"" [1] See Table f results.
A.6 PERFECT ILLUSORY ATTACKS IMPLEMENTATION
We implement perfect illusory attacks as detailed in Algorithm 2 The first observation Op is set to the negative of the true first state sampled from the environment. Note that in HalfCheetah and Hopper the initial state distribution is centered around the origin, hence first subtract the offset, and then compute the negative of the observation and add the offset again. As the distribution over initial states symmetric in environments (after removing the offset), this approach satisfies the conditions of perfect illusory attack (see Definition 4.2)
httaas//tthhbb.cco//amamaaaa222a ruuus R
17

---

Published as a conference paper at ICLR 2024
Algorithm 2 Perfect illusory adversarial attack
Input: environment env, environment transition function whose initial state distribution p(-10) is symmetric with respect to the point Psymmeetry in S, victim policy To-
80 = env.reset()
Og (so Psymmetry) + Psymmetry
Og = (on)
done env.step(ag)
while not done do
k: = k
~ 1(ok-1 UA (k-1)
a = To (ok)
- done enc.steppaa))
end while
A.7 LEARNING C-ILLUSORY ATTACKS WITH REINFORCEMENT LEARNING
next describe algorithm used to learn e-illusory attacks and training procedures used to compute the results in Table A.7.1 We use the CartPole, Pendulum, HalfCheetah and Hopper environments as given in [Brockman et al. (016). We shortened the episodes in Hopper and HalfCheetah to 300 steps to speed up training. The transition function implemented using the physics engines given in all environments. We normalize observations by maximum absolute observation. We train the victim PPO (Schulman et all 2017) and use the implementation PPO given in Raffin et al. (202T), while not making any changes to given hyperparameters. In both environments we train the victim for million environment steps.
We implement the illusory adversary agent with SAC (Haarnoja et al. 2018)) where likewise use the imple- mentation given in Raffin al. [202I. We initially ran small study and investigated four different algorithms as possible implementations for adversary agent, where we found that SAC yields best performance and training stability. We outline the dual ascent update steps in Algorithm [ which, like RCPO (Tessler et al. 2018), pulls single-sample approximation of the constraint into the reward objective. We approximate DRL by taking the mean of constraint violation ||o p(ootd, aata) l| over the last 50 time steps. further ran a small study over hyperparameters a E (0.01, 0.1, 1) and the initial value for E (10, 100) and chose the best performing combination. We train all adversarial attacks for four million environment steps.
Disection rate acme enviconents
Detection Porzent
Table Adversary scores under different at- tacks and defenses.
Norm, adversany [%]
attak
no defencee
smoohhing
Dily andilale Castie
Budget
MNP (Kumrr el 2021
0.05
Figure 7: Detection results for = 0.05, Dif- ferent adversarial attacks shown the axis, with detection rates on the y-axis. We see that the automated reliably detector detects SA- MDP and MNP attacks, while e-illusory attacks less likely to be detected.
SA-MOP Zlies el ar 2:21a
0/05
85 +
50 00+:5
75 7
MNP Kunin 2011
0.2
07 +
97
+
SA-MOP
2021a
7
Computational overhead of c-illusory attacks. Note that there no computational overhead our method at tist.time. We found our experiments that the computational overhead during training of the adversarial attack scaled with the quality of the learned attack. In general, we found that the training wall-clock time for the e-illusory attacks attacks results presented in Table was about twice that of the SA-MDP attack (note that MNP attacks and perfect illusory attacks do require training).
18

---

Published as a conference paper at ICLR 2024
A.7.1 RESULTS FOR PERTURBATION BUDGET 0.05
We show the remaining results for perturbation budget of 0.05 in Figures and corresponding Figures in the main paper are for a perturbation budget of 0.2.
Note that the
Posdslan
CartPolc
HalCheothh
Hupper
E-Laary sttack (eurs
Parfect thaay attack (oun) SA-MPP sttack
MMP uttuck"
Raw adenary
stju detectio
Figure 8: Results for B 0.05. We display normalised adversary scores, indicating the reduction in the victim's 's reward, on the y-axis. Each plot shows results in different environments, with different adversarial attacks on the x-axis. We show both the raw adversary score, well the adversary score adjusted for detection rates of different adversarial attacks (see Figure While the SA-MDP and MNP benchmark attacks achieve higher unadjusted scores, their high detection rates result in significantly lower adjusted scores. Note that MNP attacks perform significantly worse for B = 0.05, as compared to 0.2 (see Figure
Table 2: Full results table for all four environments
allack
budget 8
Detection Rate
Victim reward
Pendulum
SA-MDP Zhang al. 2021a
76.3++0.05
-797.2-4-999
c-illusory allack (Durs]
0.00
104,,
SA-MDP Zhang 2021a
0.2
100+:0.03
-1387.0 19,00
e-illusory btack
3.6::0.02
-980.0=88..0
Perfect illusory atlack (surs)
3.0.10.02
1.8.6.8188.6
unattacked
3.2:00.33
-189.4
CariPole
MNP Kumar et al. 2021
0.05
86.9+0.3
485.04-33.5
SA-MDP Zhang 2021a
80.5=0.8
9,42:0.2
e-illusory allack (ours
1.5:20.02
12.9:20.3
et
100+-0
18.34-20.8
SA-MDP Zhang Zhing all 2021a
100+00
9,3++0,1
e-illusory lttakk (ours)
3.7:10.01
11.0=:0.5
Perfect illusory attack (ours)
3.1
30.1.12.2
unattacked
3 3.2+0.01
500.0
HalRChectah
SA-MDP 7hang
2021a]
0.05
100+0
-1570.8- 117.4
cillusory allack (Duss]
0.00
1180.8.1 50.1
SA-MDP Zhang
2021a
100.0
16443881.44,
e-illusory altck [ous)
01:0
-240.62 18.0
illusory (ours)
2.9.10.04
5.9 1.36.8
unatlacked
3.1.10.02
2594.6
Hopper
SA-MDP Zhang et al. 2021a]
0.05
87.420.02
144.1=265.4
e-illusory attack [ouns]
01:0
209.41:90.8
SA-MDP Zhang al. 2021a]
0.2
95.6-10.02
261.55117.4
<-illusory
Mttck (00N)
1.56=0.44
-260.9:140.8
Perfect illusory attack (ours)
3.1:10.02
679.2363.9
unattacked
2.8++0.88
958.1
A.8 HUMAN STUDY
Study approval. Our study approved by independent ethics committee under reference R84123/RE001.
19

---

Published as a conference paper at ICLR 2024
Table 3: Results from our study with human participants.
Enviroement
both
Penduluum
CariPole
fffalse
no attack)
34.2 =: 11.4
31.5 # 10.5
37.0 a: 12.3
Pffalic
SA-MIPP)
27.2
96.3 32.1
22.2
P. Jalso
e-illusory atlackk
32.1 10.8
37.0 12.3
27.7 = 9.3
Fflaee
MNP)
63.3 0: 27.8
83.3 a: 27.8
Setup. We performed a controlled study with n 10 human participants. All participants were graduate- level university students. None had prior knowledge about the objective of the study. Participants participated voluntarily; estimate the time needed per participant to he around 15 minutes. Participants were handed slide show which contained all relevant information. This slide show included in the supplementary material the respective folder. We further add the sheet with ground truth labels for video sequences.
After consenting to participate, participants were provided with the slide show and an online sheet to indicate their answers. The study was self-paced and done hy the participants without further supervision. We further advised participants not to do the study if they would not feel comfortable at any point in time.
The first 18 videos in the slide show 100-frame long sequences of the Pendulum environment, i.e. six each for either no adversarial attack, an c-illusory attack (for which we added Lo penalty the le penalty in Algorithm [T} or an SA-MDP attack (Zhang et al., (2021a). The subsequent 24 10-frame long sequences of the CartPole include the same attacks as for Pendulum, plus an additional six sequences under an MNP attack (Kumar et al. 2021). Note that MNP attacks cannot be applied to the Pendulum environment, as it has a continuous action space. For all attacks, the budget B was 0.2. Within both groups, the order the videos choose for each video between correct and false, where false indicates that subjects find that the system observed in the respective video is not the same as the one observed in the introductory video. We likewise refer to video marked false as a suspicious video. Table 3 includes all empirically estimated probabilities and standard deviations for classifying segment from given class false.
shown was randomised. Subjects
Result analysis. We find that subjects reported one-third of unattacked sequences as false. Subjects clas- sified sequences under an MNP or SA-MDP (Zhang et al. 2020) attack as false with probability higher than 80%, while categorizing illusory attacks as false with probability of only 32.4%. We computed the double- sided z-statistics and were able to reject both the hypothesis that Pffalse SA PA-MDP)=P(false no attack) and the hypothesis that P(false (Na)= no attack) for 0.05, while the hypothesis that P(false E E-illusory attack (false no attack) cannot rejected. conclude that subjects were able to distinguish SA-MDP and MNP attacks from unattacked sequences while heing unable to distinguish illusory attacks from unattacked sequences.
A.9 RUNTIME COMPARISON
We investigate wall-clock time for training different adversarial attacks. We first recall that MNP attacks (Ku- mar et al. 202T) as well as perfect illusory attacks do not require training. For SA-MDP attacks (Zhang et al. 2021a) and e-illuoory attacks, training time highly dependent on the complexity of the environment, with lower training times for the CartPole and Pendulum environments, and higher training times for Hopper and HalfCheetah environments. All reported times are measured using an NVIDIA GeForce GTX 1080 and an Intel Xeon Silver 41 CPU, We trained SA-MDD attacks for 6 hours, and 12 hours in the simpler and more complex environments respectively. We trained e-illusory attacks for 10 hours, and 20 hours in the simpler and more complex environments respectively. At test-time, inference times for e-illusory attacks identical SA-MDP attacks as they only consist a neural network forward pass, Memory requirements are identical.
20


Reviews:
[{"review_id": "AZ96Q3na07", "reviewer": "Reviewer_JhVN", "summary": "The authors propose a novel attack on sequential decision-makers. In their attack framework, the authors introduce an information-theoretic detectability constraint, where they minimize the KL distance between the trajectory density of the victim’s observation process and the unattacked environment’s trajectory density. Thereby, they ensure that the difference in the distributions is below a specific detectability threshold that makes the attacks difficult or even impossible to detect. The authors propose a dual ascent algorithm to simultaneously optimize the adversarial reward and the detectability objective and demonstrate that their attack achieves a better detectability success trade-off compared to previous attacks for automatic detection methods and human assessment.", "questions": "* Could the authors elaborate on the runtime of their attack and its scalability to more complex environments", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Clear presentation of related work and positioning of the paper in the literature\n* Concept is intuitively explained in Figure 1\n* I found the information-theoretic approach to designing difficult-to-detect adversarial attacks convincing. While the proposed approach may not scale to complex real-world scenarios, it may be used as a baseline for more efficient variations of the proposed algorithm\n* To the best of my knowledge, the detectability success trade-off of the proposed attack framework is considerably better than in previous work\n* Diverse benchmarks with different robustification methods and an additional human study (although limited complexity of benchmarks)", "weaknesses": "* (Minor) The paper introduces a considerable amount of notation. I feel like the main results could be conveyed with less mathematical notation, which could be moved to the appendix (a lot of basic RFL notation and concepts)\n* (Minor) The evaluation is limited to simple simulated environments\n* (Medium) I'm missing a runtime analysis to compare the efficiency of the different attacks. This would also highlight if the framework could be scaled to more complicated problems", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NSYZJloecX", "reviewer": "Reviewer_5HNt", "summary": "This paper considers adversarial attacks against deep reinforcement learning agents which work by directly modifying the observations input to RL policies. The authors point out a flaw of previous work in this area: the observations which are produced by attacks may be very unrealistic and thus detectable by the victim, which could lead to mitigation of the attack's effects. They aim to remedy this by constraining attacks to be illusory, a set of properties they introduce which mean an attack is in some way hard to detect. Illusory attacks are defined by placing a constraint on the KL divergence between the distribution over sequences of observations seen by the victim policy when under attack and when not under attack. This is a reasonable constraint because optimal hypothesis testing can be shown to become harder as the KL is lower, i.e., it becomes less and less feasible for the victim to detect an attack. The authors devise an algorithm to generate KL-constrained attacks and show that it works well in practice, evading a state-of-the-art anomaly detector and human detection.", "questions": "* In Figure 5, it seems strange to the detection rate averaged across environments for human detection and automated detection when one is only evaluated on two environments and the other is evaluated on all four. Since the automated detection rate is already shown in Figure 4, why not just show the automated detection rate for the environments where human detection was tested in Figure 5?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The idea of undetectable attacks in the MDP setting is really nice and, to my knowledge, novel. It seems like a natural question to see what the information-theoretic limit of undetectable attacks is and look at how attacks can be made to approach this limit. The quality of the mathematical analysis and of the experiments seems generally good.", "weaknesses": "My main concern with the paper is the presentation, which is very dense and at times difficult to understand. For instance, many of the equations in Section 4 are presented without much motivation or intuitive explanation. It seems at some points that the authors include additional mathematical details which are unconnected to the main message of the paper (e.g., discussing nonparameteric density estimation on line 188); it might be best to omit these and devote the room instead to making other points more clear. I have listed some specific problems with the writing below.\n\nAdditionally, I found many of the figures had only limited explanation. I don't find Figure 1 to be so clear—there is a lot going on and I could only understand it completely after going through the rest of the paper. Figure 2 is presented with only high-level explanation in the paper, and it's not clear what all the arrows and lines mean. Algorithm 1 is confusing since $p(\\emptyset)$ and $p(o_\\\\text{old}, a_\\\\text{old})$ seem to be undefined and there is no caption.\n\nSpecific issues:\n * Line 132: I believe it should be $\\text{supp} \\\\: \\xi(\\cdot \\mid s) \\subseteq \\mathcal{B}(s)$.\n * The definition of additive perturbations on line 134 is a bit confusing. First, why are the perturbations themselves in the state space $\\mathcal{S}$. Shouldn't they be in $\\mathbb{R}^d$, where $\\mathcal{S} \\subseteq \\mathbb{R}^d$? Second, the $\\delta$ notation for a point mass is not previously introduced.\n * Line 225: I believe \"relative entropy\" is actually another term for KL-divergence, and what is mean is \"cross entropy.\" The notation used of $H[X, Y]$ is standard for \"joint entropy,\" though! So this is quite confusing—it would be best to clean up the notation.\n * In Algorithm 1, $\\hat{D}_\\text{KL}$ is not defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B50LBhlI0d", "reviewer": "Reviewer_iko8", "summary": "The authors propose an attack on RL called the illusory attack.  This is an observation-space attack which enforces that the attack is difficult to detect by enforcing that the unattacked trajectory density and attacked trajectory density is bounded by $\\epsilon$.  The authors demonstrate that this attack is significantly harder to detect compared to former attacks on RL by both OOD detectors and humans.", "questions": "- I'm a little confused by the definition of adversary score, what exactly is the normalization used to compute this value? The paper states that the value is normalized with respect to the \"highest adversarial return in each class, as well as the victim’s expected return in the unattacked environment\" but I don't understand what this means.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- paper is clear\n- illusory attacks are not as easily detected compared to prior attacks so they cannot be easily defended against by using an ood detector\n- experimental scope is good, authors compare to prior attacks for multiple RL problems and demonstrate with both ood detection and human detection that illusory attacks are difficult to detect", "weaknesses": "- the paper does not seem to give much guidance on how to design better defenses against these attacks/ how can illusory attacks allow us to design more robust RL techniques?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)