Adversarial Inception Backdoor Attacks against Reinforcement Learning
Authors: Ethan Rathbun
Alina Oprea
Christopher Amato
Venue: ICML 2025 poster (2025)
Keywords: backdoor attacks
adversarial machine learning
reinforcement learning
poisoning attacks
Primary area: social_aspects->security
TL;DR: First class of backdoor attacks against DRL with theoretical guarantees of attack success under natural reward constraints

Abstract: Recent works have demonstrated the vulnerability of Deep Reinforcement Learning (DRL) algorithms against training-time, backdoor poisoning attacks. The objectives of these attacks are twofold: induce pre-determined, adversarial behavior in the agent upon observing a fixed trigger during deployment while allowing the agent to solve its intended task during training. Prior attacks assume arbitrary control over the agent's rewards, inducing values far outside the environment's natural constraints. This results in brittle attacks that fail once the proper reward constraints are enforced. Thus, in this work we propose a new class of backdoor attacks against DRL which are the first to achieve state of the art performance under strict reward constraints. These ``inception'' attacks manipulate the agent's training data -- inserting the trigger into prior observations and replacing high return actions with those of the targeted adversarial behavior. We formally define these attacks and prove they achieve both adversarial objectives against arbitrary Markov Decision Processes (MDP). Using this framework we devise an online inception attack which achieves an 100% attack success rate on multiple environments under constrained rewards while minimally impacting the agent's task performance.

Adversarial Inception Backdoor Attacks against Reinforcement Learning
Ethan Rathbun
Alina Oprea'
Christopher Amato
Abstract
This necessitates further investigation into the capabilities of adversarial attacks DRL to provide practitioners with insights into effective defense strategies.
Recent works have demonstrated the vulnerabil- ity of Deep Reinforcement Learning (DRL) al- gorithms against training-time, backdoor poison- ing attacks. The objectives of these attacks are twofold: induce pre-determined, adversarial be- havior in the agent upon observing fixed trigger during deployment while allowing the agent to solve its intended task during training. Prior at- tacks assume arbitrary control over the agent's s rewards, inducing values far outside the environ- ment's natural constraints. This results in brittle attacks that fail once the proper reward constraints are enforced. Thus, in this work we propose new class of backdoor attacks against DRL which are the first to achieve state of the art performance under strict reward constraints. These "inception" attacks manipulate the agent's 's training data in- serting the trigger into prior observations and re- placing high return actions with those of the tar- geted adversarial behavior. We formally define these attacks and prove they achieve both adver- sarial objectives against arbitrary Markov Deci- sion Processes (MDP). Using this framework we devise an online inception attack which achieves an 100% attack success rate on multiple environ- ments under constrained rewards while minimally impacting the agent's task performance.
In this work, we focus on gaining a deeper understanding of backdoor poisoning attacks, which manipulate an agent's training to enable direct control over its behavior during deployment upon encountering a predefined trigger. State- of-the-art DRL backdoor attacks (Kiourti et al., 2019; Cui et al., 2023; Rathbun et al., 2024; Wang al., 2021) assume an adversary with arbitrary control over the magnitude of the agent's 's reward signal implanting extremely large positive rewards and negative penalties into randomly selected states. This strongly biases the agent's projected returns, causing the adversarial behavior to appear optimal whenever the trigger is observed.
However, RL environments have natural upper and lower bounds to their reward functions which these attacks ig- nore, exposing them to detection as outliers. Additionally, many practical RL implementations enforce reward post- processing such as clipping or normalization (Mnih et al., 2013; Huang et al., 2022) which can effectively erase per- turbed rewards. These restrictions on the adversary pre- vents them from sufficiently biasing the agent's projected returns, resulting in attack failure. Developing a principled poisoning strategy under these constraints poses a signif- icant challenge as manipulating randomly selected states and rewards alone during training is no longer effective. Our insight leads us to develop a novel adversarial strategy that carefully selects high-return time steps to poison in the agent's 's training data and manipulates their actions to re- flect the adversary's desired behavior. Manipulating actions that result in high returns, which are approximated with a approach (Mnih et al., 2013), is key to compensate for the restrictions constrained rewards. Through these strategies we provide multiple contributions towards our understanding of more realistic, reward con- strained backdoor attacks against reinforcement learning. Specifically we:
1. Introduction
The wide-spread applicability of DRL (Mnih et al., 2013; Schulman et al., 2017) in security critical domains such as automated cyber defenses (Vyas et al., 2023), self-driving vehicles (Kiran et al., 2021), robotic warchouse manage- ment (Krnjacc et al., 2023), and space traffic coordina- tion (Dolan et al., 2023) makes it target for external ad- versaries wishing to influence the trained agent's 's behavior.
Deep
KKhoury College of Computer Sciences, Northeastern Uni- versity, Boston, MA, United States. Correspondence to: Ethan Rathbun <rathbun. saaatbbun.enothhcasteen... northccateenn.ddu>
Prove the limitations of prior poisoning attacks in achieving attack success against general DRL envi- ronments while respecting reward constraints.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
Formulate the adversarial inception attack framework

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
(Figure which uses novel action manipulation tech- niques to guarantee backdoor attack success under con- strained rewards while maintaining the victim agent's performance in their intended task.
to explore the targeted action at more often during train- ing. Rathbun et al. (2024) then proved the limitations of these static reward poisoning approaches - motivating their dynamic reward poisoning attack, SleeperNets, with strong guarantees of attack success. Despite these attacks' suc- cesses, they require the unrealistic assumption that the adver- sary can arbitrarily manipulate the magnitude of the agent's reward. This assumption violates intrinsic constraints on rewards given in tasks such standard Atari Benchmarks where rewards are constrained into [0, 1] range (Mnih et al., 2013). Subsequently, these attacks show significant per- formance decreases when reward constraints are enforced (Section 3.2). Other works have studied different forms of training time attacks such as adversarial cheap talk (Lu et al., 2023) and policy replacement attacks (Rangi al., 2022).
3. Develop a novel backdoor poisoning attack "Q- "Q-Incept" leveraging adversarial inception increase in attack success over prior attacks under real- istic reward constraints.
to achieve significant
4. Provide in-depth evaluation of Q-Incept on environ- ments spanning Atari game playing, cyber network defending. simplified self driving, and safety-aware navigating tasks (Code is available on github).
In contrast to training time attacks, many works have stud- ied test time attacks in RI. which target fully trained agents employing static policies during deployment. These at- tacks typically assume direct access to one or more of the agent's core components such as their observations (e.g., sensors) (Linn et al., 2020), environment (Gleave al., 2019), or actions (e.g., robotic controler) (Tessler et al., 2019; Liang et al., 2023; McMahan et al., 2024; Franzmeyer al., 2022). Of particular relevance which differ from our training time Q-Incept attack in terms of objective and methodology. While Q-Incept alters the policy learned by an agent through manipulating its training data, these test time attacks aim to minimize the agent's return test time by directly changing their actions.
Training
Deployment
Tligper
Trigper
Chosen Action: Turn Right Transition: Turn Right
the action manipulation methods
Chosen Action: Turn Right
Transition: Forward
Figure 1. Visualization of inception attacks. During training the adversary manipulates agent's stored trajectories offline in- serting the trigger into the agent's prior observations and replacing optimal actions (Forward) with the target action (Tum Right). At test time the agent has learned to expect optimal outcomes when Turns Right upon observing the trigger, causing it to drive the road. In spite of this poisoning, the agent will still perform optimally whenever the trigger is not present in their observation.
3. Problem Formulation
In backdoor attacks against DRL there are two primary parties - the victim and the adversary. The victim at- tempts to train an agent to solve some benign MDP M (S, A, T, where the state space C $ is subset of some larger space (c.g., set of all possible 32x32 im- ages) and the action space A is discrete. The agent is trained with learning algorithm L(M) which returns pol- icy S x A + [0, 1]. The adversary induces the agent to instead train with respect to an adversarially modified MDP M' = (S USU US. A, R', T', 7, 6, B) where S + S applies the trigger to given state, S,- {8(s) Vs E 5} is the set of poisoned states, Ht' is the adversarial reward function, T" is the adversarial transition function, and B E [0, 1] bounds the frequency with which the agent soned states. Formally, T' and R' must respect the following constraints with respect to all s, E S and a E A:
2. Related Work and Background
When executing a backdoor attack, the adversary manip- ulates the Markov Decision Process (MDP) which an RL agent is being trained optimize. MDPs are often de- fined M (S, A, R, 7) where S is the set of states in the environment, A is the set of possible actions for the agent to take. R S x A x S + R is the reward function, T S x A x 5 + [o, 1] represents the transition proha- bilities between states given actions, and y E [o, 1] is the discount factor.
transition to poi-
Backdoor attacks against DRL were first explored by Kiourti al. (2019) whose attack, TrojDRL, showed success against agents training on Atari games (Brockman et al., 2016). Multiple other works (Wang et al., 2021; Yang et al., 2019; Yu et al., 2022: Cui et al., 2023) have used similar ap- proaches in different domains - all statically altering the agent's 's reward to a fixed tc., and many forcing the agent
(1 (1-3)T7 (5- (T (a, T(s'la,s)
(1)
l' a, 8') = R(s,
(2)
These constraints reflect the practical limitations of the at- tacker as they can not directly alter the source code of the
2

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
environment simulator, nor can they modify the dynamies of real world environments. Instead, the attacker modifies the data which the agent is training on, inserting the trigger pattern ò into their state with probability / after natural environment transitions have occurred. Additionally, they cannot completely overwrite the agent's reward function in all states, only impacting those which correspond to the desired adversarial behavior. This greatly limits the ad- versary's is capabilities, making for challenging but more realistic attack. Note that these constraints do not prohibit the adversary from altering transitions out of poisoned states T(s'la 0(s) in theory, although in practice the adversary merely alters the agent's training data. These constraints are implicitly enforced and discussed in prior works (Kiourti et al., 2019; Rathbun et al., 2024; Cui al., 2023), but we are the first to formalize it here.
RL environments have natural upper and lower bounds to their reward function in order facilitate the convergence of cumulative returns. Additionally, practical implementations of RL algorithms post-process the agent's 's rewards, often clipping or normalizing them within a tight interval (Mnih et al., 2013; Huang et al., 2022). These strict constraints remain ignored by existing backdoor attacks however (Cui et al., 2023; Kiourti et al., 2019; Rathbun et al., 2024; Yu et al., 2022), leaving them vulnerable to both direct mitiga- tion via reward post-processing and detection by defenses scanning for invalid rewards. Specifically, SleeperNets and TrojDRL utilize dynamic (R²) and static (It") reward poi- soning strategies, respectively, as defined below given target action a=:
(6(s,, = 1[a=at]->-4 (8) R ( = c. a1aa=at]-aa+
(5)
3.1. Adversarial Objectives
for some states s, E S. Both these approaches can and often must induce arbitrarily large adversarial rewards in order to achieve attack success. For instance, consider the example MDP defined in Figure 2 with discount factor 7, Here, the agent has two actions in the "Start" state, at and a. When the agent takes action a they prosper, receiving a reward of +1 on every time step for a return of 1- overall. When they take action at they receive no reward and terminate immediately, receiving a return of 0.
We consider "targeted attacks" (Kiourti et al., 2019) in which the desired adversarial behavior is fixed target action at E A. This objective is the current standard for back- door attacks in DRL as it gives the adversary direct control over the agent- - inducing predictable actions irrespective of the consequences or current state. Thus the adversary's s objective is to induce the agent to learn a poisoned policy a+ ((M)) which takes action at with high probability when observing the trigger:
Attack Success: max,+ 1 EEEEs (6E)) (6(s),
III
Terminate
0 0
Start
0la
Prosper
1|at,
(3) The attack must be stealthy, however, requiring the attacker to minimize the likelihood of detection by maintaining the agent's is return in benign states. The most relevant definition of stealth can vary depending on the application domain, so in this work we use the most established and well defined notion of attack stealth in the literature (Rathbun et al., 2024; Kiourti et al., 2019) defined below:
Figure 2. Simple MDP for which prior backdoor attack formula- tions fail to achieve attack success.
Now let's assume the MDP is impacted by backdoor attack with target action a+ using static or dynamic reward poison- ing, as defined in Equation 5. Note that here the value of the "Prosper" state approaches infinity as 7 approaches 1. There- fore, the adversary must also induce perturbed rewards with absolute values approaching infinity for the target action at to be optimal, despite the natural rewards of the MDP being bounded in [0, 1]. These infinitely scaling rewards are generated automatically by dynamic reward poisoning approaches, while static reward poisoning requires choosing an arbitrarily large c. We also observe this in practice, as demonstrated empirically on the Atari Qe environment in Section 6 Figure 4, the unconstrained poisoning strategies of TrojDRL and SleeperNets result in rewards as high as +-6 and as low - despite Q*bert's 's reward range of [o, 1).
Stealth: min 1 ExEs VV - M (+)II
(4)
where ग ~ L(M), and V.M (s), V (s) are the expected val- ues of policies v and + in MDP M respectively given state (Sutton& & Barto, 2018). Thus the adversary's 's objective is to minimize the difference in value between an unpoisoned policy L(M), and poisoned policy m+ [(M'). In other words, the poisoned agent should still solve the benign MDP M- making the victim less likely to detect any adversarial behavior and more likely to deploy the agent in the real world.
In both scenarios, SleeperNets and TrojDRL fail to achieve attack success if their rewards are clipped or normalized within a [0. 1] range by the victim's 's training algorithm. Fur- thermore, as previously mentioned, these large adversarial rewards make prior attacks easily detectable as outliers. For
3.2. Why Reward Constrained Attacks?
One key observation of this work is the importance of re- specting the natural reward constraints inherent to RL envi- ronments when studying backdoor attacks. Specifically, all
3

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
instance, the simple, rule-based detector D defined be- low given input reward rt and benign reward function R detects both TrojDRL and SleepenNets in the prior exam- ples with no false positives and without impacting agent performance.
the adversary poisons fewer time steps, allowing the agent to more easily optimize the benign MDP, but potentially decreasing the attack's s success rate. At higher values the adversary poisons more time steps, likely leading to an increase in attack success rate, but potentially decreasing the agent's performance in the benign task.
D(r))= adversarial rf < Rn] Vr, > sup benign
otherwise
4. Theoretical Analysis
(6) Therefore it is in the adversary's 's best interest to ensure their rewards stay within the upper and lower bounds of R to avoid detection. Motivated by the above exploration, we add an additional constraint to the adversary's 's reward function R' with respect to the benign reward function R::
Here we first prove that prior reward and action manipula- tion techniques are insufficient for attack success. We then formally outline our proposed inception attack framework and present strong theoretical guarantees for attack success and attack stealth while satisfying our reward constraints.
sup[R'] I su][ and inf[ > inf [R]
4.1. Prior Action Manipulation is Ineffective
Thus, the adversary must induce rewards no larger or smaller than those given in the benign MDP M. Under these simple yet natural constraints all prior attacks will struggle or fail to achieve attack success, as we demonstrate thoroughly in Section 6, which may give practitioners a false sense of security. Therefore, in the following sections we formulate new class of backdoor attacks against DRL which respect these constraints while also have strong guarantees of attack success across general MDPs
In Section 3.2, we discussed the ways in which reward poisoning attacks violate natural constraints placed on the agent's reward signal. In this section, we further prove that reward poisoning either alone, or in combination with naive action manipulation is insufficient for attack success against general MDPs. Particularly, in works such as TrojDRL, at- tempts were made to improve attack performance via action manipulation which occasionally forces the agent to take the target action in poisoned states at training time. We call this approach "forced action manipulation" and model it as an adversarial policy n- which alters the agent's true policy t so they forced to take action at with probability p in poisoned states, otherwise leaving the policy unmodified:
3.3. Threat Model
In this work we consider the outer-loop threat model defined by SleeperNets, which provides the attacker global visibil- ity into a history of observations before modifying states, actions and rewards. The outer-loop threat model, while assuming the same level of adversarial access to the train- ing process as the inner-loop threat model (Kiourti al., 2019), offers increased versatility (Rathbun et al., 2024). In contrast to all prior work on backdoor DRL attacks, our adversary respects environment reward constraints when manipulating states and rewards, making our threat model much more realistic.
T (sp: = p1la a+ + (1 p))pppp)spppp a)
(8)
for some poisoned state 8p Sp, where 1 is the indicator function. The key observation here is that the ad adversary is directly manipulating the agent's ' policy during training without accounting for any negative outcomes of the target action. Subsequently, forced action manipulation makes no contribution towards increasing the expected return of the target action in any poisoned state 5(s).
Under our threat model, the adversary observes episodes H = [(s, a, r)) of size l generated by the agent in M during training. They then alter states 8t, actions de, and rewards r, stored in the trajectory before the agent uses them in their policy optimization. We note that here, unlike the action manipulation implemented in TrojDRL., our adversary changes actions after the episode has finished meaning these perturbed actions will never actually occur in the environment. The adversary is also constrained by poisoning rate parameter B which bounds the proportion of total training time steps in which the adversary can apply the trigger to the agent's 's current state. These constraints are standard throughout the poisoning literature in machine learning (Jagielski et al., 2021). In DRL 3 acts similar to a hyper parameter for the adversary. At lower values
For instance, let's again consider the example MDP defined in Figure 2, and assume it is impacted by backdoor at- tack with target action a+ using static or dynamic reward poisoning, as defined in Equation 5, and forced action ma- nipulation, as defined in Equation 8. We can then evaluate the value of each action within the adversarial state 6Staart) in Table 1. Since rewards in this MDP are constrained to [0, 1J. both static and dynamic reward poisoning will give the agent a reward of 1 if they take action at or 0 otherwise. Thus, we need not analyze them separately. We see that forced action manipulation has no impact on the value of either a or a - forcing the agent to take actions during training does not alter the value of those actions, it only impacts the agent's 's training- time exploratory behavior.
4

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
Q- Vaauuss
Manipulation
No Manipulation
Thus, given any E Sp U S the MDP transitions a poi- soned state s' E Sp with probability 3 and otherwise transi- tions to a benign state. When in a poisoned state, and if the agent takes action at, the MDP will transition according to the optimal action as calculated by Φ guaranteeing at is also optimal. Furthermore, all transition probabilities are computed with respect to the benign states 8-,(s,) for Sp E Sp. This maintains the underlying transition dynam- ies of the environment with respect states and actions, retaining optimality between policies in M and M'.
Q4 (8(Start), Q ()(
1
Table Q-values for an arbitrary policy ι in our example MDP (Figure 2) under a constrained backdoor attack with and without forced action manipulation. Here at is suboptimal with and with- out action manipulation for any 2'
This counterexample proves that both dynamic and static reward poisoning are insufficient for attack success against general MDPs, even when forced action manipulation is used. Note that, since the values of the "Terminate" and "Prosper" states are fixed with respect to any π and the Q- value provides the first action taken, this result holds for both on and off policy learning.
The next key piece of adversarial inception is our adver- sarial reward function R' whose goal is to give the agent extra rewards or penalties when they do or don't 't choose a" in poisoned states, respectively. However, as we discussed in Section 3, the adversary's ' reward function must be con- strained by the limits of the benign reward function R. To account for this we define the function to give the agent as much reward or penalty as possible while remaining within the bounds [L, U] where L = in)( U = st)(
4.2. Adversarial Inception Framework
In this section, we formalize our new adversarial inception framework which overcomes the shortcomings of prior at- tacks when subject reward poisoning constraints. To achieve this, we design an adversarial MDP with a novel transition function T' and reward function R'. Under this transition function, the agent is guaranteed to transition with respect to the optimal action given they choose the target action at in a poisoned state Sp E Sp. We enforce this through the function below
T: AAARXx + R
(11)
min fr Pー U] maxir - - L]
if at
T(a,r,f) =
otherwise
where f the reward received on the previous time step. We want to cancel out this 7 term in the value of the prior benign state E S. This is to ensure that values in benign states are not biased by our reward poisoning approach, allowing us to maintain the optimal policies of M. Returns in prior benign states will receive t discounted by a factor of 7, hence the division by y present here. Thus we define Ht as
Sp x A
(9)
arg max 1Q (8 (sp),
a at otherwise
(Spp a, T)
a
R' : (SUS)) x A x SS x R
(12)
Adversarial Reward Function H' Reward
The goal of @ is to calculate and return the optimal action in state o (xp) given π if the agent chooses the target action at. In practice, the adversary doesn't have direct access to QT"" so approximation is necessary. This will be explored in more detail in Section 5. Further note that for our theoretical results we assume S and Sp are disjoint sets and ō is bijection between them. The adversary never needs to compute 5- (sp) in practice, however, as they can directly observe 8 before the trigger is applied. & is then incorporated into the transition function I" defined below where II is the set of all valid policies in M':
8
s
S
R(s,a,s')
S
Sp
R(s,0,8 ((s,a,8-(8))) )= + Barnrll r(a,-) 7(a, R(5 (8), (),
Sp
SUSp
where e ((a, -)] is used our aforementioned bias correction. R' is designed to work in tandem with T' to guarantee attack success when taking action at in some poisoned state Sp Sp the MDP will not only transition according to the optimal action as calculated by & but the agent will also receive an increased immediate reward as specified by т. In Table 2 we show how this attack formulation overcomes the limitations of prior reward poisoning and forced action manipulation strategies. Here, we once again return to our example MDP from Figure 2, except now, under inception poisoning, at is finally the optimal action in the poisoned state 8(start) for any y.
T' US X Ax (SU 11 - [0
(10)
Adversarial Transition Function 7
8
Transition Probability (1- B) a(s.a,s' a.s)
S
Sp
3.T(s, 8, (s'))
Sp
S
(1- () ( (s). 0(s T) 8)
When the agent takes action a under adversarial inception, and according to d the agent will instead transition with
Sp
Sp
8- T(8- a(s, 8-1 (s))
5

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
Q- Valuusss
Inception
No Inception
Lemma 2 Vad (8) 2 (s) Vs E S, E HI Therefore, the value of any policy in the adversarial MDP M' is greater than equal to its value in the benign MDP M for all benign states E S.
2Hd (o(Start),at) Q (5(Start), a)
1
Table 2. Q values for an arbitrary policy и in our example MDP (Figure 2) under backdoor attack with and without adversarial in- ception. Here at is the optimal action under adversarial inception.
Through these lemmas we now have a direct relationship between the value of a policy in the benign MDP M and the adversarial MDP M'. With this we can prove Theorem 2.
Theorem 2 V.M (s) 2 V. (s) Vs S, E II + V (+) > VA (8) V8 E 8, E II Therefore, x' is op- timal in M' for all benign states E S if and only if x is optimal in M.
respect to action a, receiving a future return of ب In ad- dition to this, according to т, the agent will receive a bonus reward of +1 for taking action a+. When choosing action a the agent still receives the future return of 1 but they don't It receive any immediate bonus reward, thus at is the optimal action. With no adversarial inception the adversary can no longer transition the agent with respect to a upon choosing at, thus they can only give the agent an imme- diate reward of +1, making a the optimal action. The key observation here, as demonstrated in Table 1 and Table 2, is that adversarial inception accounts for the negative impacts of at during training while forced action manipulation does not. We claim that this result generalizes across MDPs, allowing us to prove that optimal policies in M' not only maximize attack success but also attack stealth, all while receiving rewards that stay within the bounds of R. We formalize these claims in the next section.
Therefore, given Theorem and Theorem we know an optimal policy in M' solves both our objectives of attack success and attack stealth while satisfying our reward con- straints. Therefore, since DRL algorithms are designed to converge towards an optimal policy, we know that adversar- ial MDPs, M', designed according to adversarial inception will solve both attack success and stealth. Formally derived proofs for all these results are given in Appendix A.I.
5. Adversarial Inception Algorithm
Algorithm 1 Generalized Inception Attack (Q-Incept)
Initialize Policy æ, Replay Memory DD max episodes N. Empirical Lower Bound É. Empirical Upper Bound i Input algorithm L. MDP M, poisoning rate B, trigger d I: for i f N, do
4.3. Theoretical Guarantees of Adversarial Inception
Vietim samples trajectory H {(s, of
Here, we present the theoretical guarantees we have proven about adversarial inception, with Theorem I and Theorem 2 relating to attack success and attack stealth, respectively. The outcome of Theorem I is fairly intuitive based upon our prior explanations of T' and R' if the agent is guaranteed an optimal outcome when choosing at in poisoned states, then a always optimal.
size p from M given policy m
3:
L. +- min[T., min [re]]. Ú max((, max(ri]]
4:
Select C' I of size [8 [H] using Fo(st.
5:
for all (s,a,r) H' do
6:
St + ö(s)), Told Tr + a if F Fo (t) > 0
7:
8:
rt r U af = at else L
1 (sp. a)] (sp: 11. Thus, the optimal action of any policy in M' in any poisoned state Sp is the targeted action at.
Theorem 1
arg max,
at Vap E Sp, E
9:
T--1 min(max(rr-- - 7(r+ - rolu), L1, 01
10: Victim stores H in D, updates π with L given D
11: Update Q for metric Fo given D using DQN
Theorem 2, on the other hand, is not as obvious. When performing action manipulation according to ó it is unclear how this will impact the dynamics of the MDP and thus the optimal policy. Therefore we proceed progressively towards our proof of Theorem 2 via Lemma I and Lemma 2. One key observation is that, if policy πT is optimal, then & does not impact the agent's 's chosen actions. Thus Lemma holds. Next is the observation that e only ever increases the value of a policy since it forces the MDP to transition optimally. Thus Lemma 2 follows.
In Algorithm 1, we present framework for inception at- tacks against DRL with the aim of replicating the adversarial MDP M' from Section 4. In M' we use & to force the MDP to transition with respect to optimal actions when the agent chooses target action at, but this is not possible under our threat model. The adversary does not have direct access to QM' (s) nor can they change the agent's 's actions during an episode H = [[ss, a, r).). Thus we take more indi- rect approach in steps 6 and 7 incepting false values in the agent replay memory D so they think they took action at in poisoned state 8(s,) when, in reality, they chose and transitioned with respect to some high value action o, in
Lemma 1 V.M (s) 2 V. (s) Vs E SU Sp, 11 V.M = V.M Therefore af is optimal then its value in M' is equal to its value M.
6

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
benign state 8t- We further apply DQN to the agent's benign environment interactions to create an estimate, Q(s, a), of the MDP's optimal Q function. With this, we can create metric F&. like the one defined in Equation 13 for Q-Incept, to approximate the relative optimality of each action.
Merge (Leurent, 2018), and Safety Car (Ji et al., 2023) ex- tend the diversity of our analysis to other domains spanning cyber network defending, simplified self driving, and safety- aware robotic navigation tasks, respectively. This allows us to verify the effectiveness of Q-Incept across multiple task domains which share little overlap. Further experi- mental details and results are given in Appendix A.2 and Appendix A.4, respectively.
Fg(s,a) = Q(s, a) - Ey || [Q(s') a')]
(13)
where we approximate the agent's current policy x using the maximum entropy, softmax distribution (Ziebart et al., 2008) (s'' R softmax [Q(s',-)1, -)] This construction allows us to approximate & by finding time steps in which the agent took near optimal actions in step 4. Time steps with a high, positive value are advantageous for inception in step 7, changing a f at in D, as the agent associates the target action with positive outcomes in poisoned states.
In Table 3, present the performance of our attack on each environment across two different poisoning rates B each. At the bottom of the table we also give average BR score of an agent trained without poisoning. Across all seven environments Q-Incept outperforms both SleeperNets and Troj in terms of ASR while maintaining better or comparable BR scores. We also see that Q-Incept is the only method which consistently scales in terms of ASR as the poisoning budget 33 increases with SleepenNets and TrojDRL sometimes dropping in performance under a larger 3 in cases such as Atari Frogger and Pacman.
Conversely, time steps with high, negative values are also useful to poison (if at * at), as the agent associates non- target actions with negative outcomes in poisoned states. In Q-Incept we use the absolute value of Fo a) as softmax logits to weigh how we sample H' S H in step 4. This allows us to bias our sampling towards high or low value states in Fo while maintaining state space coverage. In steps 8-9 we opt to implement slightly stronger version of which perturbs the agent's 's rewards to U or L if at at or # at, respectively. In practice this results in better attack success rates over a direct implementation of 7 while also attaining similar levels of episodic return.
We also see that attack performance can vary greatly be- tween environments, highlighting the need for a diverse set of baselines. Atari Breakout, a baseline also studied by Rathbun et al. (2024) and Kiourti et al. (2019), seems to be the easiest environment to poison by far. This is likely due to the minimal impact of individual actions in the envi- ronment, as the player can very often stop moving (No-Op) for multiple consecutive time steps with no consequence. Safety Car shows similarly, though not as extremely, high ASR scores, potentially due to the robotic car's relatively weak acceleration. Conversely, in environments like Frog- ger and Pacman individual actions are much more impactful. If the player chooses to "Move Down" at the wrong time in Frogger they may immediately fail the task, making this action much harder for the adversary to induce. Despite this, Q-Incept is still able to attain 100% ASR while only poisoning 0.3% of time steps in both Frogger and Pacman.
6. Experimental Results
Here, we evaluate Inceet against TrojDRL and Sleeper- Nets, representing the state of the art in forced action manip- ulation and dynamic reward poisoning attacks. We perform our evaluation in terms of Attack Success Rate (ASR) and Benign Return (BR), relating to our objectives of attack success and attack stealth, respectively, defined below:
ASR(T (T 18)= = E.es ( (8(s)) BR( . [M) = - [V (so)]
We also see that CAGE-2 and Highway Merge were the hardest environments for any attack to poison, being the only cases where Q-Incept does not attain 100% ASR. Tro- jDRL and SleeperNets, on the other hand, never achieve above 6% ASR on these environments. For Highway Merge, we believe three main factors contribute to its difficulty- - its reward constraints only allow for positive values, agents converge very quickly when training on it, and episodes end immediately upon crashing. Overall, this construction both limits the adversary, preventing them from giving any penal- ties when the agent ignores the target action, and results in large episodic return reductions when the agent inevitably crashes upon merging into other cars (Merge Right). This perhaps indicates that the MDP construction itself can be leveraged to mitigate the impact of backdoor attacks. For CAGE-2 the reasons are less clear, though we suspect its
(14)
where so is (potentially random) initial state given by M and - is the poisoned policy we are evaluating, Both of these metrics are calculated in practice by averaging over 100 trajectories and 5 different initial training seeds. All attacks are evaluated under constrained reward poisoning, defined in Equation 7 requuiin each to restrict their reward perturbations to be within the min and max of the benign rewards they have observed so far (e.g., lines 3 and in Algorithm 1). We evaluate these attacks using cleanrl's im- plementation of PPO (Huang et al., 2022) on 7 environments. Atari Qett Frogger, Pacman, and Breakout (Brockman et al., 2016) represent standard baseline tasks in RL to verify the capabilities of Q-Inept on complex environments. Ad- ditionally, CAGE Challenge (Kiely et al., 2023), Highway
7

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
Environment
Qbert Move Right
Frogger Move Down
Pacman No-Op
Breakout No-Op
CAGE-2
Highway Merge
Safety Goal
Target Action
No-Op
Merge Right
Accelerate
B
0.3%
0.3%
0.3%
0.1%
1%
10%
0.1%
Metric
ASR
ASR
ASR
ASR
ASR
ASR
ASR
Q-Incept
100%
0.0%
100%
0.0%
100.0%
0.0%
100%
0.0%
93.21%
15.13%
61.60%
23.29%
100%
0.00%
SleeperNets
55.6%
39.3%
0.00%
0.00%
11.3%
6.9%
100%
0.0%
0.06%
0.12%
1.50%
0.53%
86.96%
29.12%
TraDRL
22.5%
20.7%
4.42%
9.88%
13.5%
4.5%
99.3%
0.45
5.64%
7.73%
1.20%
0.67%
54.04%
2.85%
Metric
BR
BR
BR
BR
BR
Q-Incept
18,381
882
392.3
37.4
583.0
116.2
460.8
20.3
-45.49
8.04
14.85
0.15
12.00
0.16
SleeperNets
16,840
1,982
366.5
50.9
665,5
281.7
443.4
13.8
-50.18
8.52
15.14
0.01
9.23
1.98
TrojDRI.
17.617
909
366.3
33,9
670.7
189.4
455.3
16.8
-53.11
8.71
15.11
0.03
10.86
1.13
0.1%
0.1%
0.1
0.05%
0.5
7.5%
0.05%
Metric
ASR
ASR
ASR
ASR
ASR
ASR
ASR
Q-Incept
100%
0.00%
89.5%
1.16%
60.1%
46.1%
100%
0.0%
30.61% %
15.01%
53.03%
25. 56%
100%
0.00%
SleeperNets
19.98%
4.39%
45.92%
9.17%
42.7%
41.4%
100%
0.0%
0.00%
0.00%
1.47%
0.84%
83.95%
13.45%
TrojRL
15.38%
3.05%
44.00%
10.63%
48.9%
30.3%
99.1%
0.5%
0.00%
0.00%
3.27%
3.56%
53.35%
9.51%
Metric
BR
BR
BR
BR
BR
BR
BR
Q-Incept
17,749
1,380
437.9
10.4
457.1
87.3
456.1
19.7
44.19
16.68
14,90
0.13
11,52
0.40
SleeperNets
17,320
1,481
412.8
11.8
584.9
89.0
456.5
15.5
-57.82
8,38
15.15
0.01
10.4
0.7
TrujDRL
17,715
1340
357.0
52.1
712.0
191.6
443.5
25.9
-39.28
9.60
15.1
0.02
9.8
1.1
Metric
BR
0
BR
BR
0
BR
0
BR
0
No Poisoning
17,322
1,773
380.4
89.1
628.7
236.5
476.5
9.4
45.17
15.65
15.17
0.01
12.03
0.16
Table Comparison between Q-Incept, SleeperNets, and TrojDRL with constrained rewards against agents training on our seven environments at different 33 values. Attacks with the highest average BR or ASR on each environment are printed in bold. BR scores of unpoisoned agents given at the bottom of the table for comparison. Standard deviations are given next each result.
relatively large action space and the universal sub-optimality of the target action (No-Op) may be contributing factors.
we compare Q-Incept both with and without inception (step of Algorithm 1) on CAGE-2, referring to this naive version as (Q-Vanilla". Given this omission, we see immediate and significant drop in ASR, with Q- Vanilla failing to attain above a 5% attack success rate. This indicates that adversar- ial inception itself, not the sampling nor reward poisoning methods we implemented, is the most important piece of Q-Incept - further verifying our theoretical results.
6.1. Verifying the Importance of Inception
1.0
0.8
0.6
Second, in the top plot of Figure 4 we compare natural, un- poisoned rewards given in the Atari Qrbert environment to those induced by unconstrained Q-Incept, SleeperNets, and TrojDRL. Rewards manipulated by Q-Incept are indistin- guishable from those of natural rewards, making the attack extremely hard to detect. In contrast, unconstrained Sleeper- Nets and Tro both result in large, obvious spikes in reward that step far outside the environment's natural [o, 1] constraints. In the bottom plot, we showease the drop off in performance of SleeperNets and TrojDRL once these con- straints are enforced Q-Incept is the only attack capable of achieving high attack success.
0.4
0.2
0.0
0M
1M
2M
3M
4M
SM
Time Steps
Q-inept
D-aanila
Steepe flitss
ToORL
40
-60
-80
Retum
-100
-120
6.2. Evading Trigger-Based Universal Defenses
-140
Many defenses against backdoor attacks in DRL, such as Bird (Chen et al., 2024) and State Sanitization (Bharti et al., 2022), claim universal defense capabilities by detecting or sanitizing the adversarial trigger directly at test time. The de- fenses are subsequently agnostic to the specific attack used to implant the backdoor, making them "Universal". This universality comes cost, however, as the defense is now trigger dependent while many backdoor attacks, including Q-Incept, are trigger agnostic allowing the adversary to craft evasive triggers to break each defense.
-160
0M
1M
2M
3M
4M4
Time Steps
Steepenkers
Teepes
Figure 3. Comparison between Q-Ieecpt with and without adver- sarial inception on the CAGE-2 environment at B = 1%. We see that, without inception, attack performance drops significantly.
Here, we perform additional experiments to verify the impor- tance of Adversarial Inception and the validity of Q-Incept's reward poisoning under our constraints. First, in Figure 3
8

---

Adversarial Inception Backdoor Attacks against Reinforcement Learning
Reward
MAMMA
HMM
125
175
Time Steps
Q-incept
TrojDRL
SteeperNets
Benign
Figure 5. Example poisoned in the Atari Q*hert environment before (left) and after (right) state sanitization. A large artifact still present the top left the image even after sanitization.
1.0
0.8
0.6
1.0
0..4
0.2
0.6
0M
2M
6M
RM
10M
17M
14M
Time Steps
Ganuui
0.2
Figure 4. (Top) Rewards induced by SleeperNets, TrojDRL, and Q-Incept compared to benign behavior on Atari Qertrt Environ- ment. (Bottom) Both SleeperNets and TrojDRL see significant drop attack success under constraints, while Q-Incept achieves high attack success. Attacks given budget 3 0.3%.
Threshold
Evasive Retive
Erasive-ASR
Kaive
Figure 6. Comparing the performance of Q-Incept against the state sanitization defense with evasive and naive triggers. We plot (normalized) return attack success rate in terms of the possible SVD thresholds the defender can use.
7. Conclusion and Diseussion
For instance, the state sanitization defense proposed by Bharti al. (2022) leverages singular value decompo- sition (SVD) based method over clean dataset of states in outlier
In this paper, we provide multiple contributions towards a deeper understanding of backdoor poisoning attacks against DRL algorithms. We highlight how prior works assume an unrealistic adversary capable of implanting arbitrarily large rewards, and demonstrate their theoretical limitations these assumptions are violated. We then propose Adversar- ial Inception novel framework for backdoor poisoning attacks against DRL under realistic reward perturbation con- straints. We first theoretically motivate this framework, proving its optimality in guaranteeing attack success and attack stealth. We then develop a practical adversarial in- ception attack, Q-Incept, which achieves satte-of-thh-ant performance multiple environments from different do- mains, while remaining stealthy. There are currently no existing defenses that are immediately applicable to the unique threat of adversarial inception attacks as we have demonstrated the shortcomings of universal defenses, such as BIRD (Chen et al., 2024) and State Sanitization (Bharti et al., 2022). Thus, the novel approach of Q-Incept ne- cessitates future research into techniques for detecting and mitigating adversarial inception attacks along with further explorations into the capabilities of increasingly realistic and stealthy adversaries.
the environment in order to remove the trigger at execution time. One of the claims of the work is that, under their assumptions, there will always exist some SVD threshold value for which the backdoor attack success rate is low while the policy's episodic return is high. practice, this process doesn't 't completely remove the trigger, however, as shown in Figure 5. As result, an adaptive adversary first the same SVD procedure themselves, and then poison the agent recognize the sanitized trigger during training. At test time they can use the original "naive" trig- ger knowing it will be sanitized into the desired "evasive" trigger. In Figure 6 we evaluate I-Iectp against the state sanitization defense using a naive and evasive trigger, rep- resenting the left and right plots of Figure 5, respectively. Under the naive trigger the defender able reduce the attack success rate 18% while maintaining a high episodic return given threshold of 10-1 Under the evasive trigger, however, there exists no SVD threshold values for which attack success rate low and return high either the attack succeeds or the agent's 's return is completely ruined. Therefore the defense broken.

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
Acknowledgements
Detectability matters in adversarial attacks on sequential decision-makers, arXiv preprint arXiv:2207.10170, 2022.
This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA), under con- tract W912CG23C0031, and by NSF awards CNS-2312875, ENS-2331082 and FMitF 2319500. We thank Simona Boboila, Peter Chin, Lisa Oakley, and Aditya Vikram Singh for discussing various aspects of this project.
Gleave, A., Dennis, M., Wild, C., Kant, N., Levine, S., and Russell, S. Adversarial policies: Attacking deep reinforcement learning, arXiv preprint arXiv:1905.16115 2019.
Huang, S., Dossa, R. F. F.J., Ye, C., Braga, J., Chakraborty, D., Mchta, K., and Araújo, J. G. Cleanrl: High-quality single- file implementations of deep reinforcement learning algo- rithms. Journal of Machine Learning Research, 23(274); 1-18, 2022. URL http:://mmll org. haagg/aaprrss. v23/21 13342.hmmll
Impact Statement
In this paper we develop new class of training-timne time, back- door poisoning attacks against deep reinforcement leamning agents. Similar to any adversarial attack paper, it is possi- ble that a sufficiently capable adversary can replicate our methodology to implement real-world attack against a DRL system. Through highlighting this threat we hope that future practitioners DRL will begin developing counter- measures against inception attacks to mitigate their real- world impact. We also hope that future researchers study these attacks from defender's perspective to find ways to detect them at training or testing time, preventing damage from occurring. From practical and immediate stand- pointt we believe that DRL. practitioners should take steps to iso- late their DRL training systems such that adversarial access exceedingly difficult.
Jagielski, M., Severi, G., Pousette Harger, N., and Oprea, A. Subpopulation data poisoning attacks. In Proceed- ings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, CCS '21, pp. 3104-3122, New York, NY, USA, 2021. Association for Comput- ing Machinery. ISBN 978 9781450384544, doi: 10.1145/ 2460110.3485368.
URL https://doo /ttp://///oooo//...... 3485368.
Ji, J., Zhang, B., Pan, X., Zhou, J., Dai, J., and Yang, Y. Sufety-gymnasing GitHub repository, 2023.
References
Kiely, M., Bowman, D., Standen, M., and Moir, C. On autonomous agents in a cyber defence environment. arXiv preprint arXiv:2309.07388 2023.
Bharti, S., Zhang, X., Singla, A., and Zhu, J. Provable de- fense against backdoor policies in reinforcement learning. Advances in Neural Information Processing Systems, 35: 14704-14714, 2022.
Kiourti, P., Wardega, K., Jha, S., and Li, W. Trojdrl: Trojan attacks on deep reinforcement learning agents. arXiv preprint Stt:::99030663. 2019.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J.,
Schulman, J., Tang, J., and Zaremba, W. OpenAl Gym. arXiv preprint 1rXiv:1606.0153
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Al Sal- lab, A. A., Yogamani, S., and Pérez, P. Deep reinforce- ment learning for autonomous driving: A survey, IEEE Transactions on Intelligent Transportation Systems, 23 (6):4909-4926, 2021.
2016.
Chen, X., Guo, W., Tao, G., Zhang. X., and Song. D. Bird: generalizable backdoor detection and removal for deep reinforcement learning. In Proceedings of the 37th Inter- national Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Krnjaic, A., Steleac, R. D., Thomas, J. D., Papoudakis, G., Schäfer, L., To, A. W. K., Lao, K.-H., Cubuktepe, M.. Haley, M., Börsting, P., and Albrecht, V. Sealable multi- agent reinforcement learning for warehouse logistics with robotic and human co-workers, 2023.
Cui, J., Han, Y., Ma, Y., Jiao, J., and Zhang, J. Badrl: Sparse targeted backdoor attack against reinforcement learning. arXiv preprint arXiv:2312.12588 2023.
Leurent, E. An environment for autonomous driv- ing decision-making. https: http::///ittub.com. e neeurnt/hhggwwyye.
Dolan, S., Nayak, S.. and Balakrishnan, H. Satellite navi- gation and coordination with limited information sharing. In Learning for Dynamics and Control Conference, pp. 1058-1071. PMLR, 2023.
2018
Liang, Y., Sun, Y., Zheng, R., Liu, X., Eysenbach, B., Sand- holm, T., Huang. F., and McAleer, S. Game-theoretic ro- bust reinforcement learning handles temporally-coupled perturbations. arXiv preprint arxiv:2307, /2062, 2023.
Franzmeyer, T., McAleer, S., Henriques, J. F., Foerster, J. N., Torr, P. H., Bibi, A., and de Witt, C. S. Illusory attacks:
10

---

Adversarial Inception Backdoor Attacks against Reinforcement Learning
Lin, J., Dzeparoska, K., Zhang, S. Q.. Leon-Garcia, A., and Papernot, N. On the robustness of cooperative multi- agent reinforcement learning. In 2020 IEEE Security and Privacy Workshops (SPW), pp. 62-68. 68. IEEE, 2020.
Yu, Y., Liu, J., Li, S., Huang, K., and Feng, X. temporal- pattern backdoor attack to deep reinforcement learning. In GLOBECOM 2022-2022 IEEE Global Communications Conference, pp. 2710-2715. IEEE, 2022.
Lu, C., Willi, T., Letcher, A., and Foerster, N. Adversar- ial cheap talk. International Conference on Machine Learning, pp. 22917-22941. PMLR, 2023.
Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K.
Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial
Intelligence Volume 3, AAAI'08, pp. 1433-1438. AAAL Press, 2008. ISBN 978157735368..
McMahan, J., Wu, Y., Zhu, X., and Xie, Q. Optimal attack and defense for reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 38, pp. 14332-14340, 2024.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint 1312 2013.
Rangi, A., Xu, H., Tran-Thanh, L., and Franceschetti, M. Understanding the limits of poisoning attacks in episodic reinforcement learning. arXiv preprint a20i 13663, 2022.
Rathbun, E., Amato, C., and Oprea, A. Sleepernets: Uni- versal backdoor poisoning attacks against reinforcement learning agents. arXiv preprint arXiv:2405.20533 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 47774:7707.06347, 2017.
Stroock, D. W. An introduction to Markov processes, vol- ume 230. Springer Science & Business Media, 2013.
Sutton, R. S. and Barto, A. G. Reinforcement Learn- ing: An Introduction. The MIT Press, second edition, 2018. URL http:/ / i cennnoolleettee weea..ddde...... net/ book/the-beel book- 2nd. html.
Tessler, C., Efroni, Y., and Mannor, S. Action robust rein- forcement learning and applications in continuous control. In International Conference Machine Learning, pp. 6215-6224. PMLR, 2019.
Vyas, S., Hannay, J., Bolton, A., and Burnap, P. P. Au- tomated cyber defence: A review. arXiv preprint 00Xi::3003.0496,1 2023.
Wang, L., Javed, Z., Wu, X., Guo, W., Xing, X., and Song, D. Backdoorl: Backdoor attack against competitive inforcement learning, arXiv preprint arXiv:2105.005 2021.
Yang, Z., Iyer, N., Reimann, J., and Virani, N. Design of intentional backdoors in sequential models. arXiv preprint /902::9902 09972, 2019.

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
A. Appendix
Table of Contents
Section
Contents
Appendix A.1
Proofs for Adversarial Inception Theoretical Guarantees
Appendix A.1.1
Theorem
Appendix A.1.2
Lemma 0
Appendix A.1.3
Lemma
Appendix A.1.4
Lemma
Appendix A.1.5
Theorem
Appendix A.2
More Experimental Details and Hyper Parameters
Appendix A.2.1
TrojDRL and SleeperNets Parameters
Appendix A.2.2
Q-Incept Attack Parameters
Appendix A.3
Further Discussion
Appendix A.3.1
Motivation for Baselines
Appendix A.3.2
Motivation for Environments
Appendix A.3.3
Comments on Compute Overhead
Appendix A.4
Further Experimental Results and Analysis
Appendix A.4.1
Q-Incept Against DQN
Appendix A.4.2
Q-Incept at Different Poisoning Rates
Appendix A.4.3
Further Ablations
Appendix A.4.4
Training Plots for Remaining Environments
Appendix A.4.5
How often does Q-Incept Alter the Agent's 's Actions?
A.1. Proofs for Adversarial Inception Theoretical Guarantees
Capabilities of Existing Backdoor Attacks DRL in Comparison Ours
Attack
Q-Incept (Ours) SleeperNets
TrojDRL
BadRL
Reward Poisoning
Constrained
Dynamic
Static
Static
Action Manipulation
Inception
None
Forced
Forced
Attack Success Guarantees of Existing Backdoor Attacks DRL
Unconstrained Rewards
Yes
Yes
No
Constrained Rewards
Yes
No
No
Table 4. Summary of the currently known attack success guarantees of different backdoor attack strategics. New results proven in this paper are printed in bold. Of particular interest is the fact that attack success can guaranteed under bounded reward poisoning constraints if the attacker implements adversarial inception. Without inception there guarantees. Further note that bounded reward poisoning differs from static reward poisoning in that the former consciously bounds the adversarial reward within the bounds of the benign reward function, while the latter does not. Static reward poisoning is technically "bounded" but only according hyper parameter c, not according to the benign MDP itself.
In this section we provide proofs of all the results claim in Section 4 first proving Theorem 1, proceeding through proofs of Lemmas through 2, and subsequently proving Theorem 2. It should be noted that Lemma 0 is not main claim of our paper and is instead a convenient result adapted from part of proof by (Rathbun et al., 2024).
A. THEOREM
Theorem arg max, (QM' (sp: a)] = a Vsp E Sp. E II. Thus, the optimal action of any policy in M' in any poisoned
state 8p is at.
Proof. Here proceed with direct algebraic proof showing that Q.'² (sp, at) 2 Q." (sp, for all # at, First we will simplify the value of Q.M (sp, for some 8p E Sp- Note that here, without loss generality, we simplify r to
12

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
+ باع orr u-f when the target action is or isn't t taken respectively.
7
Let: Sp E Sp, 7 E 11, i U], a" = arg max 12 (6 (sp).a)
(15)
Q(sp
E as , 8R)lR'(s at,
,,,tt,s ,) + +VV2"
(16)
d'ESUS,
= Σ
(16-((p))1 8') ()) ,') f) + VV (s')
(17)
S'E
T( (sp). , ())[[(( [)) s' + 2 ー- L
Σ d'SSUS
+ gV (s')
(18)
y
Σ S'S
T( 1 (sp): a² 8)) a², 80) + VV (5')
(19)
Σ *'BUS
T(ō T(0-1(()), (sp),
p- L
7
= 2 (8 (sp), + P-L L
(20)
Since f E [2, U] we know that 1-ع > 0. From here we will simplify the value of Q2 (sp, a) for some E A such that
a * at.
()pp =
Σ ('s (s)' + +V (s')
(21)
8EBUS
Σ
T(o 1 (sp), (() R(8 (sp), (s) s)), + va"((s')
(22)
S'EUS,,
Σ S'EUS
T(O (8p), a, (s | (s)), + V
VV
(23)
ū- Σ (sp), a, 8)
?
EsSUs
U- = M (8-))
(24)
7
Since f E [L we know that U-ビ S. We additionally know, by definition of optimal action, and for any E A, Q4' (8-) 2 Q² (5-1 a). Therefore Q.3"' (sp, at) Q (sp, a) for all a + at.
7
QED
A.1.2. LEMMA 0
Lemma O (s)= fss a) EeesT
())) + VV S (8) S. In other words, if the value of nt in M' reduces to the above form, then it is equivalent to the value of the policy in M for all benign states s
T(s, a.
a
This is labeled Lemma O as it is useful result which will be used in both Lemma and Lemma 2, but isn't 't a key result for this paper. It should be noted that the derivation is identical to one seen in (Rathbun et al., 2024), though here we are generalizing and replicating the result so it can be referenced with confidence in Lemma 1 and Lemma 2
Proof. Here we will prove that the difference between each value function O for all benign states, thus making them equal.
13

---

Adversarial Inception Backdoor Attacks against Reinforcement Learning
In other words: Vs S, D2 = VV (8) ViA (s) 0:
D3 = Σ T(S,a) a)
Σ a (') 5') + vV2" (s')
aEd
KEES
(25)
Σ (8,a)
Σ
T(8, a, 8 8 () 7V
aEA
S'ES
Σ T(S,
Σ
T(s, [)(s,,, v (s')
aEA
s'ES
(26)
Σ T(s s (Rs,,a + (
S'ES
Σ T(ss a)
Σ T(s, ([R()s,,,ss)))))) - -Va" MM (s))
aEd
s'E
(27)
[R(s, a, s') + +
ΣET T(s a)l ET(s, T(s, s sVV (5') - (V"
(28)
AEA
s'E8
In this form the problem gets a little cumbersome to handle, thus we will convert to an equivalent matrix form, allowing us to utilize properties of linear algebra. Such transformations are common in literature analyzing Markov chains in closed form (Stroock, 2013).
Let: D E IS such that D, = VA (8) (s)
(29)
Let: P e [s(ss such that Pa,s' Σ T(s a(s,
(30)
We know that is Markovian matrix by definition every row P, represents a probability vector over next states s' given initial state therefore each row sums value of 1. Given this, one property of Markovian matrices we can leverage that:
- aD=
(31)
In other words, the largest eigenvalue of a valid Markovian matrix P is (Stroock, 2013). Using our above definitions we can rewrite Equation (28) as:
= P(xD) 3
(32)
PD
(33)
Let's now assume, for the purpose of contradiction, that D + Since 7 E [0, this implies P has eigenvalue larger than 1. However, P is Markovian matrix and thus cannot have an eigenvalue greater than 1. Thus D must be true.
QED
A.1.3. LEMMA
Lemma V. (A (e) VM' M xf is optimal.
VM VM Therefore the value of T in M' is equal to its value in
Proof. In this proof we will expand the definition of VM (s) and show that it reduces to form equivalent V (s).
14

---

Adversarial Inception Backdoor Attacks against Reinforcement Learning
VM n
(8)
Σ t (s, Σ
"'s,,,,', (' +N +v (s')
(34)
aEA
S'SSUS
Σ x (s, B)
vV (s))
aEA
s'ES
(35)
a', Σ a ( 7 (s)]]
SE
s')[R(s,, Σ (s, a)s, - (1) [)(ssa)))) (s)]
aEA
s'ES
(36)
+ 3
Σ
a(s,
8 Br V M (s)]
Sp
From here, for the sake of space and clarity, we will choose to focus on simplifying the following piece of the summation:
R(s, a, (5')) + y Bna - - r( VV (s')
(37)
= a + + -r )] +7
Σ (s') (s',
(38)
d'EA
Since x" is optimal, and using the results of Theorem 1, we know that πT ( aat without loss of generality, we simplify to + 7- orr U-F when the target action is or isn't ' taken respectively. Thus can derive the following:
and T('', a) for + at, Again,
M' (s', "3, ,at ,x')) R(s, 00 (5)()) + + +7(0 (0 (s',,")) a²)+ +
0 + Ba' Ba'r br T(d, + 20
(39)
(40)
7
R(s,,,, (s)) + 7 222
(41)
R(ss,,, a, 8- (5)) + 7V (0 (s'))
(42)
Here we use the shorthand a" arg max 1Q (8- (s'), a')] Since the policy is already optimal, the optimal action chosen by does impact the policy's value the policy would have chosen a" in d='(s") without the inclusion of g. We
can additionally complete this last step by the definition of the bellman equation as x r"(6,, at) 1. Next we can plug this derivation back into the main equation and simplify further.
Σ T" (8, ((( -3) T(s,,,,')R + +VV (5)
(43)
aEA
6'ES
+ 3
T(s, a, 8 (8 a 5 ( + + (0 (s'))]
(44)
SEPP
From here, similar to (Rathbun et al., 2024), we note that the second summation is over s E Sp, yet the term is always inverted with 8-1, Since ó is bijective we can therefore convert the summation to one over s' E S:
Σ (s, B) "TTTT, a(ss,a,''(Rss,a, + (s')
YES
(45)
+ 8
T(s, a, (aa,,,')(R(ss,,5 + + (s'Il
#ES
Σ>' (s,a) T(ss,as)))(ss.. s') + vV
(46)
a
Therefore, by Lemma 0 we have proven the desired result.
QED
15

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
A. 1.1.4. LEMMA
Lemma 2 VA (s) 2 (s) S, E 11. Therefore, the value of any policy a in the adversarial MDP M' is greater than equal to its value in the benign MDP M for all benign states E S.
Proof. Lemma we proved that for an optimal policy x² M', its value in benign states is maintained between the adversarial MDP M'' and the benign MDP M.
Here we will prove that, in general, the value of any policy π E II given benign state 8 E 5 in M' is greater than equal to its value in M. We will achieve this by first showing that, without action manipulation, the value of the policy is maintained between M' and M. We will refer to this as the "base case". Following this we will show that ó induces a policy improvement over æ in M', proving our desired result. We will begin by defining a modified version of Φ:
Φ1 (Sp. a, =
(47)
In other words since (5pp a, π) for all trigger states, actions, and policies no action manipulation occurs. For the sake of convenience we will notate the value of a policy under this modified Φ, V-M (s]z) and the action value as Q. (I))
Base Case VV. (s|T)- = VzM Vs E S, x E II. Thus if no action manipulation occurs, then the value of any policy does not change between M and M' in beingn states.
Due to the nature of this proof, many of the steps are nearly identical to the proof given for Lemma 1, with some minor notational differences (using æ instead of x*). Thus we will provide an abridged version of the proof, with citations the relevant steps from Lemma when relevant. Thus we quickly derive an intermediate result similar to 36:
M (s|1)= = Σ a( Σ
(' + vVal (5'12)
(48)
RESUS,
Σ T(, - 3)
Σ
T(s.a, + VV7 (8(T))
dEA
SES
(49)
B
Σ
T(s a, 5 (
Enawn r, )] (s(Ii))
S'ES,
We will again focus our attention on the innermost term of the summation using the shorthand r' R(s a', m):
R(s, 5 (s') + Er,a- - r(a,r, V 2 (5(IT)
(50)
= a ( B,, Blaaa + Σ
m a') (6. (3)
(51)
o'EA
= a(s,, + - r(,,,,
+7 Σ 7(s 10
m))l) + -(a,t' -r'l
(52)
a'EA
= M(s,a, (6)) + Bar (0,r,))] Σ
7(X (ld,a')r(a,,r")
aEA
(53)
Σ a(s''''Q M' (s'),
d'EA
= R(s, a (5)) + VM
M' (8-1 (s))
(54)
From here, plugging this piece back into our equation for V.M' and using similar steps to our derivation for Equation 46 we once again arrive at a equation similar to that of M (s):
VM
Ts,
Σ s'ES
T(s, s')[R(s, + VV (s')
(55)
aEA
Thus by Lemma O we have proven the desired result.
16

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
Modeling & policy improvement
In the "base case" we showed that the value of any policy in benign states in M' is equal to its value in M if no action manipulation occurs. Here we will show that one can model the utilization of ø policy improvement over without action manipulation. In order to prove this result we must merely show the following:
Vala (s|T)= - D(s) Bawn 12 (s, 6(8 a, a))) Vs E SUS,
(56)
First we will show that this inequality holds for all poisoned states Sp E Sp
((sp)= = Σ
T( a) 2 (s, Φ(sp.(pp.p a, T)1
(57)
aEA
(8p, at 1Q (5 +
Σ
(12)))))
(58)
aEAlat
= (1
(59)
Σ
TT a0 (6 7(a,,, rr
aEAlat
(5 (6 (6)))
T(pp ( 110
Σ T(p 1) (1)
(60)
(Sps 1Q (5 (sp), Q.a (8- (sp), a [T] + V.M (spl7)
(61)
Here again the short hand a' arg max 1Q (8 (s'), a' [T] Thus by the definition of at we know the following:
(sp, 12
M (8 (sp), a [1) - MM (o (sp), at a 2
(62)
Therefore, for all 8p E Sp we know that ((sp) 2 V-M (T) Next we must show that this holds for benign states. This
much easier to show as no action manipulation occurs:
D(s) Σ ((ss.)QQ (s, as,
(63)
Σ ass.)) aEA
(s,al1) (s
V.M (sl1)
(64)
Therefore V-M (s < D(s) for all benign states Thus we have proven that the policy induced by o in M' results in policy improvement over any policy æ E II. Therefore, using the results of the base case, we know:
M (s) 2 V. (s|1) = M (s) V
(65)
Thus our desired result has been proven.
QED
A.1.5. THEOREM 2
Theorem V. (s) 2 VM (s) Vs S (s) (s) S. II. Therefore, is optimal in M' for all benign states E S if and only n² is optimal in M.
Proof. Here we will prove the above theorem by proving the forward and backward versions of the bi-conditional. After proving Lemma and this result becomes fairly straight forward.
Forward Direction:
17

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
Proof. Let n² be optimal policy in M'. For the purpose of contradiction assume x is not optimal in M.
It follows that 3 n' E 11, E S such that V
(s)
V (s).
From here, using Lemma 1 and 2, we know V. in M
V
V (8)
V (s), this contradicts the fact that x²" is optimal
QED
Backward Direction:
Proof. Let n" be an optimal policy in M.
It follows that's V n' II, E S the following is true V.M
V (8)
V ( VM (5).
Therefore x² must be optimal in M' for all benign states, thus we have proven the desired result.
QED
Thus by our forward and backward proof we have proven Theorem 2.
QED
A.2. More Experimental Details and Hyper Parameters
In this section we give further details on the hyper parameters and setups we used for our experimental results. In Table 5 we summarize each environment we studied, their properties, and the learning parameters we used in each experiment. Parameters not mentioned in the table are simply default values chosen in the cleanrl (Huang et al., 2022) implementation of PPO.
Training Environment Details
Favironment
Task Type
Observations
Time Sleps
Learning Rale
Environment Id.
Qert
Atari Game
Image
ISM
0.00025
Frranss-Famsssiip-v
Frogger
Atari Game
Image
10M
0.00025
PooggerNooramnskiiie
Pacman
Atari Game
Image
40M
0.00025
PacmanNoFrameskippy
Breakout
Atari Game
Image
15M
0.00025
BreakoutNo BreakoutNoFrmmskki-ve
Highway Merge
Self Driving
Image
100k
0.00025
merge-vU
Safety Car
Rohoties
ProproProei
3M
0.00025
SafetyCarGoal 1-v0
CAGE-2
Cyber Defense
One-Hot
5M
0.0005
cage
Table Further details for each environment tested in this work. All action spaces were discrete in some form, though for Safety Car discretized versions of its continuous action space was used. The "Environment Id." column refers to the environment Id used when generating each environment through the gymnasium interface (Brockman al., 2016).
Across all our image based domains we utilized a 6x6, checkerboard pattern of Is and Os in the top left corner of the image as a trigger. For Safety Car and CAGE-2 we simply append a boolean value to the end of the agent's 's observation which we set to poisoned states, or otherwise. This is also summarized in Table These environments in particular have relatively low dimension and dense state spaces, making the most natural trigger unclear. Thus we implemented this boolean indicator approach for poisoned states make it it completely clear that there are no collisions between benign and poisoned states in the state space maintaining our assumption that S and Sp disjoint. Through this design we can better isolate and compare the poisoning strategies of each attack without additional mitigating factors caused by the design of the trigger.
We additionally chose values for Stow and Shigh to balance attack success and episodic return. At values of B higher than Brigh one or more attacks would suffer in terms of episodic return, while at values of 3 lower than Blow attack success across all three methods would begin to drop significantly. 33 values were chosen on a per-environment basis using the parameters chosen by (Rathbun et al., 2024) as starting point.
1. TROJDRL AND SLEEPERNETS PARAMETERS
Across all environments we chose hyper parameters for TrojDRL and SleeperNets which maximize the amount by which
each attack perturbs the agent's 's reward. This guarantees that each attack takes full advantage of the range [L, U] provided to it, giving no additional advantage to Q-Inetpt in terms of reward perturbation. In particular, for TrojDRL we set its reward perturbation constant, c, to 100; and for SleeperNets we set its reward perturbation factor to the max value a and its
18

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
base reward perturbation to 1. For SleeperNets e is set to value of 1 as a alone results in perturbations far outside of [T., U] in all environments without clipping.
A.2.2. Q-INCEPT ATTACK PARAMETERS
For the Q-Incept attack there are a few parameters the adversary has to choose in regards to the Q-function approximator Q. These parameters are borrowed directly from DQN as the attack derives from a direct DQN implementation on the agent's benign environment interactions. In Table we summarize the two relevant parameters we varied across environments, Steps per Update and Start Poisoning Threshold. Steps per Update represents the number of benign environment steps that would occur between each DQN update of Q On Highway Merge much lower value was needed here as the adversary has little time to learn the agent's Q-fuction. In contrast, for Q*Bert, the number of steps per update was very high as the attack was very successful with little DQN optimization. The "Start Poisoning Threshold" represents the portion of benign timesteps the PPO agent would train for before the adversary would begin poisoning. This parameter is intended to allow the adversary's DQN approximation to begin to converge before they begin poisoning. Otherwise the adversary's 's Q would be effectively random when they start poisoning. Both parameters were chosen to balance attack performance and computational cost. All other DQN parameters not mentioned in this section set to the default values provided in cleanrl's implementation of DQN.
Environment Attack Parameters
Environment
Steps per Update
Start Poisoning Threshold
Trigger
@bbert
50
6.7%
6X6 Checkerboard
Frogger
50
6.7%
6X6 Checkerboard
Paeman
50
6.7%
6X6 Checkerboard
Breakout
50
6.7%
6X6 Checkerboard
Highway Merge
2
10%
6X6 Checkerboard
Safety Car
4
4.0%
Boolean Indicator
CAGE-2
4
4.0%
Boolean Indicator
Table Comparison of Q-Incept hyper parameters used across the different enviromments. Here Steps per Update represents the number environment steps per DQN update for Q, and Start Poisoning Threshold represents the portion of PPO training that needs to finish before the adversary would begin poisonining.
A.3. Further Discussion
In this section we provide further discussion on design choices made in this paper which were unable to fit in the main body.
1. MOTIVATION FOR BASELINES
As mentioned in Section we compare our Q-Incett attack against SleeperNets and TrojDRL as they represent the current state of the art for ubounded reward poisoning and forced action manipulation attacks respectively. For SleeperNets there are no other existing, ubounded reward poisoning attacks, so this decision is fairly clear. For TrojDRL there are other attacks which utilize static reward poisoning and forced action manipulation, however most only apply to specific application domains like competitive, multi-agent RL (Wang et al., 2021) partially-observabl settings utilizing recurrent neural networks (Yang et al., 2019; Yu et al., 2022).
The only other, somewhat comparable attack is BadRL (Cui et al., 2023) which builds upon TrojDRL by optimizing the adversary's 's trigger pattern to achieve greater attack success. Trigger optimization is effective but orthogonal to the goals of this work as can be generically applied to any attack. Furthermore, (Rathbun et al., 2024) showed that BadRL without this trigger fine-tuning often performs worse than TrojDRL, likely since it uses methods poison the most important and thus hardest to poison states in the MDP. Taking all of this into consideration we decided to omit BadRL from our empirical study, Therefore TrojDRL is the best baseline to use when comparing against static reward poisoning attacks using forced action manipulation.
A.3.2. MOTIVATION FOR ENVIRONMENTS
In this paper we study 4 environments Bert Frogger, Safety Car, CAGE-2, and Highway Merge. In the TrojDRL paper the authors focused their empirical studies towards Atari game tasks in the gym API (Brockman et al., 2016). We think it
19

---

Adversarial Inception Backdoor Attacks against Reinforeement Learning
useful include some of these environments like Q*Bert, as they are standard baselines for RL in general, however we believe it is critical to extend this study further domains when studying the potential impacts of adversarial attacks. This belief'ss supported by the findings of (Rathbun et al., 2024) who showed that TrojDRL which consistently attains near 100% ASR on Atari environments without bounded reward poisoning constraints often fails to achieve high ASR when tested on non-Atari environments.
Thus, to extend our study beyond the confines of Atari, we chose three other environments within the gymnasium API, allowing our code to work seamlessly between environments. We first chose Highway Merge since it seemed to be the most difficult environment for attacks to poison based upon the results of SleeperNets. Next we chose CAGE-2 as it not only represented& a safety and security-critical domain, being an application of RL to cyber-network defense, but also because it uses non-image observations. Lastly we selected Safety Car, also from the environments studied in SleeperNets, as it represents a simulation of real-world, robotic applications of RL and, similar to CAGE-2, uses non non-image observations.
A.3.3. COMMENTS ON COMPUTE OVERHEAD
Training the Q-network used in Q- Q-Incept does cause some computational overhead which may be detectable, but fortunately it isn't 't too extreme. We ran tests on desktop machine (2x RTX 4090, Threadripper 7980x) and found that SleeperNets, TrojDRL, and Q-Incept run at 1038, 987, and 730 simulation steps per second respectively against Atari Q*bert. However, our Q-network training runs in series with our PPO training, it's likely that significant performance increases can be found for Q-Incept by training in parallel.
A.4. Further Experimental Results and Analysis
A.4.1. Q-INCEPT AGAINST DQN
In Table 7 we present some preliminary results of Q-Incept against DQN. The results show that Q-Incept works well against both policy (PPO) and off policy (DQN) methods. A better choice of metric Fo may be necessary to maximize performance against DQN, however. For this experiment we slightly alter the methodology of Q-Incept. Instead of poisoning data after each episode and storing it in the agent's 's replay buffer, we instead poison data points after they's 're sampled from the replay buffer. The reason for this is that the agent's 's policy along with the Q-Network of Q-Incept change throughout training, therefore datapoints that were poisoned early in training may become less effective as training progresses. In the case of PPO this wasn't an issue, since trajectories are only used once and then discarded. For DQN, however, trajectories stored over longer time horizon during training, so we need account for this an attacker. Equivalently the attacker could keep track of all the data points has poisoned and properly update them over time.
Beta
ASR
StDev(ASR)
BR
StDev(BR)
0% (No Poisoning)
N/a
N/a
13,724
974
0.5% 1.0%
92.6%
4.1%
14,092
1,127
88.5%
3.4%
13,574
910
Table 7. Performance of Q-Incept against DQN agents training on Atari Q*tber.. Results are averaged over 5 runs.
A.4.2. Q-ICCEPT AT DIFFERENT POISONING RATES
In this section we evaluate the Attack Success Rate performance of Q-Incept at lower 3 values along with the stability of the agent's Benign Return at higher / values. First, in Table 8 we evaluate Q-Incept on Atari Qbbert at B values lower than those presented in the main body of the paper. We can see that even at a poisoning rate as low as 3 0.03% Q-Incept still able to achieve near 100% ASR. This is in line with the performance of methods like SleeperNets which presented similar results at 3 0.03% while operating under unconstrained rewards. B = 0.03% seems to be the minimum necessary poisoning rate for QnIectpt on @bbr however, as the attack success rate drops off dramatically at 3 0.01%.
In Table 9 we compare Incept to the baselines on the CAGE-2 environment to study their stability at higher 3 values. We see that QnIect is the only method that improves in ASR as 3 increases, and is also the most stable in terms of BR- never falling below -50.9%% In contrast, SleeperNets and TrojDRL are both highly inconsistent in terms of BR - falling to 57.82 and 64.411, respectively - while also failing to achieve an ASR above 6%, even at B = 2%.
20

---

Adversarial Inception Backdoor Attacks against Reinforcement Learning
Beta
ASR
StDev(ASR)
BR
StDev(BR)
0.3%
100%
0%
18,381
882
0.1%
100%
0%
17,749
1,380
0.05%
100%
0%
17,937
1,304
0.03%
98.3%
2.9%
16,573
873
0.01% %
21.1%
6.2%
16,374
2,088
Table 8. Performance of Q-Incept lower poisoning against PPO agent training Q"bert.
B
0.5 %
1 %
1.5%
2 %
Metric
ASR
ASR
ASR
ASR
Q-Incept
30.06 %
15.01% %
93.21% %
15.13%
100 %
0.00%
98.62 %
2.4
SleeperNets
0.00%
0.00%
0.06%
0.12%
0.54%
0.89%
1.86%
3.21 %
TrojDRL
0.00%
0.00%
5.64%
7.73%
2.24%
3.74%
5.11 %
4.99%
Metric
BR
Br
0
BR
0
BR
Q-Incept
-48.64
16.68
-45,49
8.04
-50,98
7.37
-49.41
7.50
SleeperNets
-57.82
8.38
-50.18
8.52
-41.44
11.59
-50.67
9.16
TrojDRL
-39.28
9.60
-53.11
8.71
-52.77
13.54
-64.41
10.63
Table Comparison of Q-Incept, SleeperNets, and TrojDRL on CAGE different values of 3.
A.4.3. Q-INCEPT WITH AN ORACLE Q-NETWORK
We noticed that Highway Merge was the only environment on which Q-Incept was unable attain an average ASR above 90%, leading to question if our Q-function based approach was incorrect if our online DQN approximation Q wasn't converging quickly enough. To test this devised until convergence can see that Incept improves greatly over Q-Incept, reaching an average ASR of 93.38%. This indicates that better Q-function approximations lead to better performance validating that both our chosen metric and attack approach scale properly with the accuracy of Q. Thus, adversaries capable of using Q-function estimations with faster convergence can expect greater attack success.
which uses oracle Q D-fuuction pre- trainee with DQN hypothetical, stronger attack by an adversary with direct access to the benign MDP. In Figure 7 we
1.0
0.6
Return
8
20k
10k
60k
100k
0k
20k
Time Steps
Time Steps
inccpp
SleeperNets
TrrDAL
Ne/Nooring
Oreck Bastt
Iooge
Figure Comparison between Q-Incept, Oracle-Incept, TrojDRL, SleeperNets on Highway Merge at 10%. can see that the Oracle-Incept shows significant improvements in ASR.
A.4.4. TRAINING PLOTS FOR REMAINING ENVIRONMENTS
Here in Figure 8, Figure 9, Figure 10 and Figure 11 we present the training curves for TrojDRL, Q-Incept, and SleeperNets on the Frogger, Pacman, Breakout and Safety Car environments respectively. Plots for Q*Bert are found in Figure 4, CAGE-2 can be found in Figure 3, and Highway Merge Figure 7. We can see that all attacks perform similarly over time in terms of episodic return on both environments, but Q-Incept is the only attack to reach 100% ASR on average in both environments doing so very quickly.
21

---

Adversarial Inception Backdoor Attacks against Reinforcement Learning
1.0
0.8
0.6
Return
ASR
200
0.4
100
0.2
0.0
0M
2M
4M
6M
BM
10M
0M
2M
4M
6M
BM
10M
Time Steps
Time Steps
Q-incept
TrojRR
No hebooinng
Q-hncepe
ecperhess
VojORL
Figure Performance of TrojDRL., Q-Incept, and SleeperNets on Frogger at 8 = 0.3%
1.0
800
0.8
600
ASR
0.6
Return
0.4
400
0.2
200
0.0
0M
5M
10M
15M
20M
25M
30M
35M
40M
0M
5M
10M
15M
20M
25M
30M
35M
40M
Time Steps
Time Steps
Q-nncpt
Faterning
Q-tccep
Guepertute
TajORi
Figure Performance TrojDRL, Q-Incept, and SleeperNets on Pacman at 0.3%
A.4.5. HOW OPTEN DOES Q-INCEPT ALTER THE AGENT'S S ACTIONS?
Here we explore how often Q-Incept chosen actions in the agent's 's replay memory D. We measure this ratio as the number of actions changed divided by the total number of timesteps the attack has poisoned. In Figure and Figure 13 we see that the attack generally balances its action poisoning over time, altering actions on roughly 50% of the time steps poisons. In the of CAGE-2 this does altering 87% of actions by the end. To this indicates that the difference in values between good and bad actions was much larger in CAGE-2 than in other environments, and furthermore that the agent was highly likely to choose these actions over others as training progressed. Since our proposed metric F& weighs time steps by the relative value of the action taken over all possible values, it makes sense that this would result a high action manipulation ratio on CAGE-2.
hold however, the adversary starts by altering around 50% of actions, but ends up
22

---

Adversarial Inception Backdoor Attacks against Reinforcement Learning
1.0
600
500
0.8
400
0.6
Return
ASR
0.4
2M
4M
GM
BM
10M
12M
14M
0M
2M
4M
6M
10M
12M
14M
Steps
Time Steps
Q-incept
SleeperNets
TrojDRL
No Putsoring
Q-hccop
Sesperhets
VojORL
Figure 10. Performance of TrojDRL, Q-Incept, and SleeperNets on Breakout at 3 = 0.3%
1.0
15
0.8
10
ASR
0.6
Return
0.4
1M
2M
3M
4M
5M
0M
2M
3M
4M
SM
Steps
Time Steps
G-incept
SleeperNets
TrojORL
No Retsoning
O-hcepr
Seeperkues
PojOR
Figure I1. Performance of TrojDRL, Q-Incept, and SleeperNets on the Safety Car enviromment at 8 = 0.1%
23

---

Adversarial Inception Backdoor Attacks against Reinforcement Learning
1.0
1.0
Fraction of Actions Changed
Changed
0.8
0.8
0.6
0.4
0.4
0.0
0.0
0M
Time Steps
Time Steps
Q-incept
G-incept
1.0
Fraction of Actions Changed
Changed
0.8
0.8
0.4
0.4
0.2
0.0
0.0
0M
6M Time Steps G-incept
Time Steps
Quincept
Figure 12. Ratio of actions changed across poisoned states in our four Atari environments (from top left bottom right) Qubert, Frogger, Pacman, and Breakout. Values are measured as ratio actions changed to number poisoned timesteps.
1.0
Fraction of Actions Changed
Actions Changed
0.8
0.8
0.4
0.4
0.2
0.0
Time Steps
Time Steps G-incept
Q-incept
1.0
Fraction of Actions Changed
0.8
0.4
0.2
0.0
0M
1M
3M
Time Steps Quincept
Figure 13, Ratio of actions changed across poisoned states in our three Ataritari environments (frop top left to bottom) CAGE, Highway Merge, Safety Car. Values measured as ratio actions changed the total number poisoned timesteps.
24


Reviews:
[{"review_id": "rbhTGZlAyi", "reviewer": "Reviewer_4J4B", "summary": "The paper proposes a new method, Q-Incept, for backdoor poisoning attacks.\nPrevious work assumes the ability to arbitrarily change the reward within some “poisoned” states in the dataset. The authors rightly point out this is not necessarily realistic, as they arbitrarily manipulate the magnitude of the reward, and this might rarely be possible in practice.\nThey add an additional constraint to solve this, so that the adversarial attacker cannot induce rewards that are larger or smaller than those given by the original MDP.\nThey additionally demonstrate that previous work causes rewards to grow arbitrarily large, and when the additional constraint is added, these previous methods fail to consistently induce the desired “poisoned” behaviour.\nThe authors provide a theoretical justification for why previous methods fail under constrained rewards and prove that Q-Incept achieves high attack success rate across multiple environments while maintaining the agent’s performance in benign tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback and questions, we look forward to further discussion with you. Based upon our response we kindly ask you to consider increasing your assessment of our paper. \n\n**\"In related work, some alternative poisoning methods are mentioned, I would also include [Lu et. al 2023]\"**\n\nWe agree that this is an interesting and relevant paper so we will include it in our related work section. \n\n**\"The title is too generic\"**\n\nThis is a fair critique. We have further thoughts about a different title \"Reward Poisoning is Not Enough: Adversarial Inception for Constrained Universal Backdoor Attacks against Reinforcement Learning\", but we are not certain that OpenReview allows us to change it. If you have other ideas we are very open to your suggestions.\n\n**\"The plots should not be weights and biases screenshots\"**\n\nWe understand your concern. For the camera ready version of the paper we will download the raw data from weights and biases and plot everything using a graphics library (e.g., seaborn) instead. \n\n**\"How does Q-Incept compare to SleeperNets and TrojDRL in beta values? (% of poisoned states needed to achieve success) … and how does Q-Incept performance drop with lower betas?\"**\n\nIn the TrojDRL paper they evaluate on Atari environments with $\\beta = 0.025\\%$, while in the SleeperNets paper they evaluate on a wider range of environments using poisoning rates from $\\beta=0.005\\%$ to $0.5\\%$. Both attacks are evaluated with unbounded rewards. Our poisoning rates are comparable, being in the range $\\beta = 0.05\\%$ to $\\beta = 1.0\\%$, with the only outlier being Highway Merge which seems particularly resilient to backdoor attacks - likely due to its short episodes (15 time steps) and training time (100,000 time steps).\n\nWe have performed some additional experiments for the rebuttal on Q*bert where we evaluate Q-Incept at smaller poisoning rates from $0.05\\%$ to $0.01\\%$. We can see that even at a far lower $\\beta=0.03\\%$, Q-Incept is still able to achieve an ASR of 98.3%. This means we are able to replicate the results of SleeperNets with the exact same $\\beta$ they used, despite operating under constrained rewards. Once we go lower to $\\beta=0.01\\%$ the attack starts to fail, however, which is to be expected as the agent very rarely sees the trigger.\n\n|  Beta |  ASR  | StDev(ASR) |   BR   | StDev(BR) |\n|:-----:|:-----:|:----------:|:------:|:---------:|\n|  0.3% |  100% |     0%     | 18,381 |    882    |\n|  0.1% |  100% |     0%     | 17,749 |   1,380   |\n| 0.05% |  100% |     0%     | 17,937 |   1,304   |\n| 0.03% | 98.3% |    2.9%    | 16,573 |    873    |\n| 0.01% | 21.1% |    6.2%    | 16,374 |   2,088   |\n\nWe expect these findings to be replicable across the all Atari environments we evaluated as they seem to yield similar attack performances. It is similarly likely that lower poisoning rates for Q-Incept can be used on the Safety Car environment - though attack performance will drop eventually, of course.", "claims_and_evidence": "The evaluation spans a diverse set of RL environments, including Atari games, cyber network defense, and autonomous driving simulations. The results convincingly demonstrate that Q-Incept maintains high attack success rates (100% in multiple environments) while ensuring the agent still performs well on the underlying benign tasks. The authors also present ablation studies confirming the necessity of their inception-based action manipulation technique.\nThey also have experiments showing the high magnitude of rewards induced by previous methods, and show that previous methods fail when the reward magnitude is constrained.", "methods_and_evaluation_criteria": "Yes, they use two main metrics: Attack Success Rate (ASR) and Benign Return (BR). ASR measures the extent to which the adversary can induce the targeted behavior, while BR ensures that the poisoned agent still performs well on its intended task, making it less likely to be detected. This makes sense given the context.", "theoretical_claims": "The theoretical analysis of why previous methods fail when rewards are constrained is clear and intuitive.", "experimental_designs_or_analyses": "I thought the experiments were straightforward and made sense given the analysed setting. I did not check the code or setups in detail (hyperparameters etc.)", "relation_to_broader_scientific_literature": "Q-Incept builds upon prior poisoning methods like TrojDRL and SleeperNets, addressing their fundamental reliance on arbitrarily large reward perturbations. Instead of reward-based manipulation, Q-Incept carefully modifies high-return states. This aligns with concerns in adversarial RL on detectability and stealth.", "essential_references_not_discussed": "In related work, some alternative poisoning methods are mentioned, I would also include \n\nLu, C., Willi, T., Letcher, A., Foerster, J.N.. (2023). Adversarial Cheap Talk. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:22917-22941 Available from https://proceedings.mlr.press/v202/lu23h.html.\n\nWhich discusses poisoning by appending extra information to agent observations.", "other_strengths_and_weaknesses": "Strengths:\n+The proposed method is based on the insight of manipulating high reward actions, rather than inducing arbitrarily high magnitude rewards.\n+ Extensive evaluation\n\n\nWeaknesses:\n- I think the title is too generic\n- The plots should not be Weights and Biases screenshots", "other_comments_or_suggestions": "NA", "questions_for_authors": "1) How does Q-Incept compare to SleeperNets and TrojDRL in beta values? (% of poisoned states needed to achieve success)\n2) How does Q-Incept performance drop with lower betas?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s3K1zrVDl6", "reviewer": "Reviewer_mydv", "summary": "This paper proposed a novel backdoor attack framework called Q-Incept to attack the deep reinforcement learning training process by changing the state, reward, and action stored in the replay buffer. The proposed method designed new transition and reward functions for the MDP under the backdoor attack. The experiment shows strong attack performance compared with other backdoor RL methods.\n\n## Update after rebuttal \nThe rebuttal has adequately addressed most of my concerns, and I am still leaning toward accepting this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and questions, we look forward to further discussion.\n\n**“Why the backdoored BR performance can outperform No Poisoning BR scores?”**\n\nFor Q-Incept, our theoretical results show that the optimal policy for benign states in $M’$ (the poisoned MDP) is the same as in $M$ (the benign MDP). Therefore we should expect the agent to learn a strong policy even under Q-Incept poisoning. This is supported by our empirical results, as you have pointed out. We believe any increase in BR score after poisoning in some environments is merely due to the variance of PPO. We would expect these two scores to get closer as we average over more runs. You can also see that the BR scores of Q-Incept and of No Poisoning are within a standard deviation of each other in these environments.\n\n**“Also, why a higher poisoning rate has a higher BR score than a lower rate in certain environments?”**\n\nSimilar to the last question, we believe this discrepancy is due to the general variance of PPO. Our other leading theory is that it has something to do with the generalization capabilities of the agent’s network. At lower poisoning rates the agent does not see the trigger as often, yet when they do, they get a (relatively) large signal to take action $a^+$. This may lead the agent to explore the action $a^+$ more often in benign states as it has not seen the trigger often enough to be certain that no other states require $a^+$ to be taken. This is just a theory, however, so we leave deeper explorations to future work. These results do lend extra evidence towards the stability and stealthiness of Q-Incept, however, as BR scores are not damaged as $\\beta$ increases.\n\n**“In the image setting, the trigger is set as 6 x 6 checkerboard, but it is not mentioned where and how the trigger is injected. Could the authors provide some poisoned images?”**\n\nCertainly! Please see our anonymous github (https://anonymous.4open.science/r/Q-Incept-ICML-2387/mydv.md) which contains images of the 6x6 checkerboard trigger on the Q*bert environment. The other reviewers also asked us for additional figures, so feel free to look through those as well.\n\nThe trigger we show for Q*bert is the same for all other image domains. In short, the checkerboard is inserted at the top left corner of the image by setting every other pixel to a value of 0 or 255, respectively. In the case of models that use framestacks as input, we insert the trigger into every frame (treating each framestack as a distinct state). We will be sure to clarify this in our amended appendix. \n\n**“Also, I guess the checkerboard trigger might be obvious for human eyes to detect. Could the authors justify the reason for choosing this trigger? Could a more visibly stealthy trigger be used in your approach?”**\n\nMore stealthy triggers can absolutely be used with Q-Incept. Our method is completely agnostic to the trigger function $\\delta$, only requiring that the trigger does not naturally occur in the environment during training. In real world environments, you can imagine the trigger is a special sticker or object the adversary can place in the environment - which is much stealthier. $\\delta$ also does not need to be a deterministic function, allowing the adversary to implement other novel trigger techniques.\n\nOur motivation for using the checkerboard trigger in our experiments is that it is visually distinct from normal states - meaning the CNN based agent should be able to easily distinguish poisoned and benign states. This means the success or failure of each method is dependent on their reward and action poisoning strategies alone.\n\nTo give further proof to our claim of trigger agnosticism, we have performed additional experiments for reviewer cgPq, using a stealthier trigger. You can see the trigger in Figure 2 (right) at the following anonymous github (https://anonymous.4open.science/r/Q-Incept-ICML-2387/cqPg.md). This trigger looks like a graphical glitch, which wasn’t uncommon with old atari systems and TVs, so having it appear for single frames at a time would appear normal. Despite this, Q-Incept is still equally effective - resulting in a 100\\% ASR and a BR score of 17618.\n\n**“Missing left parentheses in the table below equation 9 … I think the third row and fourth row of the table below equation 9 …”**\n\nThanks for pointing this out, we’ve fixed these errors in our overleaf.\n\n**“Could the author compare training overhead between your approach and baseline approaches?“**\n\nSure thing. Training the Q-network used in Q-Incept causes some computational overhead, but fortunately it isn’t too extreme. We ran tessts on a desktop machine (2x RTX 4090, Threadripper 7980x) and found that SleeperNets, TrojDRL, and Q-Incept run at 1038, 987, and 730 simulation steps per second respectively against Atari Q*bert. Note that our Q-network training runs in series with our PPO training, so it’s likely that significant performance increases can be found for Q-Incept by training in parallel.", "claims_and_evidence": "1. The proposed algorithm is well-motivated and supported by the proof. The author also provides theoretical guarantees of Q-Incept, which strengthens their contributions.", "methods_and_evaluation_criteria": "1. The authors conduct experiments on various environments, including Atari games, CAGE, Highway Merge, and Safety Car. They report two key metrics, which are ASR and BR to capture the attack performance and stealthiness. The results show that Q-Incept achieve the best ASR across all scenarios.", "theoretical_claims": "1. I skimmed the proofs in the Appendix and they make sense to me.", "experimental_designs_or_analyses": "1. The experimental designs are reasonable. However, I do have some questions about the BR results. Why the backdoored BR performance can outperform No Poisoning BR scores? Also, why a higher $\\beta$ poisoning rate has a higher BR score than a lower $\\beta$ rate in certain environments? Could the authors provide some insights on this?", "relation_to_broader_scientific_literature": "This work is related to robust reinforcement learning, trustworthy AI, and AI safety in general.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. I do have some concerns about the triggers. In the image setting, the trigger is set as $6\\times6$ checkerboard, but it is not mentioned where and how the trigger is injected. Could the authors provide some poisoned images? Also, I guess the $6\\times6$ checkerboard trigger might be obvious for human eyes to detect. Could the authors justify the reason for choosing this trigger? Could a more visibly stealthy trigger be used in your approach?", "other_comments_or_suggestions": "1. Missing left parentheses in the table below equation 9\n2. I think the third row and fourth row of the table below equation 9 should be exchanged according to the transition function the authors provide.", "questions_for_authors": "1. Please refer to the Experimental Designs section.\n2. Please refer to the Other Strengths And Weaknesses section.\n3. Could the author compare training overhead between your approach and baseline approaches?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RN4qqiiKh4", "reviewer": "Reviewer_cgPq", "summary": "This paper introduces a new backdoor attack against deep reinforcement learning agents, specifically addressing the constraint that an attacker cannot arbitrarily modify the reward function to some extremely large value. The key insight is to selectively poison high-return time steps in the agent’s training data, manipulating actions to induce adversary-desired behaviors. The authors formalize the attack within a theoretical framework, providing guarantees on both the attack's success and stealthiness. They further propose Q-Incept, a training algorithm designed to poison DRL agents effectively. Experimental results across various Gym simulation environments demonstrate the attack’s efficacy.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and questions!\n\n**“Existing works like TrojDRL do not introduce arbitrarily large reward values either”**\n\nIt is true that TrojDRL perturbs the agent’s reward by a fixed value $\\pm c$, but this $c$ may need to be arbitrarily large for attack success. Let’s return to our example in Figure 2. As $\\gamma$ approaches $1$, $Q(\\text(start) a)$ approaches infinity. Therefore, in order for the attack to be successful, i.e. $Q(\\delta(\\text(start)) a^+)  > Q(\\delta(\\text(start)) a)$, the attacker’s reward poisoning constant $c$ must also approach infinity. \n\n**“I suggest that the authors clarify their contribution… why does the attacker need to adhere to the reward bound at all?“**\n\nThe adversary should adhere to our proposed reward bounds because they otherwise become trivially detectable, irrespective of an offline or online attack. As an example, imagine a simple, rule based defender $D$ that takes in a reward value $r$ and verifies $\\inf(R) \\leq r \\leq \\sup(R)$, for benign reward function $R$, otherwise labeling the reward as adversarial. This defense detects both unbounded SleeperNets and TrojDRL (with a sufficiently large $c$ hyper parameter) while having a 0% false positive rate. Furthermore, simply clipping rewards in data collected offline breaks SleeperNets and TrojDRL, but not Q-Incept.\n\nTherefore there are many realistic scenarios in which our reward bounds will be enforced. This is what led us to explore constrained reward attacks and subsequently design Q-Incept. We are happy to clarify further if you have any questions. We will also be sure to include this in an extended motivation in our updated manuscript.\n\n**“Could the authors clarify whether their approach fundamentally differs from TrojDRL in this regard?”**\n\nOf course! TrojDRL and Q-Incept’s action poisoning techniques are fundamentally different, as we aimed to capture in Section 4.1. In short, TrojDRL’s approach alters the agent’s *policy* at training time (see Equation 7), meaning the agent *chooses* **and** *transitions* with respect to $a^+$. In contrast, Q-Incept poisons the *perceived transition function* of the MDP, meaning they *choose action* $a^+$ but *transition with respect to an optimal action*. In both theory and in practice, the action manipulation of TrojDRL does not improve attack performance (see Table 1), while Q-Incept has theoretical guarantees of attack success (see Table 2 and Section 4.3).\n\nThe core insight of Q-Incept is that manipulating how the agent perceives the transition function is sufficient for achieving both attack success and stealth. Under constrained rewards simply forcing exploration the target action, as TrojDRL does, is not enough. Understanding this distinction is very important to understanding our contributions, so please feel free to ask more questions.\n\n**“Missing evaluations against defenses… such as provable defenses [1]”**\n\nDefenses like [1] achieve “universal” results by targeting the attack’s trigger directly, aiming to “sanitize” the state and remove the trigger. This comes at a cost, however, as they are subsequently *trigger dependent*. Furthermore, most backdoor attacks, including Q-Incept, are trigger agnostic, meaning they can use any trigger pattern to achieve attack success. Therefore, evading a defense like [1] simply requires devising an evasive trigger. To prove this, we perform additional evaluations against [1] and are able to successfully break the defense, resulting in **100% ASR after state sanitization**. Captioned figures detailing our method for breaking [1] are provided at the following anonymous github (https://anonymous.4open.science/r/Q-Incept-ICML-2387/cqPg.md). We will be sure to include these results in a revised version of the paper.\n\n**“Limited RL algorithm evaluation…”**\n\nThis is a fair criticism, therefore we have performed additional evaluations of Q-Incept against DQN, proving the attack is successful against both on and off-policy DRL algorithms. Captioned results can be found at our anonymous github.\n\n**“Poisoned return is not reported…”**\n\nWe do not report poisoned return as it is not the metric we are aiming to optimize. At test time the adversary can have a multitude of objectives they aim to solve by exploiting the backdoor, many of which will not be in direct opposition to the agent’s return (e.g. biasing a warehouse bot to handle some products more often than others). Therefore we report ASR alone as it is a more atomic metric that captures the level of control afforded to the adversary by the backdoor.\n\nThat being said, we can include poisoned returns in the appendix. For instance, on Q*bert we attain a poisoned return of 0 with Q-Incept - the minimum possible score.\n\n**“Applicability to offline RL…”**\n\nQ-incept is certainly and directly applicable to offline RL, as you point out. Due to limited time we can’t perform these experiments right now, but we look forward to future works extending Q-Incept to offline RL.", "claims_and_evidence": "- **Motivation and Justification:**\nThe motivation of the paper is not well-supported. While I understand that reward clipping is common in widely used simulators, existing works like TrojDRL do not introduce arbitrarily large reward values either—they simply modify the reward to the maximum allowable value within the clipping range (i.e., +1). I suggest that the authors clarify their contribution by emphasizing that their primary focus is on the poisoning strategy, rather than on the reward modification constraint. Unlike prior works that apply to poison randomly, their approach leverages an additional Q-value network to guide the poisoning process, which is a more structured and strategic method. Furthermore, if the training trajectories are perturbed offline—meaning they are modified after interaction with the environment—then the altered reward is never subjected to the environment’s clipping constraints. In that case, why does the attacker need to adhere to the reward bound at all? Since the attacker has full access to the training process, any reward clipping applied during training can simply be ignored. Given this flexibility, what is the fundamental limitation of existing arbitrary reward poisoning methods?\n\n\n- **Clarification on Action Manipulation:**\nStarting from Line 155, the authors state that “our adversary changes actions after the episode has finished, meaning these perturbed actions will never actually occur in the environment.” However, in TrojDRL, while actions are manipulated during the agent’s training process, the true state transition is still determined by the original (unmodified) action. As a result, neither the environment nor the agent perceives the modified action during training. This modified action only affects the updates of the policy network. Given this similarity, could the authors clarify whether their approach fundamentally differs from TrojDRL in this regard?", "methods_and_evaluation_criteria": "The selected simulator and datasets make sense for the problem that the paper studied, although more extensive experiments are needed to further support its attack effectiveness, please see the \"Experiment Designs\" for more details.", "theoretical_claims": "The theoretical claims look fine to me.", "experimental_designs_or_analyses": "- Missing evaluations against defenses: The experiments do not assess the attack’s effectiveness against relevant defenses such as provable defenses [1], BIRD [2], or simply fine-tuning, although there are brief discussions in the paper. Including some preliminary results would provide a more comprehensive understanding of the attack’s robustness and stealthiness.\n- Limited RL algorithm evaluation: Only PPO-trained agents are evaluated, while results on other widely-used RL algorithms like DQN and A2C are missing.\n- Poisoned return is not reported: The paper does not report the poisoned return, which is an important metric for understanding how effectively the attack degrades the agent’s performance. Including this would provide a clearer assessment of the attack’s impact.\n- Applicability to offline RL: The inception attack modifies stored trajectories offline, which suggests that it could also be applied to offline RL-trained agents. It would be beneficial to discuss whether it would be possible to apply the attack to an offline RL setup.\n\n[1] Bharti, et al., Provable defense against backdoor policies in reinforcement learning, NeurIPS 2022.\n\n[2] Chen et al., BIRD: Generalizable Backdoor Detection and Removal for Deep Reinforcement Learning, NeurIPS 2023.", "relation_to_broader_scientific_literature": "The paper contributes to the broader literature on adversarial attacks inRL by introducing an inception backdoor attack that manipulates stored trajectories offline rather than injecting poisoned samples during online interactions. This approach builds upon prior works such as TrojDRL, which introduces backdoors in RL through reward manipulation but differs by focusing on action perturbation at high-return time steps rather than arbitrary reward poisoning.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths:**\n- The attack is formally defined within a theoretical framework, providing guarantees on both attack success and stealthiness, which strengthens its conceptual foundation.\n- Unlike prior works that randomly poison training data, the proposed attack leverages an extra Q-value network to strategically select high-return time steps for poisoning, making it more targeted and effective.\n\n**Weaknesses:**\n- More clarification on the paper's motivation is needed.\n- The paper does not evaluate the attack against existing defenses or compare its effectiveness across multiple RL training algorithms (e.g., DQN, A2C).", "other_comments_or_suggestions": "No.", "questions_for_authors": "Please see the \"Claims\" and \"Experiment designs\" part for questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Both the problem domain and solution approach were appreciated by all the reviewers.  One of the key concerns was, and remains, motivation for the particular restriction on the adversarial reward manipulation, while also allowing the adversary to modify transition dynamics.  I looked at the paper as well, and find the motivation weak because it is focused on simulated environments.  In such settings, the very motivation behind attacks is undermined, since nothing in practice is actuated.  I know that these provide a key first step to pre-train robots that are subsequently fine-tuned/adapted to real settings, but the broader point is that the paper should work a bit more on a compelling motivation.  In addition to spelling out why the setting and threat model are credible, I also suggest adding illustrative examples that demonstrate their claim that reward modifications used in SOTA (troDRL, in particular) can become arbitrarily large.
Decision: Accept (poster)