Universal Black-Box Targeted Reward Poisoning Attack Against Online Deep Reinforcement Learning
Authors: Yinglun Xu
Gagandeep Singh
Venue: Accepted by TMLR ()

Abstract: This work proposes the first universal black-box targeted attack against online reinforcement learning through reward poisoning during training time. Our attack is universally efficient against any efficient learning algorithm training in general RL environments and requires limited attack budgets and computational resources. We generalize a common feature of the efficient learning algorithms and assume that such algorithms would mostly take the optimal actions or actions close to them during training.  We quantify the efficiency of an attack and propose an attack framework where it is feasible to evaluate the efficiency of any attack instance in the framework based on the assumption. Finally, we find an instance in the framework that requires a minimal per-step perturbation, which we call `adaptive target attack.' We theoretically analyze and prove a lower bound for the attack efficiency of our attack in the general RL setting. Empirically, on a diverse set of popular DRL environments learned by state-of-the-art DRL algorithms, we verify that our attack efficiently leads the learning agent to various target policies with limited budgets.

Published in Transactions on Machine Learning Research (11/2025)
Universal Black-Box Targeted Reward Poisoning Attack Against Online Deep Reinforcement Learning
Yinglun Xu
yinglan6dérlinois.eet
Deportment of Computer Science University of Ilinois at Urbana- Champaign
Gagandeep Singh
ggnds@hilinoii.. edu
Department of Computer Science University of Blinois at Urbona- Champoign
Reviewed on OpenReview: https: openrevie. net/ pett/foum?? id= MX00DDu8
Abstract
This work proposes the first universal black-box targeted atfack against online reinforcement learning through reward poisoning during training time. The attack is universal in that it can effectively mislead any efficient learning algorithms to follow targeted behaviors specified by the attacker under general assumptions. More specifically, we generalize a common feature of the efficient. learning algorithms and assume that such algorithms would mostly take the optimal actions or actions close to them during training. We quantify the efficiency of an attack and propose an attack framework where is feasible to evaluate the efficiency of any attack instance in the framework based on the assumption. Finally, we find an instance in the framework that requires minimal per-step perturbation budget, which call 'adaptive target attack.' We theoretically analyze and prove a lower bound for the attack efficiency of our attack in the general RL setting. Empirically, on a diverse set of popular DRL environments learned by state-of-the-art DRL algorithms, we verify that our attack efficiently leads the learning agent to varions target policies with limited budgets.
1 Introduction
Online deep reinforcement learning (DRL) algorithms have great potential to be applied industrial applications such as robotics [Christiano et al. 2017), recommendation systems (Afsar et al.) (021), and natural language processing Ouyang et al. (002).. In such applications, the reward signals during training are usually feedback from human users, which raises the threat of training-time reward poisoning attacks. There may exist adversarial human users who deliberately provide malicious rewards. A malicious atlack goal can be manipulating the DRL agent to follow certain behaviors during training with malicious properties, such as unsafe or harmful behavior, to benefit. the attacker. Such behavior can still lead to high performance according to the agent's 's metric, making the attack harder to detect. For example, consider the scenario of training restaurant's rocommendation system. A restaurant. wants mislead the recommendation system to recommend itself more likely. For this purpose, the restaurant can deliberately provide false negative feedback for its competitors and positive feedback for itself. Then, during training, vulnerable recommendation system might mostly recommend the malicious restaurant, which is unfair and harms the benefits of the restaurants, users, and the system. We show the framework of targeted poisoning attacks against online RL in Figure Prior studies on targeted allacks mainly focus on simpler labular settings (Rakhsha et al. 2020 Xu et al., 2021 Zhang et al.) 2020 Banihashem et al. 2022). The results only show the vulnerability of current tabular RI. algorithms, and the attacks usually only work in the whito-box setting against specific learning algorithms, which may not be practical. In this work, we aim to expose the practical vulnerabilities of state-of-the-an DRL algorithms by finding targeted attack that works efficiently in realistic setting.

---

Published in Transactions on Machine Learning Research (11/2025)
Universal black-box targeted attack in DRL and the challenges: Here, we discuss the practical constraints on an attack.
1. Universal black box: The learning agent can directly defend against whitc-box attacks by hiding its learning algorithm and training parameters from the public. Given that many different efficient RL algorithms have been developed (van Hasselt et al. [2016; Wang et al. 2016 Haarnoja et al.) 2018 Dankwa & Zheng 2019), the agent can arbitrarily choose an efficient algorithm. Therefore, realistic attacker should be oblivious to the learning agent (black-box attack), and its attack strategy should work against any efficient learning algorithms an agent might use (universal attack).
2. Real-time with limited computational power: In online RL, the data collection during the training process happens in real-time. Therefore, a realistic attack should also be able to inject perturbations during training in real-time. In addition, more practical to run the attack with limited computational resources, as rich computational resources may not always be available to malicious atlacker. We do not consider the computational cost of constructing the target policy, as it is an input to the targeted attack problem.
We aim at developing an attack that can mislead the agent to take larget actions with strict constraints and budgets. We formally define the attack constraints and budgets in Section 2 Particularly, we consider optimizing the attack efficiency that is mcasured by how much attack budgets are required to successfully mislead the learning agent. We formally define the allack efficiency in Section 3 Developing an efficient. attack is challenging. First, universal attack should work for different algorithms an agent might use, and it is hard to determine exactly what these algorithms are for an agent. Second, in the black-box setting, the attacker cannot exactly predict an agent's 's behavior without detailed knowledge of the training setup. To achieve high efficiency, the attack should optimize the budget. needs during the training process while working with limited computational resources. This becomes especially difficult without predicting the agent's 's behavior under attack.
To the best of our knowledge, our attack is the first universal black-box targeted attack applicable to the DRL sctting. The key insight for making the attack universal is a general assumption based elficient DRL algorithm can learn the optimal solution or a
Our contributions:
a feature of elficient. learning algorithms: similar one through strategic exploration in the environment. Under the assumption, we characterize the set of efficient algorithms a rational learning agent might consider. In the universal black box setting, it is hard to approximate the behavior of all efficient learning algorithms under an arbitrary attack. We develop an attack framework called 'efficient adversarial reward engineering' such that the behavior of an arbitrary efficient learning algorithm becomes predictable under any attack under the framework. In this framework, the attacker perturbs the feedback signals during training enviromment desired by the attacker. Finally, we find the instance in the framework that requires a minimal attack budget on per-step perturbation and call it the 'adaptive target attack.' We provide detailed theoretical analysis of the attack efficiency in the general RL setting and conduct complementary experiments in the DRL setting. The empirical results verify that our attack can successfully mislead the agent to a target policy with limited budgets and computational resources. Our attack works for general RL problems with both discrete and continuous action spaces and applies to diverse learning algorithms. We experimentally verify the efficiency of our attack on various popular DRL environments for continuous and classical control problems, including HalfCheetah, Hopper, Walker, and Acrobot, and various state-of-the-arr DRL algorithms, including SAC (Haarnoja et al.) 2018), TD3 (Dankwa & Zheng] 2019). DDPG (Lillicrap et al. 2015), and DQN (van Hasselt et al.) 2016). We consider various largel policies to show that our altack works in a wide range of practical scenarios. Among all the attack settings we consider, our results show that our black-box attack can always successfully make an agent take actions close to target actions during training with very limited budgets. As the target actions are also the optimal actions in the adversarial RL environment creatod by the attacker, the results also indirectly verify the assumption made the efficient learning algorithms.
if they are generated by an adversarial RL
2

---

Published in Transactions on Machine Learning Research (11/2025)
Ideal behavior
Environment
Take an action
Agent
00
Training
True feedback
Environment
Agent
Target behavior
Take an action
Training
Corrupted feedback
perturb
Adversary
Figure 1: The framework online reinforcement learning under adversarial poisoning attacks. The learning agent can interact with the environment by taking an action each timestep. Idcally, when there is no atlack, the agent can observe the true feedback from the enviromment and then learn ideal behavior desired by the agent. However, in practice, adversaries exist in the environment that inject corruption into the feedback from the environment. The agent can only train on the corrupted dataset and may learn target behavior desired by the attacker instead.
2 Background
Online reinforcement learning: We consider a standard and general online reinforcement learning setting with a discounted infinite horizon (Sutton & Barto 2018). An environment characterized by an MDP M = AS, P, R. Mo). Here S the state space, is the action space, P S transition function where A(S) is the space of state distribution, S A R the reward function, 7 is the discount factor, and μ40 the distribution of the initial states. The environment initialized at a state sampled from μo. The agent takes action a, and the environment evolves to the next state s' ~ P(s, and returns an instant reward signal r ~ R(s, a). At each time step, if there is no poisoning attack, the agent will receive the actual observation tuple (s, s', A policy π : S + A(A) for the environment is a mapping from a state to a probability distribution over the action space. Here ((AA) represents the space of probability distributions over the action space. Let n² be the discounted state distribution visited by policy μº (s) = Exo=po.a" ~T((ss= Ei=oy" 1[s s] (Agarwal et al. 2019). The performance of a policy then defined JR(m)= Eewg"",nn(e) anvn(s) R(s,a). The optimal policy is the policy with the highest. ( performance. Since deterministic optimal policy always exists (Sutton & Barto 2018), we use x to denote the deterministic optimal policy and n" (s) to represent the optimal action at state 8.
A(S) is the state
Threat model: We consider an adversary that can inject perturbations into the reward during agent training. Let T' be the total number of training steps. At each time step t, the current state is st, and the action taken by the agent is a!. The true feedback of the environment next state gf+1 and reward nd, Before the agent observes the true feedback (s',a', "' s ,r"), the adversary observes it and injects perturbation A' the reward. Then, the agent receives (s', a², st+1, 2²). We denote A {A ...., AT) the attacker's perturbation strategy. It is random variable depending the observations of the interaction between the agent and the environment.
Constraints on the attack: We want the attack to work in a black box setting with minimal information about the learner and environment. We formally describe the constraints realistic attack below.
3

---

Published in Transactions on Machine Learning Research (11/2025)
1. Oblivious to the learning agent & Universal attack: The attacker is oblivious to the learning algorithms and sctups of the agent. The attack should universally work on different algorithms the agents use.
2. Oblivious to the environment: The attacker has no knowledge of the transition function P and reward function R of the environment. It is aware of the state space S and action space A.
Budgets required by the attack: The efficiency of the attack depends on the budgets required by the attacker. In this work, we consider the following attack budgets.
1. The total amount of the reward perturbation attacker applies.
|||||| - Σi= | during the training process the
2. The maximal amount of reward perturbation || |00 arg max | at each step. This ensures the perturbation at each step is bounded.
The goal of the attack: In general, the attacker wants the agent to follow certain target behaviors during training that are characterized by a deterministic target policy æt (Sun et al. 2020 Zhang et al. 20200. The action given by the target policy a state is the target behavior that the attacker wants the agent to take at the state. Denote a' as the action taken by the agent under the attack and (s') the target action. Let d(.,-): A x A [0, be a distance measure between two actions. Taking the continuous action space as an example, the distance can be defined based on L2 distance as d(aj, (2) = ||a a2l12/L, L, where L = max ||æ-~ is the maximum 2-norm distance between any two actions. We adopt this definition for the experiments in Section 4 Note that the analysis and the method in this work apply to general distance measures. Denote d' dia', mt (s')) as the distance between the agent's action and the target action timestep and d = {d² ...., d²) as the distances during the training process. Formally, the goal of the allack is set follows:
The attack aims to induce minimal difference between the actions selected by the agent and the target actions during training |d] = Ei=(dd) (d') with respect to a p-norm.
We want find an attack that works under the constraints and achieves the goal with limited budget. In the next section, we formulate the attacker's goal as an optimization problem and develop efficient. attack methods.
3 Adaptive Target Attack and Analysis
3.1 Reward poisoning attack as an optimization problem
Let T be the total number training steps, L be the class of learning algorithms from which the agent will choose its learning algorithm. Denote the state and selected action time during training under the attack A', which is random variable determined by the
st and a'. The perturbation at time is denoted attack strategy depending on the previous observation {s at 1 A,A--111 during training. The attack strategy is represented by the perturbations A = [ (TA....,AT) A7).
We focus on universal black box attacks. The attack should be efficient against any algorithm Alg E L that the agent might use without knowing exactly which algorithı is used. Ideally, an efficient attack should minimize: 1. the distance e between actions taken by the agent with any Alg E and the target actions 2. the total perturbation C, and 3. the maximum perturbation at each step B. However, such an optimal attack might not exist attack could induce smaller value of with more budget B and C. Therefore, we define the most efficient attack for a fixed perturbation budget B, C in Eq 1
min E, s.t. VAlg E C, ?
(|lg.M[|ddB3 Sc,
(1)
EAi A|]1] S C, || B
4

---

Published in Transactions on Machine Learning Research (11/2025)
To complete the definition of the attack problem, we need to define C, the class of algorithms the agent might use. First we argue that letting & include all possible algorithms makes the problem trivial and is meaningless for the attacker. In this case, there always exists an algorithm that takes specific actions regardless of the environment and the attack to maximize c. For example, the target policy is always taking the action in the center of the action space (suppose the action space a multi-dimensional box), and the learning algorithm is to always take actions on the corners of the action space. Such an algorithm dominates the value of and makes it the same for all attacks. As a result, any attack that satisfies the budget constraint will be an optimal solution, which is meaningless. It is also pointless for the attacker to consider attacking an inefficient and not data-driven algorithm that any rational agent will not use. Therefore, it is more meaningful for the attacker to consider agents using 'efficient' algorithms. To quantify which kind of algorithms are 'efficient,' we generalize a common feature shared by existing efficient DRL hods in the definition below.
Definition 3.1. (6,-efficient learning algorithm) Let the total training steps be and the optimal policy of the environment be x². A 6p-efficient learning algorithm guarantees that, in expectation, it will take the actions close to the optimal actions for most of the time during training: E د d(a', TT (s'))² S 6p, where 6, > 0 is a constant, and d(a',, (s")) is the distance between the agent's 's action s² and the optimal action of the environment Tt' T**) at time t.
In general, an efficient learning algorithm should be capable of identifying majority of optimal actions actions close to them within T steps in expectation. During training, it will take actions that are close to the optimal actions most of the time. We argue that existing elficient DRL algorit are likely to be op-efficient with a small value of 6p. First, the goal of designing learning algorithms is to guarantee that the learning algorithm can find the optimal actions for a large set of environments. Second, a reinforcement learning algorithm needs to balance the exploration-explottttonn trade-off, making it necessary to explore the sub-optimal actions strategically. In practice, most algorithms select the empirically optimal action with the highest probability, such as e-greedy exploration (van Hasselt et al. 2016). So, the agent will identify the optimal actions actions close to them as empirically best actions and correspondingly select those actions for most of the timesteps. This is indirectly supported by our empirical observations in Section 4 For the rest of this section, we consider L to be the set of 6p-efficient learning algorithms in Eq [
The oplimality of a solution to the optimization problem in Eq. 1 can measure the efficiency of an attack. More specifically, Eq. [ implies a definition for the attack efficiency as in Definition 3.2
Definition 3.2. ((c, B) efficient attack) Given a learning algorithms famility L and a target policy ri, an attack is (e, C, B) efficient if its perturbation A satisfies the following conditions:
VAlg E C, Eag. [|dd]] Se
(2)
E Ellg. |II [1] S C, M, || || S B,
where d = (d(a',=t(ss)) (s are the distances between the larget actions and the actions taken by the agent under the attack.
nt
An attack is more efficient if it is (c, C, B)-efficient with a lower value of e for the same value of B and C. By [ finding the optimal attack is equivalent to finding an (e, C, -efffiicient attack with the minimal value of e given B and C. For the rest of the paper, we focus on Eq. 2 in the theoretical sections as it is more convenient to analyze. Next, we show that finding the efficiency of an arbitrary attack strategy is hard.
First, the attacker needs to predict the influence of the attack on all ôy-efficiint algorithms from L. There can be infinitely many efffeiiccn algorithms, and the attacker does not know exactly what they are. Second, predicting the actions taken by specific unknown learning algorithm Alg is still hard. The attacker needs to know the learning algorithm and the environment dynamics to predict the actions the agent takes and the state transitions in the future. However, it can only observe the state transitions and reward signals during training in the black box setting. It is hard to approximate the environment accurately with such limited observations. With no further limitation on the learning algorithm, the learning stratogy at different. timesteps can be independent, making it impossible to predict the agent's future strategy through its behavior in history.
5

---

Published in Transactions on Machine Learning Research (11/2025)
For targeted attack that aims at misleading the agent to follow specific behavior, it is unreliable to adopt attack strategy if the agent's 8 behavior under the attack is unpredictable. Therefore, it is crucial to identify the attacks that predictably influence the learning agent's behavior. Next, we show a class of attacks that satisfy this property and find an efficient instance from the class with minimal per-step corruption.
3.2 Efficient adversarial MDP attack framework
Here, we build an attack framework called 'efficient adversarial MDP attack' so that the efficiency of any attack instances in the framework can be estimated. We formally define the framework in Definition 3.3
Definition 3.3. (Efficient Adversarial MDP attack) An adversarial MDP attack specifies an adversarial environment M = (S, P, R.), which has the same state space, action space, and state transition function as the true environment but different reward function R. The perturbation at time is A' R(s' a')-R(s' a'), where st is the state visited by the agent and a' is the action taken by the agent at time t. Under the attack, the agent will receive reward R(s*,a)) at time t. Therefore, the agent is actually trained in the adversarial M under the attack. In addition, an elficient adversarial MDP attack characterized by M should satisly the conditions below for perturbation bound A O.
1. The Larget policy x² is the optimal policy under the adversarial enviromment M.
2. The differences between the adversarial and true reward functions are small for the actions whose distance to the target actions is small at any state. Given value of A E Rt, for all state E S, [R(s, a) R(s,a)) < A d(a, x (s))..
We denote FM as the class of all efficient adversarial MDP attacks. We denote instance of FM as M(A), where A is its perturbation bound. In Theorem 3.4 show guarantee on the attack efficiency of an efficient adversarial MDP attack M(A) E EM.
Theorem 3.4. An efficient adversarial MDP allack M(A) EM is VVB,CC BBB.))effciien by Eq. with op: ==A, and C' 8p A.
The proof for Theorem 3.4 can be found in Appendix A Here, we explain how the framework is constructed and how its efficiency evaluated. The framework is built to work closely with the 6y-efficient learning assumption. We note that the assumption describes the behavior of the agent learning from static environment. Therefore, this assumption applics if the attacker makes the agent train as if in a malicious enviromment M obtained by perturbing the true environment M. In this case, for any algorithm Alg E C, it is always true that E dia', (s'))²] I 6p. where ** is the adversarial optimal policy on the adversarial environment. If the adversarial optimal policy is exactly the target policy # = πÝ, then the attack guarantees that c = EI (d(a', #" (s))))) S 5p. To make the agent train as if in M, the perturbation at time T should satisfy A' = R(s'saa)= - R(s", ((s".a"). To limit the requirement budget, the attacker can limit the perturbation around the adversarial optimal policy, as the agent will mostly take those actions. The idea we adopt is to connect the perturbation applied to an action to its distance from the target action. More specifically, we require (R(s,a) - R(s, a)l & A d(a, x (s))P. . In this case, for any algorithm Alg E L, we have B= = A directly by the attack definition. The total perturbation C applicd by the attack is connectod to the total distance between the agent's 's actions and larget actions: E, |R(s", a²) R(s", aa)l < A ||d|| < 6, A.
The guarantees above suggest that attack could be more efficient with smaller value of A. However, the value of A must be significant enough to make the target policy xt optimal in the adversarial environment M. Next, we aim to find this threshold and the corresponding efficient adversarial MDP attack with the minimal value of A.
3.3 Efficient adversarial MDP attack with minimal A: adaptive target attack
This section shows an instance of the efficient adversarial MDP attack framework that requires the minimal value of A. First, define the 'adaptive targeted attack' in Definition 3.5 below.
6

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm Adaptive Target Attack Framework
1: Input: target policy x!
2: Params: distance measure d, maximal per-step perturbation A, polynomial factor g
for 1 1= 1, do
Observe environment state s², agent's action a', and reward signal yd
Perturb reward signal r² <- p² A dlaa, xt - (s)))8
end for
Definition 3.5. (Adaptive target attack) The adaptive target attack AT(A, q) as described in Algorithm is parameterized by the maximal reward perturbation A E IR+, and a positive value E IR+. Given a largel policy æt and a distance measure d, the reward perturbation at round t by the adaptive target attack satisfies A' A d(a', ( (s')), where s is the state visited by the agent and is the action taken by the agent at time t.
The adaptive target attack penalizes the agent for taking an action different from the target action. The attack essentially constructs an adversarial environment with adversarial reward function satisfying R(s,a) R(s,a) A d(a, =))) The computational resource required by the attack is the memory to store the target policy π², and the computational power to calculate the target action given by æt and its distance to the action taken by the learning agent.
Formally, in Theorem 3.6 we show that the efficient adversarial MDP attacks with the minimal value of perturbation bound A' must be an adaptive target attack with A A and q = p. The proof can be found the Appendix A Intuitively, in the adversarial environment, the adaptive target attack with q = p penalizes all non-targeted policics more than any other instances of efficient adversarial MDP attack with the same value of A, making it easier lo highlight the larget policy.
Theorem 3.6. Let EM be the class of efficient adversarial MDP allacks and M(A) be an instance with perturbation bound A. Denote A MiJM( A as the minimal value of A for an efficient adversarial MDP attack. The value of A is
max JR(T) -TR(m))) T:D(n,x')>0
A
D(T,,T)
where JT( is the performance of policy π on the actual reward model R of the true environment, and D(11, x2) == Egnger d (s)) +2()) represents the distance between two polices. In addition, the adaptive larget allack AT(A" p) is the adversarial MDP attack with minimall perturbation bound M(A)).
In summary, Theorem 3.4 guarantees lower bound on the allack elficiency of an adversarial MDP atlack in Definition 3.3 The lower bound decreases as the perturbation bound A of an adversarial MDP attack decreases, suggesting that efficient adversarial MDP attacks should have low values of A. We further find that the adversarial MDP attack with the minimal value of A is an adaptive target attack Definition 3.5 with q = p. Therefore, we believe an adaptive target attack efficient. with sufficient value of A and g = p. However, note that Theorem 3.4 provides bound on the attack efficiency instead of the exact value. So, an adversarial MDP attack with lower value of A does not necessarily have to be less efficient in practice. Also, under the (e, B, C) definition for attack efficiency, the efficiencies of any two attacks not directly comparable as there can be a tradc-off between and the budgets B, C. Therefore, the versions of adaptive target attacks with A might also be efficient in practice, and we empirically investigate this in the next section.
4 Experiments
Although the assumption that the algorithm used by the agent is an efficient learning algorithm in Definition 3.1 may not strictly hold in practice, we demonstrate this section that our adaptive target attack is effective in the universal learning scenarios we test. In the theoretical analysis Section 3 we study the general case where the attacker aims to minimize the total distances [d] between the agent's actions and the target.
7

---

Published in Transactions on Machine Learning Research (11/2025)
actions. For the simplicity of empírical evaluation, we focus on natural case where p 1, that is, the linear summation of the distances at each timestep ||]]]] = =i= d(aaa=dass')) (s')) where at is agent's 's action and x (s(s) is the target action. Accordingly, we set 1 for the adaptive target attack in most of our experiments, and we study the attack with different values of 9 in the ablation study. Note that our attack applies to general RL settings. The rest of this section mainly focuses on the standard continuous control problems with continuous action spaces. In Appendix C RL environments with discrete action spaces.
we show the attack setup and the empirical evaluations in the
4.1 Experiment Setup
Learning scenarios and target policies: Here, we focus on the standard continuous robotic control problems from Mujoco (Todorov et al. 2012), including HalfCheetah, Hopper, and Walker. We verify that our attack works efficiently against the environments with discrete action spaces in Appendix C For the learning algorithms, we consider state-of-the-art DRL algorithms, including DDPG (Lillicrap et al. 2015), TD3 (Dankwa & Zheng, 2019), and SAC (Haarnoja et al. 2018). The number of training steps is set as 6 x 10S to ensure that the algorithrs can collect high rewards on average during training with no corruption.
Our attack is designed to work for universal target policies. Therefore, we adopt the idea of the random, medium, and expert policies from (Fu et al. 2020) to represent policies with diverse performances. A random policy is randomly generated and is usually associated with low performance. An expert policy is a policy of the highest performance that efficient learning algorithm can learn. A medium policy has a performance in the middle of the above two. We construct such policies following the process in (Fu et al. 2020) Here, we mainly focus on the medium policy in the experiments and test all three types of policies in the ablation study. We believe that setting the target policy as a medium policy is relatively more realistic. A target policy usually has some behavior desired by the attacker but not desired by the agent, so its performance should be sub-optimal. However, if the performance of a target policy is comparable to a randomly generated policy, then the agent will find no improvement during training and may discard the trained policy in the end. Therefore, a medium policy is relatively more suitable as the target. In Appendix D we show the details of the behavior of the medium policy we chose target for the HalfCheetah enviroument. In general, show that the robot controlled by the medium policy exhibits unnatural behavior compared to that controlled by an expert policy. We further show that the agent learns the same strange behavior under the adaptive target attack with the medium policy as the target. In Appendix E we show another example to more intuitively highlight the influence of the attack on the learning agent. More specifically, we Lest. the case with a larget policy optimized on the opposite of the original goal of the environment. We show that under our attack with this specific target policy, the agent learns to achieve high performance the opposite of the actual environment learning goal, which is desired by the allacker but undesired by the agent.
Baseline Attacks: To the best of our knowledge, there is no existing baseline for black-box targeted online DRL attack. We adopt a statc-of-the-art black-box online DRL attack from Xu et al. (0023 to our sctting as a baseline, which we call the 'neighborhood attack.' The neighborhood attack is parameterized by A E R+ and r e RR+, and its corruption strategy is A A (dd(a (s')) 2 r). That is, it applics a fixed penalty when the agent takes an action far from the target action and no penalty otherwise. In addition, we consider random attack strategy that randomly decides corruption at each training step a naive baseline. More specifically, the random attack is parameterized by A E R- At each training step, the corruption it applies is uniformly at random drawn from i-a, A]. In Appendix
we provide an additional comparison to the white-box online DRL atlack 'va2c-p' from Sun el al. (0020) We discuss the difference between their al Lacks
and ours in Section 5
4.2 Efficiency of Attacks
First, we investigate the efficiency of altacks in different learning scenarios. The main goal is to verify that our adaptive target attack is universally effective against different efficient learning algorithms. For this purpose, we consider thrce environments: HalfCheotah (HC), Hopper (HP), and Walker (WK), learned by three algorithms: DDPG, TD3, and SAC. Here, we only consider the target policy to be a medium policy, as previously discussed. In the ablation study, we test other types of target policies. To ensure fair comparison,
8

---

Published in Transactions on Machine Learning Research (11/2025)
all the attacks share the same value of maximal per-step corruption B. That is, the corruption applied by the attacks at each step can not be greater than B. In this case, all the attacks set A = B to utilize the maximal per-step corruption budget fully. For the HalfCheetah environment, the maximal per-step corruption is set as B = 50; for the Hopper and Walker environments, the value is set as B = 20. In the ablation study, we test. the attack with smaller values of A. For the baseline neighborhood attack, to highlight its influence in mislcading the agent, we test the values of E [0.5,33 3.0] and show the results where the attack requires a limited budget on C' and achieves low values of e. To evaluate the efficiency of the attacks, we measure the average corruption C/T applied by the attack and the average distance e/T between the agent's action and the target action over the number of steps T. Each experiment is repeated 10 times, and we report the average values together with the sample standard deviation. For more intuitive comparisons of different methods, we also report the range of the values and perform Mann-Whilney U tests in Appendix B
The results are shown in Table [ For our adaptive target attack, in all learning scenarios, it can always make the agent closcly follow the target behavior (low value of c) with a limited budget required (low value of C).. Note that the standard deviation of the results under the attack is also small, implying that the influence of the attack on the agent is stable. This indirectly verifies our assumption of the efficient learning algorithms and efficiency guarantees on our attack. More importantly, the attack results are consistent for all three learning algorithms we test with, suggesting that our attack is universally efficient against different efficient learning algorithms. In comparison, the random attack can hardly make the agent follow the target behavior while requiring much more budget. The neighborhood attack can also make the agent take actions much closer to the target actions compared to the case of no attack and random attack, but the effect is much less than that of the adaptive target attack. While it also requires limited budget of C compared to the random attack, it generally requires similar or more budget than the adaptive attack average in most cases, indicating that the adaplive atlack is more efficient in general. The standard deviations for e/T and C/T' under the neighborhood attack are also large in some cases, indicating that the attack has a less stable effect compared to our attack. For a more direct and rigorous comparison, in Appendix B we provide the range of the values of all experiment runs in Table and perform Mann Whitney tests to directly compare the adaptive larget attack and the neighborhood attack.
Env-Alg
Clean (
Adapt (=)
Neigh (+)
Random (
Adapt (+)
Neigh (4)
Random (
IC-DDDP
(.50()
0.0900.00)
0.22(0.12)
(.00)
4.52(0.23)
10.12(8.28)
25.00(0.01)
IICTTD3
(0.07)
0.11().02)
0.28(0.14)
0.54(0.01)
5.72(0.88)
(3.73(9.95)
24.99(0.01)
HC-SAC
0.52(0.02)
0.11(..02)
0.18(0.08)
0.52(0.00)
(.61(0
6.68(7.38)
25.02(0.02)
HP-DDPG
0.50(
0.17(0.00)
0.35(0.19)
0.52(0.00)
3.37(0.08)
6.16(4.73)
(.99(0.01)
HP-TD3
0.45(0.03)
0.17(0.00)
0.34(0.04)
0.47(0.02)
3.37(0.06)
6.12(1.03)
10.00(0.01)
UP-SAC
0.43(0.01)
0.17(0.00)
0.26(0.01)
0.46(0.01)
3.33(0.05)
2.75(0.14)
10.00(0.01)
WK-DDPG
0.49(0.02)
0.15(0.02)
0.22(0.03)
0.51().04)
3.05(0.31)
(.22(
10.01(0.01)
WK-TD3
0.54(0.07)
0.16(0.01)
0.30(0.05)
0.46(0.06)
3.24(0.20)
6.67(2.31)
10.00(0.01)
WK-SAC
(0.05)
0.18(0.01)
0.26(0.03)
0.39(..03)
3.65(0.19)
4.87(1.51)
10.00(0.01)
Table 1: Performance of attacks in different learning scenarios. In the 'Dataset-Alg' column, 'HC' represents HalfCheetah, 'HP" represents Hopper, and 'WK' represents Walker. For other columns, 'Adapt' represents the adaptive target attack, 'Ncigh' represents the neighborhood attack, and 'Random' represents the random attack. The values in the brackets are the sample standard deviation. The bold values have the minimal value of e/T in each row.
4.3 Ablation Study
In this section, we focus on mcasuring the efficiency of the attacks according to Definition 3.2 in different setups of the attack and the learning scenario. In Appendix C we provide additional ablation studies on maximal per-step perturbation A. We also test the case of the attack in the RI. environments with discrete action spaces and the scenario where the attack have to stop applying corruption in the middle of the training.
Different types of target policies: Here, we verify that the adaptive target attacks remain efficient for the dataset collected by different types of policies. We take the HallCheetah environment. an example and
9

---

Published in Transactions on Machine Learning Research (11/2025)
test the attack with target policies being random, medium, and expert policies. We set A = 50 as in the main experiments. We measure the efficiency of the attack and show the results in Table 2 We observe that the adaptive target attack always successfully misleads the agent to follow the target behavior of all kinds of target policies. The results also show that the difficulties of mislcading the agent to different target behaviors differ. The random policy is the easiest larget as the altacker can make the agent take actions very close to the target actions (low value of c) while using the lowest corruption budget of C among all targets. In comparison, the expert policy is the most challenging.
Alg-Target DDPG-Rand
Clean (4)
Adapt (f)
Adapt (7)
0.51(0.01)
0.06(0.00)
2.76(0.01)
TD3-Rand
0.54(0.02)
0.06(0.00)
2.80(0.02)
SAC-Rand
0.43(0.01)
0.03(0.00)
1.66(0.02)
DDPG-Med
0.52(0.05)
0.09(0.01)
4.58(0..1)
TD3-Med
0.51(0.07)
0.10(0.02)
5.24(0.49)
SAC-Med
0.52(0.02)
0.12(0.01)
5.75(0.76)
DDPG-Exp
0.52(0.07)
0.15(0.02)
7.36(1.07)
TD3-Exp
0.45(0.09)
0.17(0.01)
8.45().66)
SAC-Exp
0.42(0.05)
0.14(0.04)
6.98(1.90)
Table 2: Performance of the adaptive target attack for different types of target policies.
Different values of q: Here, westudy the elficiency of adaptive target attack when setting the hyperparameter to different values. We choose the HalfCheetah environment as an example and test the attack with q = {0.5, (,,,,1,2,44 4}, while the other setups are the same as the experiments in Section 4.2 The results are shown in Table B We observe a trade-off that with a higher value of 9, the attack less less influence on the learning agent with higher value of e, but the required budget on C also decreases. We observe that when the value of q is no greater than decreasing the value of q has a marginal influence e, but the increment on C' is significant, which is undesired by the attacker. Similarly, when > 2, increasing the value of q has a limited influence on C, but the increment in e is significant. Therefore, in the general case, [1, intuitively achieves a good balance on the trade-off. If the corruption budget becomes a critical concern, then setting a high value of q = is also a reasonable choice, as the attack can still effectively mislead the agent to its target policy.
Alg-p
Clean (+)
Adapt (4)
Adapt (4)
DDPG-0.5
0.52(0.05)
0.10(0.01)
13.59(0.76)
TD3-0.5
0.51(0.07)
15.01(0.41) 0.12(0.01)
SAC-0.5
0.52(0.02)
0.12(0.01)
14.39(0.93)
DDPG-1
0.52(0.05)
0.09(0.01)
4.58(0.41)
TD3-1
0.51(0.07)
0.10(0.02)
5.24(0.49)
SAC-1
0.52(0.02)
0.12(0.01)
5.75(0.76)
DDPG-2
0.52(0.05)
0.11(0.01)
1.29(0.12)
TD3-2
0.51().07)
0.13(0.02)
1.22(0.11)
SAC-2
0.52(0.02)
0.12(0.01)
1.45(0.30)
DDPG-4
(.52()
0.20(0.02)
0.69(
TD3-4
0.51(0.07)
0.19(0.02)
0.49().07)
SAC-4
0.52(0.02)
0.22(0.01)
0.60(0.06)
Table 3: Performance of the adaptive target attack for different p-norm.
In this work, we do not focus on the agent's ability to detect abnormal data, which can be used to defend against data corruption. To the best of our knowledge, it remains unknown how to detoct abnormal data induced by a poisoning attack during training time in an online RL setting. In Appendix F we discuss anomaly detection at training time in RL in detail.
10

---

Published in Transactions on Machine Learning Research (11/2025)
5
Related Work
Data poisoning attack in simpler reinforcement learning Data poisoning attacks in reinforcement
learning have been considered in simpler bandit. cases (Jun et al.
2018
Garcelon et al. 2020 Liu & Shroll
2019
and tabular MDP cases (Rakhsha et al. 2020 Xu et al.
Banihashem et al., Liu & Lai 2021 Rakhsha et al. 2021 Li et al. 2024 Since the deep MDP case has a much more
2021 Zhang et al., 2020
2022
complicated environment to learn from, the attacks mentioned here cannot be applied to the deep MDP case. Banihashem et al. (2022); Rakhsha et al. (2020) adopt the same basic attack framework as our work. Besides the difference in the definition of the attack's budget, the main limitation in their works is that their methods are only defined on discrete state and action spaces and require full knowledge of the environment. The attack problems proposed in the works also impossible to solve in the deep MDP case due to the complexity of the environment.
Observation perturbation attack in deep reinforcement learning A line of work study observation perturbation attack and defense in DRL (Behzadan & Munir 2017a by Inkawhich et al. 2019 Liang et al. 2022; Liu et al. 2022 Qiaoben et al.) 2024). There are three main differences between these works and ours. First, observation perturbation atlacks perturb the state signal, while our atlack perturbs the reward signal. Second, observation perturbation attacks do not change the environment's dynamics but change the environment's signal observed by the agent. Our attack changes the dynamics of the environment. Third, the observation perturbation attack focuses on the intrinsic generalization vulnerabilities of the neural networks used by the learning agent. In contrast, our attack focuses on the vulnerabilities of the exploration stralegy of the learning agent.
Data poisoning attack in deep reinforcement learning Xu al. (2023) study untargeted reward poisoning attacks against DRL, Sun et al. (2020) is the only other work considering targeted attacks in DRL. There are three main limitations of this attack compared to ours and therefore less practical: (a) the attack is white-box that requires the knowledge of the learning algorithm, (b) the atlack only works for ou-policy learning algorithms, and (c) the attacker decides the perturbation after a batch of training steps. In addition, their attack problem is designed to minimize the cost of corruption per training batch rather than each training step. Our attack is designed to minimize the per-step corruption and total corruption, making our attack more general. For empirical evaluation, they only cover naive larget policies that are not representative of practical attack scenarios. In our work, we experiment with multiple non-trivial types of target policies.
6 Conclusion and Limitations
In this work, we propose the first black-box targeted reward poisoning attack against online DRL. Our attack is practical because (1) it does not have knowledge of the learning agent and environment dynamics, (2) it requires a limited budget for corruption over cach step and the whole training process, and (3) it requires limited computational resources. The limitations of our attack include: (1) it only studies reward poisoning attacks, (2) it only works for online reinforcement learning settings. We hope our study will motivate the development of DRL algorithms robust against data poisoning attacks in the future.
Acknowledgments
We sincerely thank the anonymous reviewers and action editor for their insightful comments and constructive suggestions. This work was supported by funding through NSF Grants No. CCF-2238079, CCF-2316233, CNS-2148583 and a Research Gift. from Amazon AGI Labs.
References
M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. arXie preprint orXiv:2101.0663 061.8686,, 2021.
Alckh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32:96, 2019.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Kiarash Banihashem, Adish Singla, Jiarui Gan, and Goran Radanovic. Admissible policy teaching through reward design. arXiv preprinl. ar vrXii:2201.021 2022.
Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. Tn Petra Perner (ed.), Machine Learning and Data Mining in Pattern Recognition 13th International Conference, MLDM 2017, New York, NY, USA, July 15-20, 2017, Proceedings, volume 10358 of Lecture Notes in Computer Science, pp. 262-275. Springer, 2017a. doi: 10.1007/9978-33319-44411
URL
Vahid Behzadan and Arslan Munir. Whatever does not kill deep reinforcement learning, makes it stronger. arXiv preprint Xrx:::712, 09344, 2017b.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiu:1606. 01500, 2016.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proc. Neural Information Processing Systems (NeurIPS), pp. 4299 4307, 2017.
Stephen Dankwa and Wenfeng Zheng. Twin-delayed ddpg: A deep reinforcement learning technique to model a continuous movement of an intelligent robot agent. Proceedings of the 3rd International Conference on Vision, Image and Signal Processing, pp. 1-5, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep clata-driven reinforcement learning. ar preprint Niw:2004407219 2020.
Evrard Garcelon, Baptiste Roziere, Laurent Meunier, Jean Tarbouriech, Olivier Teytaud, Alessandro Lazaric, and Matteo Pirotta. Adversarial attacks on linear contextual bandits. Advances in Neural Information Processing Systems, 33:14362 14373, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Tom Haider, Karsten Roscher, Felippe Schmoeller da Roza, and Stephan Günnemann. Out-of-distribution detection for reinforcement learning agents with probabilistic dynamics models. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 851-859, 2023.
Matthew Inkawhich, Yiran Chen, and Hai Li. Snooping attacks on deep reinforcement learning. arXiv preprint arXiv:1905. 11832, 2019.
Kwang-Sung Jun, Lihong Li, Yuzhe Ma, and Jerry Zhu. Adversarial attacks on stochastic bandits. Advances in neural information processing systems, 31, 2018
Jianhui Li, Bokang Zhang, and Junfeng Wu. Online poisoning attack against reinforcement learning under black-box environments. arXie preprint orXiv:2412.00797, 2024.
Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial training without
attacking: Worst-case-awar case- robust reinforcement learning. Advances in Neural Information Processing Systems, 55:22547-22561,, 2022.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Contimuous control with deep reinforcement learning. arXiv preprint Xrxiv: Xr0v 2015.
Fang Liu and Ness Shroff. Data poisoning attacks on stochastic bandits. In International Conference Machine Learning, pp. 4042-04 40200 PMLR, 2019.
Guanlin Liu and Lifeng Lai Provably efficient black-bor box action poisoning attacks against reinforcement learning. Advances in Neural Information Processing Systems, 34:12400-12410 2021.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. or preprinl ar 2rv: 14691, 2022.
Robert Müller, Steffen Illium, Thomy Phan, Tom Haider, and Claudia Linnhoff-Popi Popien. Towards anomaly detection in reinforcement. learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp. 1799-1803, 2022.
Linas Nasvytis, Kai Sandbrink, Jakob Foerster, Tim Franzmeyer, and Christian Schroeder de Witt. Rethinking out-of-distribution detection for reinforcement learning: Advancing methods for evaluation and detection. arXiv preprinl 2rXix:2404.000 04.07999, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feodback. Advances in Neural Information Processing Systems, 35:27730-2774 2022.
You Qiaoben, Chengyang Ying, Xinning Zhou, Hang Su, Jun Zhu,, and Bo Zhang. Understanding adversarial attacks on observations in deep reinforcement learning. Science China Information Sciences, 67(5):152104, 2024.
Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environ- ment poisoning: Training- time adversarial attacks against reinforcement learning, CoRR, abs/ Abss/000..J0 12909, 2020. URL https://arxiv arrg/bbs/2003. 12909
Amin Rakhsha, Xuezhou Zhang, Xiaojin Zhu, and Adish Singla. Reward poisoning in reinforcement learning: Attacks against unknown learners in unknown environments. arXiv preprint (xXv:2202.08492, 2021.
Yanchao Sun, Da Huo, and Furong Huang. Vulnerability-wwar poisoning mechanism for online rl with unknown dynamics. arXiv preprint arXiv:2009.001 00774, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for mosel-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5033. IEEE, 2012. doi: 10. 10.1109/TROS.200 201 20000062880000 2.6386109.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-carning. Tn Dale Schuurmans and Michael P. Wellman (eds.), Proceedings of the Thirtieth AAAT Conference Artificial Intelligence, February 12-17, 2016, Phoenir, Arizona, USA, pp. 2094-2100. AAAI Press, 2016.
URL http://ww:: www aaai. orgg//cs/iddexx..oooo/oo/oo/o wttp:/pp///AAAI111AAA1 aataa///.a/o//////////////////// pappr//ves//12900 228
Ziyu Wang, Tom Schaul, Mattco Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Ducling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995- 2003. PMLR, 2016.
Pawel Wawrzyúski. A cat-like robot real-time learning to run. In Adaptive and Natural Computing Algorithms: 9th International Conference, ICANNGA 2009, Kuopio, Finland, April 23-25, 2009, Revised Selected Popers 9, pp. 380 390. Springer, 2009.
Hang Xu, Rundong Wang, Lev Raizman, and Zinovi Rabinovich. Transferable environment poisoning: Training-tiin time attack on reinforcement learning. In Proceedings of the 20th international conference on autonomous agents and multingent systems, pp. 1398-1406, 2021.
Yinglun Xu, Qi Zeng, and Gagandeep Singh. Efficient reward poisoning attacks on online deep reinforcement learning. Thans. Mach. Learn. Res., 2023, 2023. URL
Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning, pp. 11225 11234. PMLR, 2020.
13

---

Published in Transactions on Machine Learning Research (11/2025)
A Proof
Proof for Theorem 3.4
By the first property of an efficient adversarial MDP attack, the target policy is also the optimal policy on the adversarial environment constructed by the attack #* = n!, For any op-efficient learning algorithms, have E d(a', nt (s*)r) E d(a', (s*)p S 6,-
Proof.
Since d(:,)) S is always true, the perturbation at each step satisfies A' - A d(a', xt (st) S. By the second property of an efficient adversarial MDP attack, the total amount of perturbation is bounded by
T
A A
E
d'a'= (s S A 8p-
summaryy an efficient adversarial MDP attack satisfies
B d(a', , (s*)p) s 6p. max, A < A and the attack is (at least) (e, C, with
E E t=1 A'] A 8p. Therefore, by the definition in Eq. 2 Sp. B = A. and C = 86
Proof for Theorem 3.6
Proof. Recall the performance of a policy JR(TT) = Ba- R(s))(s)). The performance of a policy и in the adversarial environment constructed by efficient adversarial MDP attack M(A) can be decomposed
JT(T) = Exng" R(s,
= JR( - Em-ry (R(s, (()) - R(s, (s)))
(3)
JT(TT) - -A Eurnp d( d()(s) (s))²
(T) A D(=,Tt)
Here we denote D(T1, m2) = Esspri (s), x2(s)). Note that if D2 m2) = 0, then Ti, T2 always take the same actions at the states they can visit an episode, which makes the two policies equivalent effect. The incquality the second last line can only take equality the attack M(A) satisfies A = A d((T(s'),, (s")),, which is also the case of adaptive target attack of q = p. Note that the rewards for the target actions the same in both true and adversarial environments, we have [T(T²) JR(TT).
Based on the decomposition above, the difference between the performance in R of the target policy and any policy æ with D(T,, x)) > at a state s satisfies
[T( - JR(TT) JR(T) JR(TT)- A D(T, x!).
The incquality holds and only if the attack M(A) satisfics = A d(((ss'), xt (88))) Next, we define a per-step perturbation threshold below:
(JR(x) -Ix(xT))
A =
max T:D(
D(T,
For any value of A < A", there exists a policy æ = arg max max (R(T))]a((x") such that Jx(#) D(a,rl)
Jm(m)) O. Therefore, the target policy is not the optimal policy in the adversarial environment, and
14

---

Published in Transactions on Machine Learning Research (11/2025)
the attack cannot be an efficient adversarial MDP attack. When A A and the attack sets A A d(m(s)); x (s))), for any policy n,, we have
Jマ(") (D-[J((())))))) = JR(R)= JT(IT) -
A - x!) S.
Therefore, the target policy is the optimal policy in the adversarial environment in this case, which is constructed by an adaptive target attack with the same value of A A' and q = p. Such attack is adversarial MDP attack, and the minimal value of A is A for an adversarial MDP attack. For any value of A A", the adaplive target attack has IΦ(T) I2(=1) 0 for any policy different from the target policy D(T, rt) > O, which means the target attack the unique optimal policy in the adversarial environment.
B Training Results Details
Training log: To intuitively illustrate the agent's behavior during training under attacks, we provide training logs for the average distance between selected actions and target actions at cach training epoch. In Fig. 2 we show the results on the HalfCheetah environment. We observe that the agent gradually takes actions closer and closer to the target actions, suggesting that it gradually learns actions close to the target actions as the optimal ones. This observation agrees with assumptions and analysis in Section 3 We note that for medium and target policies, the average distance will also decrease as time goes on. The reason could be that there are certain similarities among policies of high performance.
random
medium
expert
Average Distance
0.6
adaptive
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
100
0
100
100 Training Epoch
Training Epoch
Training Epoch
Figure& 2: Training log for HalfCheetah environment under no attack and adaptive target attack with A = 50.
Training Result Range and Mann-Whitney U test:
To better highlight the advantages of our method compared to the baselines, we provide the range of the values e/T and C/T from all runs of each experiment in Table. ] The results show that the range of e/T under the adaptive attack is clearly disjoint from that under the random attack and no attack. This clearly demonstrates the advantage of the adaptive attack in misleading the agent to target policy compared to the naive baselines.
For the adaptive attack and the neighborhood attack, the ranges of e or C' under the two attacks have overlap in some cases. Therefore, to more clearly and rigorously compare the efficiency the two attacks, we perform Mann- WhinnWh U test. In Table 5 we explicitly mark the cases with 'disjoint' where the two ranges have no overlap. Except for the range of C/T in the Hopper environment learned by the SAC algorithm, the maximal value of c/T and C/T under the adaptive attack are always less than the minimal value of those under the neighborhood attack. Note that in the rest of the cases, the average values of e/T and C/T under the adaptive target attack are always less than those under the neighborhood attack. Therefore, we show the value for the alternative hypothesis that the distribution of e/T or C/T under the adaptive target atlack is stochastically less than that under the neighborhood attack. For the values of e/T' the two ranges are disjoint most cases, and the p values are very small the rest of the cases, indicating that the advantage of our algorithm clear in misleading the agent into taking target actions. For the values of C/T, in 4 out of cases, the two ranges disjoint the p value is small. In 2 case, the value of p close to or two ranges
15

---

Published in Transactions on Machine Learning Research (11/2025)
are disjoint and the neighborhood attack has smaller values. In the rest 3 cases, while the average values under the adaptive target attacks are smaller, the p values are not low. In summary, the Mann-Whitme tests show that, using similar or smaller budget of C² the adaptive target attack can better mislead the agent to take actions close to the target actions compared to the neighborhood attack.
Env-Alg
Clean (f)
Adapt (+)
Neigh (f)
Random (+)
Adapt (9)
Neigh (4)
Random (7)
HC-DDDG
0.44, 0.59
0.08, 0.10
[0.10, 0.49
[0.62, 0.63
[4.23, 4.96
2.70, 25.94]
[24.98, 25.01
HC-TD3
00.37, 0.58
[0.09, 0.14
[0.10, 0.52
[0.53, 0.56
[4.44, 6.89
[.779, 25.79]
[24.98, 25.04
HC-SAC
00.49, 0.54
[0.08, 0.15
[0.11, 0.37
00.52, 0.53)
[4.05, 7.57
[1883, 21.07]
[24.98. 25.04
HP-DDPG
0.49, 0.52
00.17, 0.18
(0.25, 0.89]
[0.52, 0.53
[3.30,358
[3.73, 19.96]
[9.99, 10.00)
HP-TD3
00.41,0.40 0.49
[0.17, 0.18
00.27,0.0 0.41]
0.44, 0.49
[3.22,3522
[4.43, 7.72]
[9.99, 10.00]
HP-SAC
00.4,, 0.45]
00.16, 0.17
00.25, 0.27]
0.44, 0.48)
[3.88, 3.46
[2.58, 3.03
[9.99, 10.01
WK-DDPG
00.47, 0.51
0.13
[0.19, 0.28
0.44, 0.59
[2.55, 3.44
[2.11, 6.36
[10.00, 10.02
WK-TD3
0.46, 0.65
00.44.
[0.24, 0.36]
00.37, 0.53]
3.42)
[3.75, 10.10]
[9.99, 10.01
WK-SAC
00.35,0.4l 0.49)
0.155, 0.19
10.19, 0.29]
[0.35, 0.43]
[3.09, 3.79
[1.00, 6.65]
[9.99, 10.01)
Table Range of the values over all runs of the experiments in Table 1
DDPG
TD3
SAC
H/T-HalfChhetall
disjoint
0.001
0.002
< /T-Hopper
disjoint
disjoint
disjoint
</ //T- Walker
disjoint
disjoint
disjoint
C/T-Halle TTT--aaffheetah
0.43
0.24
0.94
C/ ToHopper
disjoint 0.69
disjoint. disjoint* *
C C/T-We Walker
disjoint 0.001
Table 5: A direct comparison between the adaptive target attack and the neighborhood attack. The names of columns are the learning algorithms, and the names of the rows are the enviromments and the value to compare. We use 'disjoint" to highlight the cases where the ranges of the values under the two attacks are disjoint. In most cases, the maximal value of c/T C/T under the adaptive target attack is less than the minimal value of that under the neighborhood attack. The only exception is marked by 'disjoint*? : In the cases where the two ranges are not disjoint, we show the value for the alternative hypothesis that the distribution of e/T' or C/T under the adaptive target attack is stochastically less than that under the neighborhood attack. Note that The average values of c/T and C/T under the adaptive target attack are always less than those under the neighborhood attack, except for the average values of C/T in the learning scenario of Hopper environment learned by the SAC algorithm.
C Additional Ablation Study
Different values of per-step corruption A
Here, we study the influence of the per-step corruption A on the adaptive target attack. We take the HalfCheetah environment and medium target policy as an example and test the attack with A = 10, 30, 50. The results from Table G show that, with a higher value of A, the attacker can better mislead the agent take actions close to the target actions, but it will also require the attacker to spend more corruption in total. Such trade-off implies that increasing the value of A may not make the attack strictly more elficient. When the value of A is too small, the attack is not effective. The reason is that the per-step corruption is too small to make the target policy optimal in the adversarial reward function. However, when A is big enough, increasing the value of A does not necessarily make the attack more efficient. When A is large enough, the agent will gradually converge to always follow the target behavior. A higher value of A may not make the convergence process much faster, but it will require much more budget.
RL environments with discrete action spaces Here, we empirically verify that the adaptive target attack is efficient. against RL environments that have discrete action spaces. In this case, we define the distance
16

---

Published in Transactions on Machine Learning Research (11/2025)
Alg-A
Clean (f)
Adapt (+)
Adapt (+)
DDPG-50
0.52(0.05)
0.09(0.01)
4.58(0.41)
TD3-50
(0.07)
0.10(0.02)
5.24(0.49)
SAC-50
0.52(0.02)
0.12(0.01)
5.75(0.76)
DDPG-30
0.52(0.05)
0.09(0.01)
2.83(0.21)
TD3-30
0.51(0.07)
0.10(0.01)
3.01(0.26)
SAC-30
0.52(0.02)
0.11(0.01)
3.29(0.27)
DDPG-10
0.52(0.05)
0.16(0.03)
(..27)
TD3-10
0.51(0.07)
0.16(0.03)
1.61(0.29)
SAC-10
0.52(0.02)
0. 0.16(0.02)
1.62(0.20)
Table 6: Performance of the adaptive target attack for different values of A.
measure between two actions as: d(a1,02) = 1aa # a2). Under this measure, the distance between two actions is they are different and 1 if they are identical. we consider two popular classical control problem from Gym (Brockman et al. 2016): MountainCar and Acrobot. For the learning algorithms, we consider the satte-of-theaand algorithm: double dueling DQN van Hasselt et al. (2016). For the target policies, we consider three different types of policies, including random, medium, and expert policies. The results are shown in Table C Similar to the case of continuous action spaces, the attack can make the agent mostly take exactly the target actions during training with very limited attack budgets. The empirical observation verifies that, the adaptive target attack is also efficient against the RL environments with discrete action space.
Tav-Tar-Alg-
(clean)
(adaptive)
(adaptive)
Acrobot-rand-double.
0.68(0.01)
0.02(0.01)
0.10(0.05)
Acrobot-medd-oubbee-
0.36(0.01)
0.08(0.01)
0.40(0.05)
Acrobol-exxp-ooublee
0.33(0.01)
0.09(0.01)
0.45(0.05)
0.64(0.01)
0.03(0.005)
0.15(0.02)
MountainnCa---od---uubb----
0.33(0.01)
0.07(0.02)
(0.100.10)
0.23(0.01)
0.07(0.02)
0.35(0.10)
Table Efficiency of the adaptive target attack in the RL environments with discrete action spaces.
Attack under a hard limit on total budget
Here, we consider another realistic attack scenario where the attack has to stop applying perturbation after the total perturbation reaches C. For the HalfCheetah environment, Hopper environments,
set C/T = 4; for the Walker and set C/T = 3. This setup ensures that the attacks consider will likely run out of budget during the training process. The results are shown in Table g Recall our analysis in Section 3 states that with a sufficient value of B, the adaptive attack can achieve small values of e and only requires a small budget of C. Here, our empirical results further show that even when the attack stops applying corruption in the middle of training, our adaptive altack can still make the agent mostly select larget actions. Compared the case when there is no hard limit on C, the average values of c/T are only slightly increased. The reason is that our attack already makes the agent learn most of the target actions as the optimal actions before running out of budget. So when there is more perturbation, the agent will only seldom take actions far from the target actions, and it will take long time for the agent to gather enough unperturbed new information to reverse the impact of previous corrupted information on the actions far from target actions.
D Robot Behavior Controlled by Different Policies
In this section, we show the simulation results of a robot controlled by different policies the HalfCheetah environment. A detailed explanation of the environment can be found in Todorov al. (2012); Wawrzyúski (2009). In general, the cat-like robot (cheetah) is controlled to run on flat terrain. The agent can apply torques at the conjunctions of the cheetah, and the goal is make the cheetah run as fast as possible.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Datasct-Alg
Clean (p)
Adapt (constrained)
Adapt (unconstrainod)
HC-DDPG
0.52(0.05)
(.10()
0.09(0.00)
HC-TD3
0.51(0.07)
0.12(0.03)
0.11(0.02)
HC-SAC
0.52(0.00)
0.16(0.03)
0.11(0.02)
HP-DDPG
(0.0(1
0.19(0.00)
0.17(0.00)
HP-TD3
0.45(0.03)
0.18(0.00)
0.17(0.00)
HP-SAC
0.43(0.01)
0.17(0.00)
0.17(0.00)
WK-DDPG
0.49(0.02)
0.15(0.01)
0.15(0.02)
WK-TD3
(.54().07)
0.18(0.01)
0.16(0.01)
WK-SAC
(.41().05)
0.20(0.00)
0.18(0.01)
Table 8: Performance of attacks in the learning scenarios with hard limit on budget C' (constrained) compared to the scenario with no hard limit (unconstrained).
First, we show the motion of the cheetah controlled by an expert-level policy, The full video clip can be found in the supplementary materials, which provides more intuitive observations. The policy is learned by training the DDPG algorithm on the actual environment for a sufficient amount of time. In Fig. 3 we show sample frames of the motion focusing the moment when the front leg of the cheetah touches the ground. We observe that the Cheetah runs fast and elegantly (it is more direct to see from the video clip), expected. Particularly, whenever its front tip touches the ground, the tip is usually almost vertical to the ground. After the contact, the front tip swings backward. This is similar to how a. horse runs in nature.
Next, we show that the motion of the cheetah is controlled by a medium policy, which is also the target policy we use in our experiments. The policy is learned by the TD3 algorithm training on the actual environment. for an insufficient time. In this case, we observe in Fig g that the Cheetah runs slower than the expert policy (which is also indicated by its performance lower than expert policy performance). In addition, sometimes when the front tip of the cheetah touches the ground, it is nearly horizontal to the ground. Even more, afterward, the front leg almost immediately swings forward. Intuitively, such an unnatural running behavior is very different from how Cheetahs run in the real world, and it might cause unwanted results, such as causing damage to the joints in practice.
In the end, we show the motion of the cheetah controlled by the policy learned under the attack. Specifically, we examine the policy learned in our experiment in Section 4.2 where the target policy set as the medium policy we show above, and the attacker applies the adaptive target attack. Although the learned policy is not exactly the same the medium policy, we observe in Fig 3 that the cheetah controlled by the learned policy exhibits the same behavior as that controlled by the medium policy. This suggests that the adaptive target attack successfully teaches the agent a dangerous behavior with a limited attack budget.
E
Target Policies with Specific Goals
Previously, we have tested our attacks with general target policies randomly chosen based on their performance in the environment. To show the impact of the attack on the agent more intuitively, we test a case where the target policies are optimized for a specific, concrete goal that differs significantly from the original goal of the environment.
More specifically, we choose the HalfCheetah environment as an example. In this environment, the original goal is to make the cheetah run forward fast possible. Therefore, we set the adversarial goal to be the opposite: make the cheetah run backward as fast as possible. Such an adversarial goal has distinct behavior from the original goal, which is concrete and unwanted by the agent. To acquire target policies that can achieve a high performance on the adversarial goal, we apply the DDPG algorithm on the Halfcheetah environment with all rewards inverted during training. The original reward in the environment is the distance the cheetah moves forward in a timestep, so the inverted reward measures the distance it moves backward, which is aligned with the adversarial goal. In this case, we acquire a target policy that achieves performance as low as -500 in the environment. The performance here measures the distance the cheetah moves forward during
18

---

Published in Transactions on Machine Learning Research (11/2025)
Expert Policy
Target Policy
Learned Policy
Figure 3: The motion of the cheetah when controlled by different policies. The 'expert policy' is learned by the agent when there is no attack. The 'target policy' is learned by the DDPG algorithm training for sufficient time with no attack. The target policy is the medium policy we use in our experiments in Section 4 It learned by the TD3 algorithm training for an insufficient time. The 'learned policy' learned by the DDPG algorithm under the adaptive target attack with the 'target policy' as the target. For each policy, sample four frameworks before and after the front tip of the cheetah touches the ground. The contact. happens between the second and the third frames.
the simulation. As comparison, random policy's performance is 0, and expert policy's performance is 12000. This suggests that the target policy here makes the cheetah run backward fast.
With the policy acquired above as the target policy, we adopt the same setup as the experiments in Section 4.2 to test the performance of our attack. Our results show that the average distance between the actions taken by the agent and the target actions is d = 0.10, and the total budget cost is = 5.12, which similar to our observations in Table 1 The performance of the learned policy is 5552, which low and similar that of the target policy. This indicates that our attack as efficient when the target policy is optimized on specific adversarial goal of making the cheetah run backward. To show the results more intuitively, in the supplementary materials, we provide the video for the motion of the cheetah controlled by the target policy and the policy learned under the attack.
F Discussion on Attack Detectability
A practical consideration for robustness against attack is detecting the abnormality in the training data induced by poisoning attacks. We find a line of literature studying the problem of detecting the anomaly during the testing time when the learned policy is deployed in the context of RL Müller et al. (2022) Haider ct al. (2023); Nasvytis et al. (2024). These works assume that the training time is uncorrupted and train their detector during the training process, which cannot be used detect corruption during training. Therefore, it is unclear how to defend against the data poisoning attack through anomaly detection at training time in the online RLHF setting. Particularly, find this challenging for the following reasons.
The distribution of the data obscrved in online training keops changing during training. As a result, natural to see 'abnormal' data points compared the current dataset distribution. For example, consider an agent that is initialized from random policy and eventually converges expert.
19

---

Published in Transactions on Machine Learning Research (11/2025)
policy during a typical online RL training process. The distribution of the data at the beginning is collected by random behavior and has low rewards. As training continues, the agent's 8 policy improves and starts collecting high rewards, which are 'abnormal' compared to the past reward distribution. How the data distribution changes depends on the environment and the learning algorithm, and it is unclear how would change before training.
In an RL task, the agent visits different states sequentially, and at each state, it collects reward from reward function depending on the action it takes. In different states, the reward function of action can be very different. It is normal to collect high rewards at state and low rewards at a different state using the same exploration policy.
In conclusion, it is challenging to judge if an 'abnormal' data point is induced by the attack because they naturally exist in the online RL training process.
Empiricallyy, we test an intuitive anomaly detection method for the dynamic data distribution. The method keeps track of the most recent N rewards collected during training. Let μ and of be the mean and variance of the past N rewards before timestep the reward y¹ received at time t is detected as abnormal if (r*--") where >> 0 is a hyper-parameter set by the method. We set moderate window size N = 500 and test. on the Halfchectah environment learned by the DDPG algorithm. In the case when there is no attack, in order not to detect true rewards as abnormal data, the method needs to set a > 100. Under this setup, when the agent is under our adaptive target attack with A = 50 (the setup in our main results), the method cannot detect any corrupted data. We have a similar observation when we set the size of the window be different values N 100, 1000. The result suggests that it can be challenging detect data corruption by anomaly detection in the online RL setting. We believe it is challenging yet important future work to develop efficient anomaly detection methods for online RL during training.
a-a"
vazcp
Average Distance
edeetivv
0.4
0.3
200
400
600
Training Epoch
Figure 4: Training log of PPO algorithm learned in Swimmer environment under the attacks.
G Comparison to va2c-p
We compare our attack to the va2c-p attack from Sun et al. (2200) As an example, we consider the Swimmer environment learned by the PPO algorithm. To avoid degrading the effect of the valc-p attack, we implement our attacks in their code and run the experiment. Due to the limitation in their code, only naive target policies are considered which always output the same action for any state. We adapt attack to work with the constraints in their work by forcing the attack to stop applying perturbation if doing so breaks the constraint. For the values of constraints, we set c = 0.5 and C/K = 1. For our adaptive target attack, we set 1 1. The result of the training log of the agent under the attacks is shown in Figure 4 We find that our attack has comparable efficiency to the va2c-p attack. Note that the comparison not fair in the first place since va2c-p attack works in white-box setting while our attack works in black-box setting. Our attack also modified here as is optimized for a different constraint on the attack that is actually more general. We also find that our attack runs about 50 times faster than the va2o-p attack. It is noteworthy that our attack
20

---

Published in Transactions on Machine Learning Research (11/2025)
is also much easier to implement, as its implementation is independent of the learning algorithm and only depends on the interactions during training.
21


Reviews:
[{"review_id": "h99QCY3XRh", "reviewer": "Reviewer_wAsU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper presents a universal black-box targeted reward poisoning attack against online deep reinforcement learning (DRL) systems. The attack is designed to work against any efficient online reinforcement learning algorithm without requiring knowledge of the specific learning algorithm or environment dynamics—only awareness of the state and action spaces is needed. \n- The attacker operates by strategically perturbing reward signals during training to manipulate a DRL agent into adopting a policy that closely approximates a specific undesirable target policy, while requiring minimal computational resources and reward perturbation budgets.\n- They develop an \"efficient adversarial MDP attack\" framework where the attacker constructs an adversarial environment with modified rewards, making the target policy optimal within this corrupted environment. The authors define $\\delta_p$-efficient learning algorithms, the class of algorithms where their attack is applicable. Briefly, these are algorithms that select actions close to optimal actions for the majority of training time and the authors suggest that existing efficient deep reinforcement algorithms are $\\delta_p$-efficient learning algorithms.\n- The authors propose an adaptive target attack algorithm that applies reward perturbations where the penalty imposed on the reward signal is proportional to the p-norm distance between the agent's current action and the target action (action preferred by the attacker).\n- The authors test their attack on DDPG, TD3 and SAC algorithms in HalfCheetah, Hopper and Walker environments and show the robustness of their method. They evaluate their method against no corruption setting and two baseline attacks - Neigh (fixed penalty if the agent takes an action that is at a distance from the target action greater than a threshold), and random (add a randomly sampled reward corruption from a specified interval).", "claims_and_evidence_yn": "No", "claims_explanation": "**Theorem 3.4** - For an efficient adversarial MDP attack $M(\\Delta) \\in EM$, it satisfies equation (1) with $\\epsilon \\leq \\delta_p$, $B = \\Delta$, and $C \\leq (\\delta_p)^p \\cdot \\Delta$.\n\n**Proof sketch for theorem 3.4**-\nBy the definition of distance, $d(\\cdot, \\cdot) \\leq 1$\nBy the definition of adversarial MDP,  \n\n$\\hat{\\pi}^* = \\pi^{\\dagger}$  \n\nBy definition 3.3 (Efficient Adversarial MDP), \n\n$\\Delta^t \\leq \\Delta \\cdot d(a^t, \\pi^{\\dagger}(s^t))^p \\leq \\Delta$\n\nThen,\n\n$\\sum_{t=1}^{T} \\Delta^T \\leq \\Delta \\cdot \\sum_{t=1}^{T} d(a^t, \\pi^{\\dagger}(s^t))^p = \\Delta \\cdot \\|\\mathbf{d}\\|_p^p$\n\nApplying expectation,\n\n$\\mathbb{E}[\\|\\Delta\\|_1] \\leq  \\mathbb{E}[\\Delta \\cdot \\|\\mathbf{d}\\|_p^p]$\n\nWhich leads to,\n\n$C \\leq E[\\Delta \\cdot \\|\\mathbf{d}\\|_p^p] \\leq \\Delta \\cdot (\\delta_p)^p$\n\n**Issues with proof for theorem 3.4** - \n- If a<=b and a<=c, it does not necessarily imply b<=c. Similarly if, $\\mathbb{E}[\\|\\Delta\\|_1] \\leq C$ (by definition 3.2) and $\\mathbb{E}[\\|\\Delta\\|_1] \\leq  \\mathbb{E}[\\Delta \\cdot \\|\\mathbf{d}\\|_p^p]$ it does not necessarily imply $C \\leq E[\\Delta \\cdot \\|\\mathbf{d}\\|_p^p]$.\n- From definition 3.1, we know $E[\\|\\mathbf{d}\\|_p] \\leq \\delta_p$. I would like to request the authors to clarify how $E[\\Delta \\cdot \\|\\mathbf{d}\\|_p^p] \\leq \\Delta \\cdot (\\delta_p)^p$ was derived.\n---\n\n**Theorem 3.6.** Let $EM$ be the class of efficient adversarial MDP attacks and $M(\\Delta)$ be an instance with perturbation bound $\\Delta$. Denote $\\Delta^* = \\min_{M(\\Delta) \\in EM} \\Delta$ as the minimal value of $\\Delta$ for an efficient adversarial MDP attack. The value of $\\Delta^*$ is\n\n$\\Delta^* = \\max_{\\pi: D(\\pi, \\pi^{\\dagger}) > 0} \\frac{\\mathcal{J_{R}}(\\pi) - \\mathcal{J_{R}}(\\pi^{\\dagger})}{D(\\pi^{\\dagger}, \\pi)}.$\n\n**Proof sketch for theorem 3.6** -\n\nRecall the performance of a policy $\\pi$ is $\\mathcal{J_{R}}(\\pi) = \\mathbb{E}_{s \\sim \\mu^\\pi} \\mathcal{R}(s, \\pi(s))$. \n\n$\\mathcal{J_{\\hat{R}}}(\\pi) = \\mathbb{E}_{s \\sim \\mu^\\pi} \\hat{\\mathcal{R}}(s, \\pi(s))$\n\n$= \\mathcal{J_{R}}(\\pi) \\mathcal{R}(s, \\pi(s)) - \\mathbb{E}_{s \\sim \\mu^\\pi} (\\mathcal{R}(s, \\pi(s)) - \\hat{\\mathcal{R}}(s, \\pi(s)))$\n\n$\\geq \\mathcal{J_{R}}(\\pi) - \\Delta \\cdot \\mathbb{E}_{s \\sim \\mu^\\pi} d(\\pi(s), \\pi^{\\dagger}(s))^p \\tag{3}$\n\n$:= \\mathcal{J_{R}}(\\pi)(s) - \\Delta \\cdot D(\\pi, \\pi^{\\dagger})$\n\n$\\mathcal{J_{\\hat{R}}}(\\pi)\\ge \\mathcal{J_{R}}(\\pi) - \\Delta \\cdot D(\\pi, \\pi^{\\dagger})$\n\nFollowing the above steps, the authors detail that the value of $\\Delta$ to make the target policy have a higher performance than other policies $\\pi : D(\\pi^{\\dagger}, \\pi) > 0$ satisfies:\n\n$\\Delta \\geq \\max_{\\pi: D(\\pi, \\pi^{\\dagger}) > 0} \\frac{\\mathcal{J_{R}}(\\pi) - \\mathcal{J_{R}}(\\pi^{\\dagger})}{D(\\pi^{\\dagger}, \\pi)} =: \\Delta^*.$\n\n**Issues with proof for theorem 3.6** -\n- $\\mathcal{J_{R}}(\\pi) \\mathcal{R}(s, \\pi(s)) - \\mathbb{E}_{s \\sim \\mu^\\pi} (\\mathcal{R}(s, \\pi(s)) - \\hat{\\mathcal{R}}(s, \\pi(s))) $, \n    \n    should be $\\mathcal{J_{R}}(\\pi) - \\mathbb{E}_{s \\sim \\mu^\\pi} (\\mathcal{R}(s, \\pi(s)) - \\hat{\\mathcal{R}}(s, \\pi(s)))$ in the second line.\n\n- Given that $\\mathcal{J_{\\hat{R}}}(\\pi)\\ge \\mathcal{J_{R}}(\\pi) - \\Delta \\cdot D(\\pi, \\pi^{\\dagger})$, we get $\\Delta \\ge \\frac{(\\mathcal{J_{R}}(\\pi) - \\mathcal{J_{\\hat{R}}}(\\pi))}{D(\\pi, \\pi^{\\dagger})}$. I would like to request the authors to clarify and provide missing steps for how they derived the $(\\mathcal{J_{R}}(\\pi) - \\mathcal{J_{R}}(\\pi^{\\dagger}))$ term instead of $(\\mathcal{J_{R}}(\\pi) - \\mathcal{J_{\\hat{R}}}(\\pi))$ in their final equation. \n\n\nIt would be helpful if the authors could clarify the above issues raised for theorem 3.4 and 3.6.", "audience_yn": "Yes", "audience_explanation": "Yes, the findings of the paper would be of interest to reinforcement learning and AI safety researchers. The authors show strong results in their empirical evaluations and propose a novel attack framework which could be of interest to the community.", "requested_changes": "- Equations 1 and 2 appear to have substantial overlap, and their repeated presentation may create unnecessary redundancy. Consider consolidating or clarifying the relationship between these equations to improve readability and flow.\n- \"First we argue that letting $\\mathcal{L}$ include all possible algorithms makes the problem trivial and is meaningless for the attacker. In this case, there always exists an algorithm that takes specific actions regardless of the environment and the attack to maximize $\\epsilon$. Such an algorithm dominates the value of $\\epsilon$ and makes it the same for all attacks.\" - It would be helpful to the readers if the authors provided an example of such an algorithm.\n- In Definition 3.1, the authors state that \"it will take the actions close to the optimal actions for most of the time during training, that is, $E[∥d∥_p] ≤ δp$\" However $d$ is the distance between the agent's action and the target action (action favored by the attacker) as described in section 2. This seems misleading, as \"optimal actions\" typically refers to actions derived from the optimal policy in the true environment, not the attacker's target policy. If this definition refers to the optimal policy in the adversarial MDP, it creates confusion since the adversarial MDP framework is introduced later in Section 3.2, making this definition difficult to understand at first reading.\n- In the proof of Theorem 3.6 (Appendix A), the authors present the inequality: $V_{\\tilde{\\mathcal{M}}}^{\\pi^{\\dagger}}(s) - V_{\\tilde{\\mathcal{M}}}^{\\pi}(s) \\leq (V_{\\mathcal{M}}^{\\pi^{\\dagger}}(s) - V_{\\mathcal{M}}^{\\pi}(s)) + \\Delta \\cdot D_s(\\pi, \\pi^{\\dagger})$. However, the derivation of this inequality and the motivation for it is not sufficiently detailed. Could the authors provide a more detailed derivation and motivation showing how they relate the value functions across these two environments? \n- Minor notation error in section 3.2 - \"In this case, for any algorithm $\\text{Alg} \\in \\mathcal{L}$, it is always true that $E\\left[\\sum_{t=1}^{T} \\left(d(a_t, \\hat{\\pi}^*(s_t))^p\\right)^{1/p}\\right] \\leq \\delta_p$\". I think it should be -\n\n    $E\\left[(\\sum_{t=1}^{T} d(a_t, \\hat{\\pi}^*(s_t))^p)^{1/p}\\right] \\leq \\delta_p$.\n\n\n- Minor notation errors in appendix A - $\\sum_{t=1}^{T} \\Delta^T \\leq \\Delta \\cdot \\sum_{t=1}^{T} d(a^t, \\pi^{\\dagger}(s^t)^p = \\Delta \\cdot \\|\\mathbf{d}\\|_p^p$. If I am right, It should be -\n\n    $\\sum_{t=1}^{T} \\Delta^t \\leq \\Delta \\cdot \\sum_{t=1}^{T} d(a^t, \\pi^{\\dagger}(s^t))^p = \\Delta \\cdot \\|\\mathbf{d}\\|_p^p$\n- Typo in section 3.2 - \"optimal in the adversarial environment $\\hat{\\mathcal{R}}$\"\n- Typo in appendix A, proof for theorem 3.6- \"has a higher\". \n- Typo in appendix B, \"We test the attack with different hard limits, C = 2.5, 5, 7.5\". Should be C/T.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qDz5WHAYM6", "reviewer": "Reviewer_znPE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "First of all I would like to apologize to the authors and action editor for taking longer than expected to review the paper. I find the topic of the paper interesting and I wanted to go through the claims and corresponding evidence in detail and evaluate them for their accuracy and clarity. \n\nThe summary of the contribution is:\n\nThis paper studies one way to do a \"targetted\" black box attack to deep reinforcement learning methods, the claim of the author is that this attack (or two attacks if one considers the base attack as separate;) is \n\na. \"efficient\": satisfies budget constraint\n\nb. does not require knowledge of the parameters of the algorithm or even the specific algorithm being used\n\nc. works for general RL problems with both discrete and continuous action spaces and applies to diverse learning algorithms\n\nTo make this problem tractable the authors consider and construct a class of learning algorithms that the learner (one using the RL algorithm would use) might use. And their framework \"attack framework called ‘efficient adversarial reward engineering’ (is developed) such that the behavior of an arbitrary efficient learning algorithm becomes predictable under any attack under the framework.\"\n\n*I'll be honest I tried understanding but the last line which is one of the core contributions remain unclear to me, and as my objections would indicate part of my concerns will be addressed if this is made more clear*.\n\nFinally the authors present experimentation in standard environments: HalfCheetah, Hopper, Walker, and Acrobot and against standard DRL algorithms: SAC, TD3, DDPG and DQN, which although might not be state-of-the-art but are certainly widely used and serve as a decent harness for numerical simulations. \n\nI would kindly request the authors if I have missed or misread any by-and-large contributions, it would help in an objective evaluation", "claims_and_evidence_yn": "No", "claims_explanation": "I will try to articulate my fundamental claims-evidence mismatch of the paper. \n\nThe authors claim \" Our attack is universally efficient against any efficient learning algorithm training in general RL environments and requires limited attack budgets and computational resources.\" (verbatim from abstract) \n\nHowever I strongly believe that the assumptions that the authors operate can only imply a weaker version of this claim and should be qualified everywhere this claim is used. Even empirically I do not find any evidence for this. \n\nLet me try to explain this.\n\n*Simple language* \n\nHere is what the more general claim from the abstract and the rest of the paper can imply (or atleast what I thought without going throught the paper)\n\nI can actually get the RL algorithm to take unsafe or actions very different from the ones they would have taken. \n\nWhat the theorems, assumptions and experiments support:\n\nThe RL algorithm can indeed be attacked but the perturbed action are not very different from the optimal actions the algorithm would have taken otherwise. \n\nThis questions the *generality* and *universality* of the approach. \n\nI believe such a work is still an important step in that direction but the claims need to be made more exact (and also the clarity of the writing) so that the evidence is calibrated. \n\nIf the authors wish to keep the claims exact, they would have to provide evidence (theoretically and empirically) for:\n\nEnvironments where the action taken was completely different (the cheetah running in opposite direction for example) still respecting the constraints? I imagine not therefore the attacks are not *universal* without further qualification. I know we live in a world obsessed with universality but I personally believe simple niche results are instrumental in scientific progress and there's no special pride in claiming *universaility*. \n\nTo the cheetah example: Such an unnatural running behavior could be why the cheetah runs slower and might cause damage to the robot in the long run in practice. Can we quantify the slowness of speed, and if there is any evidence to damage for the same for real-life  studies for their damage? In the figure 2 I don't see a drastic difference in the policies. \n\nAlso are there fundamental results which can show such a *universal* attack is impossible. \n\n*Technical*\n\nThe authors assume that the learning algorithms learn near the optimal action most of the time, this is untrue in general, therefore one should make it clearn It is true when the algorithm is minimizing cummulative regret, however there too during the exploration or *learning* the algorithms can take widely different actions. There is even work which shows that adversarial learning can benefit the actual performance of the algorithm. In fact in practice one stops RL and deploys the agent once the reward curve plateus. Therefore this central assumption does not cover efficient learning algorithms in practice and one needs to be clear in their claims and specify algorithms which take the optimal actions most of the time. (This assumption is valid for bandit algorithms and recsys algorithms though so might be worth mentioning it).\n\nSecondly Point 2 of the efficient adversarial MDP attack, \"The differences between the adversarial and true reward functions are small for the actions whose distance to the target actions is small at any state.\" - I mean this makes intuitive sense but shouldnt this be a feature of the MDP (or Lipschitzness of its reward function) I do not understand what the authors mean by it, but I do feel this is the assumption that leads to a tractable framework but combined with the budget constraints leads to actions which are not very different. \n\n\n\nI also have certain problems with the framework: \n\n1. Why do the authors not consider horizon based RL? If they do not want to, this should be made clear in the claim. \n\n2.  Further constructing an environment given a target policy is non-trivial and so it is not clear to me how one constructs the $\\hat{\\mathcal{R}}$. \n\n3. What if there is a set of optimal actions, in this case the definition of the paper are invalid and so one needs to either reduce the class of algorithms or update the definitions.\n\n4. Please explain what p and q are? I thought both of them were selected by the attacker but then $p\\neq q$ makes no sense if optimality is in $p=q$. A clear exposition of this would be very helpful. I would be able to evaluate your numerical experiments once you clarify this and other queries. \n\nI would love to engage with the authors and help them if necessary in making the claims more accurate and the evidence more convincing.", "audience_yn": "Yes", "audience_explanation": "I believe that this paper definitely contributes to the literature of poisoning attacks in reinforcement learning. Especially I imagine in the near future, we would be able to achieve truly online continual reinforcement learning, such attacks would be concerning and studying how one can do such attacks or modelling them is very relevant to designing countermeasures. Or even using these algorithms for the good when one is trying to bypass reinforcement learning of bad actors.", "requested_changes": "There many changes that I would request the authors to make, however I would like to reserve them once the discussion is over so that I can make more targetted suggestions and fundamental changes.\n\nAs a general suggestion, I would appreciate if the authors could improve clarity of their writing and mathematical rigour. Here are few non-exhaustive suggestions. Note that I know the answer to all of these questions having read and thought about the paper extensively. However ideally a scientific paper should have a clear, simple and easy to follow exposition - it is what helps make it's dissemination easier. This list is indicative of the type of changes I would request you evaluate your paper with and improve it. \n\n1. \"The performance of a policy π is then defined as JR(π) := Es∼µπ,a∼π(s)R(s, a)\" -> there are many objective for reinforcement learning (infinite horizon with discounted/average cost, finite time horizon: what is your horizon? Is your method restricted to onpolicy or applicable to off-policy methods as well?)\n\n2. Some symbols are never defined before they are used, e.g. what does T denote?  what does $\\Delta$ denote? (most likely the probability simplex but there's no point in keeping the reader guessing)\n\n3.the term \"efficiency\" is abused to a certain extent, please be precise in what you mean by efficiency: sample, data, computation; similarly budget - what is the budget on? is it general? Let me take another example: \"We do not consider the computational cost of constructing the target policy, as it is an input to the targeted attack problem. In general, the efficiency of an attack is measured by the attack’s influence on the learning agent’s output with limited perturbations on the training process (attack budgets).\". \nThis is the first two lines of the paragraph. The first does not naturally follow the second one. What is the point of the paragraph is not clear. There's a lot of vagueness in the second sentence - what is influence? what is the limited perturbation? is attack budgets a modifier on training process or phrase before. Again I know what you mean because i spent a minute deciphering it, but I believe you can rewrite many such sentences much much more simply. Just a suggestion: \"An attack on the RL algorithm can influence its learning. We consider perturbation of the rewards by the attack to be constrainted. Then the magnitude of the shift on the RL policy to the attacker's target policy quantifies the efficiency of the attack. The more closer a attack is able to push the RL policy, the more efficient it is\" (I assume you mean closeness or influence in terms of policy as it was not clear from the original writing) There are many such examples. \n\n\n4. All theorem statements and definitions should be self-complete, a reader should be able to read them independently without searching for the notation. Reintroduce assumptions, notations etc. \n\n5. Isn't the action space and state space an input to the algorithm? \n\n6. Why are there two constraints?? Isn't one enough (the other one can be implied with appropriate value of B/C)\n\nOther than this please ensure that each paragraph and line has a purpose. Redundancy is important but too much fluff makes the time-sensitive reader confused. Use simple small sentences. \nAs I said, I'll suggest more changes once we discuss the claims and evidence more clearly. \n\nI like the organization of the numerical section!", "broader_impact_concerns": "I feel that the work is relevant in studying the behaviour of reinforcement learning algorithms in the wild and adversarial attacks on them. I do not think there are any immediate impact concerns, however I would appreicate if the authors can add a mention to how such attacks can be used if the reinforcement learning is being done for good and the attacks are by adversarial attackers.", "additional_comments": "Is there any zero-game perspective on this problem that the authors can give?", "extra_scores": {}, "extra_text": {}}, {"review_id": "JhDF0f6Sck", "reviewer": "Reviewer_BFz4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a  black‑box reward‑poisoning attack against online deep reinforcement learning. The key idea is to view reward corruption as constructing a new adversarial MDP in which the optimal policy is the attacker's policy of interest. By exploiting the property that modern RL algorithm spend most of the training near the optimal actions, the attacker is able to steer learning by modifying the observed rewards. The paper formalizes this observation, proposes an attack family that works in real time without knowing the learner or the environment, demonstrates efficiency guarantees, and studies the empirical performance of the approach in both discrete and continuous control benchmarks.", "claims_and_evidence_yn": "No", "claims_explanation": "- Invalid threat model\n  - The threat model does not seem to fit into a reasonable RL framework. How would an adversary have the freedom to intercept and perturb the rewards in an arbitrary fashion? In practice it is the practitioner who designs the reward function so as to induce a desired behavior; the attacker can't directly modify the reward function so directly. Any adversary that can arbitrarily perturb the observed reward sequence would make any problem trivial: they can always ensure that the learner observes zero reward everywhere and learns nothing. \n\n  - It seems to me that the only way an attacker could modify the observed without breaking the standard protocol would be in settings similar to the restaurant recommendation example in the introduction: the observed rewards are not changed directly but many fake rewards are created by creating fake reviews. This is not the same as being able to arbitrarily perturb each observed reward. Instead, I think the right way to model reward poisoning attacks would be to say that the adversary has the power to perturb the reward *distribution*, rather than each individually realized reward.\n\n    \n- The experiments are not convincing\n    - Experiments are repeated for only 10 random seeds and confidence intervals \n    are reported. The results suggest statistically significant results, but I find it hard\n    to believe that the reporting of confidence intervals is justified in the first place with \n    so few samples; how was approximate normality assessed?\n\n    - Baselines: it's not clear to me that the implementation of the baseline from \n    Xu 2023 is a fair representation of this baseline's performance: the threshold \n    for triggering the perturbation is set very low in all instances tested. For reference, \n    what is the average value of $d(a_t, \\pi^\\dagger(s^t))$ in these environments?\n    If this is much higher than 2 it would explain why the baseline appears to be inefficient. I think the \n    optimal instatiation of this baseline would only penalize very different actions, rather than \n    almost every action.", "audience_yn": "Yes", "audience_explanation": "The reward poisoning approach presented here has the benefit of being completely black-box and seems to be reasonably efficient", "requested_changes": "- For securing my recommendation, I request that the concerns about experiments addressed\n- I think the paper would be strengthened by adding more discussion/justification/examples of the threat model. It currently seems like it aligns with almost no real RL problem", "broader_impact_concerns": "n/a", "additional_comments": "- I think there should be new notation for the specific class\nof algorithms you're restricting to in section 3.1, since \nif you just look at eq 1 and the definition it is \neasy to miss the fact that the specific thing that has\nchanged is that you have restricted the class of algorithms.\n\n- Theorem 3.4 appears to end abruptly before finishing the statement", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)