Robust Deep Reinforcement Learning against Adversarial Behavior Manipulation
Authors: Shojiro Yamabe
Kazuto Fukuchi
Jun Sakuma
Venue: ICLR 2026 Poster (2026)
Keywords: Renforcement Learning
Robustness
Adversarial Attack
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce a novel attack method for manipulating reinforcement learning agent's behavior using imitation learning, and a first defense strategy based on our theoretical analysis against such attacks.

Abstract: This study investigates behavior-targeted attacks on reinforcement learning and their countermeasures. Behavior-targeted attacks aim to manipulate the victim's behavior as desired by the adversary through adversarial interventions in state observations. Existing behavior-targeted attacks have some limitations, such as requiring white-box access to the victim's policy. To address this, we propose a novel attack method using imitation learning from adversarial demonstrations, which works under limited access to the victim's policy and is environment-agnostic. In addition, our theoretical analysis proves that the policy's sensitivity to state changes impacts defense performance, particularly in the early stages of the trajectory. Based on this insight, we propose time-discounted regularization, which enhances robustness against attacks while maintaining task performance. To the best of our knowledge, this is the first defense strategy specifically designed for behavior-targeted attacks.

Published as a conference paper at ICLR 2026
ROBUST DEEP REINFORCEMENT LEARNING AGAINST ADVERSARIAL BEHAVIOR MANIPULATION
Shojiro Yamabe', Kazuto Fukuchi Jun Sakuma 1.3 Institute of Science Tokyo, University of Tsukuba, SRIKEN AIP yamabe. s. 25000m. isst. accct.ac... jp
ABSTRACT
This study investigates behavior-targeted attacks on reinforcement learning and their countermeasures. Behavior- targeted attacks aim to manipulate the victim's ' behavior as desired by the adversary through adversarial interventions in state obser- vations. Existing behavior-targeted attacks have some limitations, such as requiring white-box access to the victim's 's policy. To address this, we propose a novel attack method using imitation learning from adversarial demonstrations, which works under limited access to the victim's 's policy and is environment-agnostic In addition, our theoretical analysis proves that the policy's s sensitivity to state changes impacts defense performance, particularly in the early stages of the trajectory. Based on this insight, we propose time-discounted regularization, which enhances robustness against attacks while maintaining task performance. To the best of our knowledge, this is the first defense strategy specifically designed for behavior targettd attacks.
INTRODUCTION
Applications of Deep Reinforcement Learning (DRL) have grown significantly in recent years (Berner et al. 2019 Ouyang et al.) (2022] Guo et al., (2025). However, when DRL is deployed in mission- critical tasks (Sallab al.] 2017 Degrave et 2022) Yu et al. 202T). it is crucial understand its susceptibilities to adversarial attacks and deal with them. One such vulnerability is the manipulation of victim's behavior by an adversary that can deceive the state observed by victim agent (Huang
eet al. 2017 Zhang et al. 2020b Sun et al. 2022).
The primary objective of this study is to introduce behavior-targeted attacks and corresponding countermeasures. In this attack, the adversary's s goal is to steer the victim's trained policy toward target policy specified by the adversary. Behavior-targeted attacks introduce novel threats that cannot be realized by conventional wwward-inimizattie attacks, where the adversary's 's goal is to minimize the victim's 's reward (Pattanaik et al.) 2018: Zhang et al.] 2021 McMahan et al., 2024). Moreover, attacks ineffective against these novel threats, which underscores the necessity of dedicated countermeasures. Below, we illustrate two scenarios where the Ochavior-targeted attacks are particularly relevant.
we detail later, existing defenses tailored to counter hewarrdminimizatte
Scenario 1. Consider an autonomous vehicle controlled by DRL agent that receives higher rewards for reaching its destination quickly and safely, but incurs large penalty if causes an accident (Kiran -wwardd-miiiizztiinn attacks would aim to induce an accident at some point along
et al. 202T). the vehicle's 's route. contrast, behavior-targenn victim's 's reward. For instance, the adversary could manipulate the vehicle to slow down at a specific crossing point, creating congestion in a strategically chosen region. Alternatively, the adversary might steer the autonomous vehicle toward particular store to generate economic benefit for the adversary.
attacks can take various forms, regardless of the
Scenario Consider a recommendation system powered by DRL (Chen et al. 2023 Fu et al, 2022) that maximizes user satisfaction (reward) by observing the user's purchase history and recommending relevant items. In this context, reward-ming minimization attacks attempt to degrade user satisfaction by generating suboptimal recommendations. In contrast, behavior- behavior-targeted attacks manipulate the system to serve specific objectives, such as prompting certain products that benefit the adversary or suppressing the recommendation of items specified by the adversary.

---

Published as a conference paper at ICLR 2026
Attack. Although several Sehavior-targeted attacks have been proposed (Hussenot et al. 2020 Boloor et al.] [020; Bai et al.] [2024] 2025), they all share a critical limitation: requiring white- box access to the victim's ' policy, including its architecture and parameters. In many real-world scenarios, the adversary is unlikely to have full knowledge of the victim's 's policy, which severely limits practical
applicability.
To overcome this limitation, we show that an optimal adversarial policy can be learned without requiring white-box access to the victim's 's policy. Specifically, we present a novel theoretical reformulation of the behavior-targeted attack objectíve as the problem of finding an adversarial policy that maximizes cumulative reward in an MDP specially constructed for the attack purpose. Here, the adversarial policy replaces the victim's 's observed state with a falsified state at each time step (see Figure D]. Crucially, because the victim's 's policy is incorporated into the transition dynamics of the constructed MDP, training the adversarial policy does not require white-box access.
Building on this formulation, we propose the Behavior Imitation Attack (BIA), a novel attack framework that is applicable even when access to the victim's 's policy is severely limited. Since the reformulated objective can be optimized via demonstration-
imitation learning, we utilize established algorithms (Ho & Ermon 2016 Kostrikov et al.) 2018 Chang et al.) 2024). The advantage of leveraging imitation learning is that BIA enables training adversarial policies directly from demonstrations of the desired behavior, climinating the need for reward modeling. The adversary can readily prepare demonstrations necessary for BIA simply by performing the desired behavior several times. We empirically show that even under the most restrictive no-box setting, where the adversary cannot observe any output of the victim's 's policy, BIA achieves attack performance competitive with baselines requiring white-box access.
Defense. Most existing defense strategies, such as adversarial training (Zhang al. 2021 Oikarinen
al. 202T Sun et al. 2022] Liang et al. 2022), certified defenses (Wu et al. 2022] Kumar el al. 2022
Mu et al. [224]] Sun et al. [2024] Wang et al. (025). and regret-based robust learning (Jin et al., 2018
Rigter et al. 2021] Belaire et al. 2024), are not readily adaptable to counter behavior-targeter attacks.
This is because the adversary's 's target policy is unknown to the defender and determined independently
of the victim's 's reward. Although policy smoothing (Shen et al. 20200] Zhang et al.) (2020b) may offer
some degree of robustness by stabilizing the policy's 's outputs against adversarial perturbations, their
theoretical robustness guarantees are limited to reward- aeward-miiimization attacks and not directly extended to tarreeee attacks. Moreover, policy smoothing often degrades performance on the victim's original tasks, as it imposes excessive constraints on the policy's representational capacity.
To address these challenges, we derive a tractable upper bound on the adversary's ' gain that holds for arbitrary target policies and integrate it into robust training objective. Our theoretical analysis of the upper bound reveals two key insights: (i) reducing the sensitivity of the policy's 's action outputs to state changes improves robustness, and (ii) this effect is particularly pronounced when sensitivity is suppressed in the early stages of trajectories.
Motivated by these insights, we introduce Timee-Discountd' Robust Training (TDRT), which in- corporates a time-discounted regularization term to suppress the sensitivity of the policy's actions. This regularization more strongly reduces the sensitivity at critical earlier stages of trajectories and progressively weakens at later stages. By concentrating regularization in carly trajectories, TDRT maintains the policy's ' representational capacity and thus preserves original-tash task performance, while simultaneously enhancing robustness against gehavior-targette attacks. In our experiments, com- pared uniform policy-smoothin defenses without -iscoounun.. TDRT improves original-task performance by 28.2% while maintaining comparable robustness. To the best of our knowledge, TDRT the first defense specifically designed for behavior-targeted attacks.
Contributions. Our primary contributions are summarized as follows:
We theoretically reformulate the objective of behavior-targett attack and introduce the Behavior Imitation Attack (BIA), a novel method that leverages well-established imitation learning algorithms and operates under limited access to the victim's policy.
We present Time-Discounted Robust Training (TDRT), the first defense tailored to behavior- targeted attacks. Time-discounted regularization in TDRT is grounded in our theoretical analysis and mitigates original-task performance degradation while preserving robustness.
2

---

Published as a conference paper at ICLR 2026
We demonstrate the effectiveness of our proposed attack and defense methods using the Meta- World (Yu et al. 20200), MuJoCo (Todorov et al. 20122) and MiniGrid (Chevalier-Boits
ct al. 2018) environments.
2 RELATED WORKS
2.1 ATTACK METHODS
Existing attack methods on DRL agents can be broadly classified into three categories: reward- minimization attacks, enchanting attacks, and behavior- attacks aim to reduce the victim's cumulative reward and have been extensively studied (Zhang et al.) 2021 Sun et al. 2022 McMahan et al. 2024). As another direction, enchanting attacks aim to lure the victim into a predetermined terminal state without specifying the full trajectory (Ying et al. 2023). Since neither reward-ming seward-minimization nor enchanting attacks are intended to manipulate the victim's entire behavior, we treat them as out of scope.
attacks. newarddminimizzttin
Behavior-targette attacks instead aim to steer the victim toward adversary-specified target policy, However, existing approaches typically assume white-box access to the victim's 's policy, as discussed above. In addition, some attacks (Hussenot et al. 2020 Boloor al. 2020) tailored to specific domains such as autonomous driving, which limits their generalizability. More recently, [Bii et al. (2024) 2025) proposed a non-heuristic attack based preference-basee based reinforcement learning, but still relies on white-box access and requires three separate stochastic optimization procedures, making it highly resource-intensive.
While several studies have proposed poisoning attacks that intervene during the victim's training phase (Sun al., 2021 Rangi et al. 2022; Xu al., 2023 Xu & Singh 2023 Rathbun et al. 2024
they fall outside our threat model (see Appendix A. for details).
2.2 DEFENSE METHODS
Adversarial training (Zhang al. 2021 Oikarinen al.] 2021] Sun et al. 2022 Liang al. 2022,
Li et al. 2024a) optimizes the agent to be robust against adversarial perturbations generated by hypothetical adversary. However, the effectiveness of adversarial training against behavior-targeted attacks is limited, as the defender cannot anticipate which behavior the adversary aims impose. Existing policy smoothing methods (Shen et al.) 2020; Zhang et al.] (2220b) are designed for reward- minimization attacks and significantly degrade performance the original tasks previously noted.
Other approaches include regret-based robust learning (Jin et al. 2018 Rigter et al., [021 Belaire et al.
2024) and certified defenses (Wu et al. [022] Kumar et al. 2022 Mu er al. [2224] Sun 2024 Wang al. 2025). Regret is defined as the difference between the victim 's attacked and unattacked rewards, and regret-based robust learning focuses on minimizing this difference. Certified defenses guarantee lower bound on the victim's 's reward under attack. As they both use reward-based metrics and are designed for
al.
attacks, they are less effective against behavior-targeted attacks that are independent of the victim's 's reward.
Liu et al. (024) proposed an adaptive defense method robust against multiple types of attacks, not limited to reward-minin minammiaiiiiinooonnnn attacks. In their method, the defender prepares multiple robust policies and then selects the policy that maximizes the victim's reward under attack based on rewards obtained in previous episodes. However, our approach focuses on single static vietim in stationary environment, so their adaptive setting differs from ours.
Several works (Sun & Zheng 2024 YANG & Xu, 2024) propose diffusion-based defenses, which perform purification inference time to remove adversarial perturbations from corrupted states. In contrast, our approach is a training- time regularization method and requires no additional prepro- cessing during inference. As result, these defenses operate different components of the pipeline, making them orthogonal and complementary. For a more detailed and comprehensive review of related work, please refer to Appendix A
3

---

Access to the victim's policy. We assume the adversary does not have full access to the victim's define two Published as a conference paper at ICLR 2026
3 PRELIMINARIES
Notation. We denote the Markov Decision Process (MDP) (S,A, 7), where S is the state space, A is the action space, R S x A + IR is the reward function, and E (0. 1) is the discount factor. We use P(x)) the set of all possible probability measures on X. We denote p: P:S x A + P(S) as the transition probability, initial state distribution. When the agent follows policy π in MDP M, the objective of reinforcement learning train a policy that maximizes JRL. (π) _ EA [R(s, a, s')] = E (E 7 R[st, a2s St+1]],
S + P(AA) as a stationary policy, and as an
the expected sum of discounted rewards from the environment.
State-Adversarial Markov Decision Process. To model the situation where adversary intervenes
in the victim's 's state observations, we use SA-MDP (Zhang et al. (0205). Let the victim follow fixed policy и in the MDP M. In an SA-MDP, the adversary introduces an adversarial policy
V: S + P(S) that interferes with the victim's 's state observations by making the victim observe
false state & v(-1s) at each time step without altering the true state of the environment (see
Figure
The SA-MDP is defined as M (S. 7), where is mapping from the true state E S to the false state space B(s) C that the adversarial policy can choose from. The size of B(s) indicates the adversary's intervention capability and is typically set in the neighborhood of s. The smaller the size of (s),, the more challenging it becomes achieve the attack objective.
palccy
Achersanol
(i1 (8)
y
8
((il) o' ~ p((..))
Figure 1: Overview of SA-MDP
4 THREAT MODEL
This section outlines the threat model for the rchavior-targeted attack. We assume that the victim follows a fixed policy æ, and it observes falsified states generated through the adversary's intervention. We define this process as an SA-MDP M (S, A, R. B, p, 7). The adversary's s objective is to manipulate the victim into performing a specific behavior desired by the adversary, which denoted by the target policy Tgg- To achieve this objective, the adversary trains adversarial policy that makes the victim observe a falsified state at each time step. In this context, the adversary's 's influence against the victim is characterized follows:
policy и, including its neural network parameters and training algorithm. Specifically, access models: black-box and no-box. In the black-box, the adversary can only observe the victim's policy inputs (i.e., observed states) and the corresponding outputs (i.e., actions taken) at each time step. In the no-box, the adversary observes only the policy inputs at each time step. Since the no-box setting provides less information than the black-box, it presents greater challenge to the adversary.
Intervention ability on victim's 's state observations. We assume that the adversary can alter the vietim's 's state observation from the true state to the false state E (()). This assumption is common in most adversarial attacks on RL (Zhagg et al. 2020b 2021
Sun et al. 2022).
ATTACK METHOD
Formulation of adversary's objective. First, we define a composite policy of the victim's policy and the adversarial policy to define the adversarial objective. Let S be true state. The adversarial policy provides the falsified observation å ~ u(-ls), and then the victim selects the next action by a ~ (-) Consequently, the victim's 's resulting behavior policy under the adversary's 's influence is represented by the composite policy
π v(als) 4
Σ (a()) BES
(1)
4

---

Published as a conference paper at ICLR 2026
Building this definition, we define the adversary's objective finding the optimal adversarial policy that aligns и o y with Tgt
arg min Jad (v) - D(1 MN, Tgg).
where D is some divergence measure between two policies.
Unfortunately, existing imitation learning algorithms are not directly applicable for optimizing equation Lunder our threat model. Typical imitation learning methods treat the composite policy Tr the optimizable policy and assume is directly updatable via gradients. However, since the adversary lacks white-box access to the victim policy it cannot backpropagate through x to compute the gradient of the objective with respect to v. Therefore, this assumption does not hold.
To overcome this difficulty, we provide a novel theoretical result that, under a mild assumption, equation 2 can be reformulated into a problem that maximizes the cumulative reward in another MDP M in which the adversarial policy itself serves as the policy to be optimized:
Theorem 5.1. Consider SA-MDP M
R,B.P. 7) with adversarial policy v. Let a denole the victim's 's policy and aig the target policy. Assume that the divergence DD admits the following variational representation:
DTTT ON, Tig) max (es 0, s')) En -
where f and g are arbitrary convex and concave functions, respectively, and d A x S R is discriminator. Let d, be the optimal discriminator in equation Under this assumption, define the reward function Ra and the state transition probability p as follows:
{
Eass [(a(s)p(s (s.)). Ened 7(a())((s1s..)
if B(s) otherwise,
Ra(s,8,55)
(4)
p(s'|s,8) - (<lls)p(s(|s.a)
(5)
aEA
where C a large negative constant. Then, for the MDP Mi (S, Ra, it holds that:
arg min D( v, Rig) arg max E. [Ra(s,
(6)
The proof is provided Appendix B.1 Crucially, any standard RL algorithm can maximize the cumulative reward under MDP M without requiring white-box access to æ since the policy for MDP M depends solely on / rather than on the composite o v. The dynamics of the MDP M encapsulate the victim's policy, and this idea has been considered in (Schott al. 2024 Bai al. 2025). However, their idea was not theoretically justified because (Schott et al. 2024) only introduces the concept abstractly without empirical validation theoretical analysis, and (Bai et al. 2025) lacks
proof and concrete reward design. We provide complete proof and present concrete reward design for learning the optimal adversarial policy for the first time.
Behavior Imitation Attack (BIA). Building the above insight, propose Behavior Imitation Attack (BIA). Specifically, we bridge the behavior-targent attacks and established Imitation Learning (IL) (Ho& & Ermon 2016, Torabi al. 2019. CCang al. 2024) via Theorem 5.1 We then present practical algorithm for implementing BIA.
IL trains policy that mimics expert policy me from given demonstrations without requiring rewards from the environment. Its objective can be expressed analogously equation 2
arg min (TT) D(n, mE).
We distinguish two IL settings: IL from demonstration (ILiD) and IL from observation (ILfO). In ILfD, demonstrations are provided sequences of {Sp, ag...., ST -r a, ~ Te (-),,, st41 ~ p(-]s;, o.)). In contrast, in ILfO, demonstrations consist of only state sequences. Due the lack of information about the expert's actions, II.0O is
pairs of length T: TB
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Behavior Imitation Attack (BIA) in GAIL
Input: victim's policy и, initial adversarial policy ng, initial discriminator Dg. demonstration Tigh Batch size B, learning rates ap, Byy
2: Output: Optimized adversarial policy Vy
3: for 0, 0,1, do
4: I T+
5:
for b = to B do
6:
8 + Project(vg)s)) B(s)
7:
(-11)) s p(-)s,a)
8:
T - T 8, a, }
9:
end for
10:
Φ - o ap V (s,a)rr log Ds(s,a) -E log(1 - (s)s)
11:
0 - + a Vo log D (s)
12: end for
known to be empirically more challenging than ILfD. In our attack setting, the adversary has access to demonstration Titt generated by the target policy Teg-
By using IL algorithms that satisfy the assumptions of Theorem we can carry out attacks even under black-box and no-box settings. Specifically, many IL algorithms rely divergences that admit a variational representation. Thus, by reformulating the adversary's 's objective using Theorem 5.1 we can leverage these well-established methods train the adversarial policy.
As concrete example of an IL algorithm that holds the assumption, we illustrate how to apply BIA GAIL(Ho & Ermon 2016), representative ILfD method. In GAIL, the adversary's objective
function can be rewritten in the following variational form using discriminator D: S x A - [0, 1]:
arg min D(T Mugs) max E (log D(s, + [ogg D(s,a))) V
(8)
The second term is computed with demonstration Ty. instead of Figi practice. In Appendix C we show that under our problem setting, the adversary's 's objective function can be reduced to distribution matching problem. Consequently, we confirm that equation holld exactly. By Theorem 5.1 this
then reduces to the following optimization problem:
arg max E [Ap(s,a)]
where RD is the function obtained by replacing the discriminator g(d)) with log D, in equation 4 This is a standard RL problem and can be solved using any RL algorithm. In Appendix [ we further show how BIA extends other approaches, including ILfO (Torabi et al. 2019) and the state-of-the-art ILfD algorithm (Chang et al. 2024).
We present practical algorithm for BIA that utilizes GAIL to train adversarial policy in Algo- rithm Since It in equation represents the conditional expectation of (s, for the adversarial policy, we simply approximate it by a single sample g(d, (s, a, s')) = log D(s, a) at each there no discrepancy between
step. This approximation preserves exactness in expectation, Theorem 5. and Algorithm|] I As our approach does not require direct access to the victim's policy, BIA can be applied in black-box setting. Furthermore, when II.OO is used, the discriminator D only needs states and the next states to train the adversarial policy. Since it does not require the victim's 's policy's ' selected actions, our method is applicable even in a no-box setting. Also, to stabilize the learning process, avoid imposing large negative rewards; instead, we limit the range of false states, ensuring that the adversarial policy is enforced to select falsified states in B(s) in our experiments.
DEFENSE METHOD
In this section, we propose a countermeasure against behavior targeted attacks. In this attack, the adversary's objective is defined independently of the victim's 's reward. As a result, the victim can be steered toward unintended and potentially malicious behaviors even when the victim's cumulative reward does not decrease. Therefore, existing defenses that aim prevent reward degradation may

---

Published as a conference paper at ICLR 2026
fail to provide sufficient robustness under this threat. To address this limitation, we incorporate the adversary's 's gain as a safety metric.
Formulation of defender's objective. Let Regi: S x A R be the reward function that is maximized when the victim's 's behavior exactly matches the adversary's 's specification. We then formulate the defender's 's objective follows:
arg min Jar((T) - JRI (T) + A mmax Ey (Bgi a)] - Ex [Ree a)l) a
(10)
where À is a hyperparameter that adjusts the off between performance and robustness. Rig represents the adversary's 's gain, and the second term represents the additional gain achievable by the worst-case adversarial policy. Thus, equation seeks minimize this worst-case increase in the adversary's 's gain. We note that equation I0 subsumes prior defense objectives against reward- makes the second term
minimization attacks via the choice Rigt. In particular, setting figt regularizer that suppresses reward drops under attack, yielding a standard robustness objective.
Since the defender cannot access the adversary's intended behavior, equation 10 cannot be optimized directly. In particular, the adversary's 's objective is specified independently of the victim's 's reward, so defender cannot compute the target reward Rigt. Consequently, the g required to optimize equation T0 are unavailable.
Wiime-Discounee Robust Training (TDRT). To optimize equation 1O we prove that the increase in the cumulative reward from the reward function Rigi in equation T0is bounded by the sensitivity of the policy's 's action outputs to state changes:
Theorem 6.1. Let Reg: S x A - TR be the adversary's 's objective reward function, and the discount factor be E (0. Assume that there exists an upper bound Rigt E R for Rig. Then:
2
[M [Rgr(s, a)] EM (ggg(s,
a)))
Esnde [Duu(T(( o-
(11)
where (s) Pr(st s|=) represents the state distribution of time t.
The proof is provided in Appendix B.2 Theorem 6.1 limplies that the smaller the sensitivity of the policy's 's action outputs to state changes, the smaller the increase in cumulative reward under attack. Thus, policy smoothing is an effective defense not only against reward- minimization attacks but also against behavior-targeted attacks. Furthermore, the theorem reveals a key insight unique to defending against behavior-targeted attacks: the sensitivity of action outputs the early stages of trajectories has greater influence on the victim's overall defense performance. Therefore, reducing early-stage action sensitivity further strengthens robustness against teraeeer-targeted attacks.
Based on Theorem 6.1 we propose robust training framework, -iscouuedd Robust Training (TDRT). Let B = (st, } state 8, observed. The defender's objective is redefined as:
be a mini-batch of state-time pairs, where is the timestep at which
Jae((=) = (m) + À max
Σ D'a (m(- u(-())))) B
(12)
The complete training procedure, implemented with PPO. is given in Algorithm 2
In our implemen- tation, the timestep / enters only through the discount factor 7² and is not included in the policy input. Furthermore, since direct calculation of the KI. term computationally expensive, we apply convex relaxation methods to obtain tight upper bounds (see Appendix in details).
The primary advantage of time-discounting to suppress performance degradation on the victim's original task. Existing uniform smoothing methods tend to overly restrict the policy's expressiveness in the later stages of a trajectory to achieve sufficient robustness. In contrast, TDRT concentrates the effect of regularization at the early stages by applying time- dime-dicounntng. Consequently. TDRT achieves sufficient robustness while preserving the policy' 's expressive capacity, thereby maintaining high performance the victim's original task.
7

---

Published as a conference paper at ICLR 2026
7 EXPERIMENTS
We empirically evaluate our proposed method on World (Yu ct al. 2020) (continuous action spaces), MuJoCo (Todorov et al. 2012) (continuous action spaces), and MiniGrid (Chevalier-Boisverl et al., 2018) (discrete action spaces, vision-base control). Due to space limitations, this section only reports results for the Meta- World environment. Other experiments can be found in Appendix
We focus on three key aspects: (i) Attack performance: Can the adversarial policy learned through BIA effectively manipulate the victim's behavior? (ii) Robustness: Does the policy smoothing in TDRT offer greater robustness against behavior-targeted attacks compared to existing defense methods such as adversarial training? (iii) Original task performance: Can the time-discounting component of TDRT mitigate performance degradation on the original tasks compared to traditional policy smoothing without time discounting?
Set up. As illustrated in the scenarios in Section ! we suppose an adversary aims to force a victim to perform an adversary's task that is entirely different from the victim's original task. Accordingly, we set the adversary's is objective to force the victim to execute an adversary's is task that is opposite to the one the victim originally learned. In the following, we refer to the reward function in the adversary's 's target task as the adversary's 's reward function and the reward function in the victim's original task as the victim's 's reward function.
We consider five opposing task pairs: (window-close, window-open), (drawer-close, drawer-open), (faucet-close, faucet-open), (handle- prendleerees-sdde. handle-pull-ee -uille- and [door-lock, door-unlock]. For example, if the victim originally learned the window-close task, the target policy is defined policy that completes the window-open task. Since the reward structures of the Meta- World benchmark are relatively complex, the adversary's reward function (e.g., reward function of window-open) cannot be obtained by simply negating the victim's 's reward function (e.g., reward function of window-close). In this sense, forcing victim agent trained to perform window-elose execute window-open cannot be achieved through a reward-minimization attack but requires behavior-targeted attack.
Sun al. 2022), we constrain the set of adversarial c), where represents the attack budget. All experiments are performed with e = 0.3. G.2 for the results of changing the attack budgets.
Following the prior works (Zhang al. 2021
states (s) using the Tix norm: (ss) {8
8
States are standardized across all tasks, with standardization coefficients calculated during the training of the victim agent. To learn adversarial policies with BIA, we employed DAC (Kostrikov et al. 2018) of ILfD and OPOLO (Zhu et al. 2020) for ILfO as IL algorithms, which are variants of GAIL. In BIA, the demonstrations consist of 20 episodes of trajectories generated by the target policy, which is fully trained on the adversary's s target task, We also vary the number of demonstrations in Appendix G.I The results show that performance changes little even when the number of episodes is reduced to four. Appendix G.3 also provides a comparison of attack performance across different target policies.
Attack Baselines. We compare our proposed attack method with three baselines. (i) Random Attack: This attack perturbs the victim's 's state observation by random noise drawn from uniform distribution. This attack works with no-box access and requires no knowledge about the vietim. (ii) Targeted PGD Attack: This naive attack method optimizes falsified states é using PGD at each time independently to align the victim's 'ss actions with those of the target policy's 's at each time: 3 arg min, dd(r(-18), (-s)) PGD requires white-box access to the victim's 's policy, giving the adversary an advantage not available in our proposed method. The detailed explanation and analysis of Targeted PGD are provided in Appendix 1.4.2 (iii) Target Reward Maximization Attack: This attack leverages Lemma B.I to learn an adversarial policy that maximizes the cumulative reward obtained by the victim from the adversary's 's reward function Raay: D² = arg max (Ratv (8, a, 8')). We used two methods for training the adversarial policy: SA-RL SA-RL et al. 2021) (black-box attack) and PA-AD et al., 2022) (white-box attack). These attacks require access to the adversary's reward function. Thus, they give the adversary an advantage not available in our proposed method.
You
Defense methods against behavior behavior-targeted attacks have not yet been proposed. Therefore, as baselines, employ defense methods against untargeted attacks: ATLA-PPO et al. 202I), et al. 2022). and (A-PPPO(hag et al. 2020b). ATLAA-POO and PA PA-ATLA-PP are methods
Defense Baselines.
et al.) (002). LAAD-POOBellarr et al. 2024). WocaR PPO
8

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of attack performances. Each value represents the average episode reward I standard deviation over 50 episodes. Each parenthesis indicates (Access model, Adversary' 's knowledge). Under the lmiteedkknwweede setting, BIA's attack performance is competitive with that of baseline methods that assume greater adversary knowledge.
Adv Tosk
Target Reward
Adversary with full lnnollegge
Adversary with limited knowledge
Targeted PGD
Rew Max (PA-AD) (elor mucer for lano
Rew Max (SA-RL
BIA ILID (ours)
BIA-ILIO (ours)
Random
Coline hur, lrigh (ulgg)
(J4ck lnc. condeciioo)
window-clss
4543
1666 2: 106
4255 2 300
4505 2: 65
3062 665
4036 2: 310
947 529
window opea
4508 121
515 421
453 542
506 644
566 2: 523
357 (7)
322
361
456h 11
1861 199
4638-11
1089
953
1697
1499
close
124)
roncet open
2718
preo-side
handle pull
doos
640
765
2043
1906
deuemmleek
4690
551
3395 1111
3421 911
1536 01
5123
301
that utilize adversarial training. RAD-PPO is a regret-based defense method that learns policies to minimize regret, defined as the difference between the rewards under non-attack and attack conditions. WocaR-PPO is a defense method that learns policies to maximize the worst-case rewards and applies regularization to the smoothness of policies, specifically in critical states where rewards significantly decrease. SA-PPO aims to increase the smoothness of the policy's ss action outputs by a regularizer. The difference between TDRT-PPO and SA-PPO is that SA-PPO does not apply time discounting in the regularization. For detailed explanations of the baselines, see Appendix 1.2
Attack Performance Comparison. We present the results in Table The attack rewards represent the cumulative reward of the adversary's task obtained by the victim under attacks. The target rewards are the cumulative reward obtained by the target policy, which is directly trained with the adversary's s reward function, serving the upper bound for attack rewards. We also report attack success rates in Table 10 by using the task-success criterion in Meta-World These results exhibit the same trend.
BIA achieves an attack performance comparable more advantaged attacks, such Target Reward Maximization (Rew Max/SA-RI, PA-AD), which has access to the adversary's reward function. This demonstrates that BIA can effectively attack using demonstrations alone, without requiring any reward modeling of the adversary's 's task. The attack performance of BIA-ILfO is nearly identical to that of BIA-ILID. We attribute this to the deterministic nature of state transitions in the tasks: without access to the victim's 's actions, the transitions can be sufficiently predicted. While we used 20 episodes as demonstrations for BIA, results in Appendix G.I confirm that the attack performance remains nearly unchanged even when the demonstrations are reduced to four episodes.
The targeted PGD attack achieves significantly lower attack rewards than BIA across all tasks. This is because targeted PGD) optimizes adversarial perturbations independently at each time step without accounting for future decisions. Consequently, under a limited e, targeted PGD fails to achieve sufficient attack effectiveness. Our analysis revealed that, even after PGD optimization, the loss remains high at multiple states along a trajectory.
All attack methods exhibit relatively low attack performance on the window-open, opener-open, and door-lock tasks. This can be attributed to the greater disparity in the sate-action distributions between the victim's 's policy and the target policy in these tasks compared to others. In such cases, their behaviors differ so significantly that orchestrating a successful attack becomes extremely difficult.
Defense Performance Comparison. The results appear in Table 2 The best attack rewards refer to the largest attack reward obtained by the victim among the six attacks listed in Table T lower values indicate greater robustness. Complete results for all attack methods are reported in Table 12
In all tasks, TDRT-PPO and SA-PPO achieve superior robustness against attacks, indicating that policy smoothing is effective against attains exceptionally high robustness, albeit with a substantial drop in original task performance (see Table Additional experiments in Appendix|H.. ex.mmmiineeeeeeeeeeeeeeeeeee examine how varying the smoothing coefficient trades off robustness and original task performance.
targettd attacks. For the drawer-close task, SA-PPO
Adversarial training methods, including ATLA-PPO and PA-ATLA- -P-ATL--PP. are still vulnerable to behavior-targeted attacks in most tasks. As noted by (Korkmaz 2021 2023 adversarial training is

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of robustness. Each value represents the average episode reward + standard deviation over 50 episodes. Policy smoothing is very effective against behavior-targeted attacks.
Attack Revard (U)
PPO (No defense)
ATLA-PRO (AdvTraning)
PA-ATIA-PPR (AdVIriinng)
RAD-PPO (Regret)
WoraR-PPO (Parial smoothing)
SA-PPO Smoothing time dissoniisonnning
TDRT-PPO (murs)
Smoothing diseeriinnnniii
windon-close
4505 + 65
4270 + 188
4041 1 96
+261 2: 306
575 + 135
485 61
482 1:
Window open
566
525
586 649
671 2: 589
501 1: 132
295 1:15
272 2: 37
254 :: 214
deseer-cleee
4760 640
4855 =
4868
4588 9225
4967
4 4=
4860 +:
deener--oen
1556 607
1158 1026
954 219
736
579 1:15
49
378
4108 +:
123
522
2829
1559 406
610
2358== 0080-0066
1763 255
361
4302
1169
532
33.1.6
lack
1229
992
392
562
dovwwallack
1365
2806 =: 2743 +:
1073:::
=
=
effective only against the vulnerable to behavior-targeted attacks. Although the regret-based approach RAD-PPO demonstrates relatively higher robustness in some tasks than adversarial training, it still relies reward-based metrics and, therefore, does not offer sufficient protection against -atrgeeee attacks. WocaR- PPO, which applies smoothing SA-PPO and TDRT-PPO, both of which uniformly smooth across the entire state space. Further experiments on robust training efficiency are presented in Appendix HH.3
attack anticipated during training. Thus, it remains
subset of states, achieves moderate robustness but underperforms
Original-Tass Task Performance Comparison. The results appear in Table 3 The clean rewards represent the reward obtained by the victim in its original task without at- tacks; higher values indicate less perfor- mance degradation due to robust training.
Table 3: Clean rewards comparison: TDRT-PPO vs. SA-PPO. Time-discounting greatly improves the performance on the original task.
Clean Reward (1)
SA-PPO
TDRI-PPO (ours)
Task
TDRT-PPO achieves higher clean rewards than SA-PPO in all tasks. Together with the results in Table this demonstrates that while uniform regularization across the en- tire trajectory in SA-PPO sacrifices original performance, time discounting in TDRT-PPO mitigates performance degradation while pre- serving robustness. In Appendix H22 we further study the effect of the regularization coefficient. Across wide range of coef- ficients, TDRT-PPO consistently achieves higher clean rewards than SA-PPO.
(wl time discounting)
(w/ time discounting)
window-close
4367 1 107
4412 : 55 1.0%)
window open
4092
57 7.1%)
drawer-close
2156 + 453
4237 + (3 96.5%)
drawer open faucet-close faucet-open
4161 :: 1537
4802 1 27 15.4%)
4304 + 42
4740 17 10.1%)
4380 + 43
4630 = 11 5.7%)
handle-press-side
3226 806
4321 + 215 33.9%)
handle pull side
4094 1: 350
4468 1: 126 (1 9.1%)
door-lock
1491
(1 20.4%)
door-unlock
2017 1 497
3680 :: 290 (T 82.5%)
avg. improvement
28.2%
CONCLUSION AND LIMITATIONS
This work introduces the Behavior Imitation Attack (BIA), which manipulates victim behavior through perturbed state observations under limited access to the victim's policy. We also introduced time-Discounted Robust Training (TDRT), the first defense method specifically designed for behavior- targeted attacks, which achieved robustness without compromising original performance.
Limitation. While our work contributes valuable insights, it also has limitations. Adversarial policy is known to be less effective in nigh-dimensional state spaces, such image inputs (Sun et al.) 2022) (see Appendix F.2 Efforts to overcome this require white-box access (Sun et al. 2022 and remain unresolved. Moreover, although TDRT empirically exhibits robustness against behavior-targeted attacks, it lacks certified guarantees. In scenarios where higher reliability requirements are imposed, TDRT may prove insufficient.
10

---

Published as a conference paper at ICLR 2026
9
ACKNOWLEDGEMENTS
This work is partially supported by JST JPMJNX25C2, JPMUKP24C3, JPMJCR2344, JPMJCR2IDS ISPS 23H00483, and 120251002. We gratefully acknowledge the insightful comments and sugges- tions provided by the anonymous reviewers.
10 ETHICS STATEMENTS
This paper focused adversarial attacks reinforcement learning and their defense methods, aiming to improve the reliability of deep reinforcement learning. Our contribution lies in proposing attack methods and defenses that envisioned for real-world scenarios. However, proposed methods still face challenges in practical real-world applications. Our research fully complies with legal and ethical standards, and there are no conflicts of interest. Throughout this study, we utilized only publicly available benchmarks. No private datasets were used in this research.
We have provided detailed algorithms for our proposed methods. The benchmarks used our experiments are public and accessible to everyone. As described above, we have made substantial efforts to ensure the reproducibility of our results.
REFERENCES
AV Arkhangel'skis ski and Vitaly Fedorchuk. The basic concepts and constructions of general topology. In General Topology I: Basic Concepts and Constructions Dimension Theory, pp. 1-90. Springer, 1990.
ski
Fengshuo Bai, Runze Liu, Yali Du, Ying Wen, and Yaodong Yang. BATTLE: Towards behavior- oriented adversarial attacks against deep reinforcement learning, 2024. URL https: openreview wwt.://oooommdddopprrriiiee.
Fengshuo Bai, Runze Liu, Yali Du, Ying Wen, and Yaodong Yang. Rat: Adversarial attacks on deep reinforcement agents for targeted behaviors. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 15453-15461, 2025.
Roman Belaire, Pradeep Varakantham, Thanh Nguyen, and David Lo. Regret-based defense in adversarial reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 2633-2640, 2024.
Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning. Advances in Neural Information Processing Systems, 36, 2024.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06682 2019.
Adith Boloor, Karthik Garimella, Xin He, Christopher Gill, Yevgeniy Vorobeychik, and Xuan Zhang. Attacking vision- based perception end-to-end autonomous driving models. Journal of Systems Architecture, 100:101766, 2020.
Prasanth Buddareddygari, Travis Zhang, Yezhou Yang, and Yi Ren. Targeted attack on deep rl-based autonomous driving with learned visual patterns. In 2022 International Conference Roboties and Automation (ICRA), pp. 10571-10577. IEEE, 2022.
Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo, Chao Zhang, Songan Zhang, and Tuo Zhao. Robust multi -agti--aget reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms. Advances in Neural Information Processing Systems, 36, 2024.
Jonathan Daniel Chang, Dhruv Sreenivas. Yingbing Huang, Kianté Brantley, and Wen Sun. Ad- versarial imitation learning via boosting. In The Twelfth International Conference on Learning Representations, 2024. http://eeeeeeeeeeeeeeeeeeeeerrrr

---

Published as a conference paper at ICLR 2026
Xiaocong Chen, Lina Yao, Julian McAuley, Guanglin Zhou, and Xianzhi Wang. Deep reinforcement learning in recommender systems: A survey and new perspectives. Knowledge-Bor Based Systems, 264: 1 10335, 2023.
Maxime Chevelier-Boiseert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: / htt/ttttu...... thub. com max me cb/ /gmmm/ymmmmmmmm mi minigrid 2018.
Jing Cui, Yufei Han, Yuzhe Ma, Jianbin Jiao, and Junge Zhang. Badrl: Sparse targeted backdoor attack against reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11687-11694, 2024.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 402(7897):414-44 419, 2022.
Tianyang Duan, Zongyuan Zhang, Zheng Lin, Yue Gao, Ling Xiong, Yong Cui, Hongbin Liang, Xianhao Chen, Heming Cui, and Dong Huang. Rethinking adversarial attacks in reinforcement learning from policy distribution perspective. arXiv preprint vrXiv:2501.03562. 2025.
X Fernique, PW Millar, DW Stroock, M Weber, and Warwick Millar. The minimax principle in asymptotic statistical theory. In Ecole d' Eté de Probabilités de Saint-Flour XI-1981, pp. 75-265. Springer, 1983.
Mingsheng Fu, Liwei Huang, Ananya Rao, Athirai A Irissappane, Jie Zhang, and Hong Qu. deep reinforcement learning recommender system with multiple policies for recommendations. IEEE 2022.
Transactions on Industrial Informatics, 19(2):2049-200
A Gleave, M Dennis, N Kant, C Wild, S Levine, and S Russsell. Adversarial policies: Attacking deep reinforcement learning. In Proc. ICLR-20, 2020.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXiv:1412.633: 2014.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural
information processing systems, 29, 2016.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. International Conference on Learning Representations, 2017.
Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Copycat: Taking control of neural policies with constant attacks. In AAMAS 2020-1990 International Conference on Autonomous Agents and Multi-Agent Systems, 2020.
Peter Jin, Kurt Keutzer, and Sergey Levine. Regret minimization for partially observable deep reinforcement learning. In International conference on machine learning, pp. 2018.
PMLR,
Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li. Trojdrl: evaluation of backdoor attacks on deep reinforcement learning. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2020.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909 4926,, 2021.
Ezgi Korkmaz. Investigating vulnerabilities of deep neural policies. In Uncertainty in Artificial Intelligence, pp. 1661-1670. PMLR, 2021.
Ezgi Korkmaz. Adversarial robust deep reinforcement learning requires redefining robustness. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 8369-8377, 2023.
12

---

Published as a conference paper at ICLR 2026
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson.
peeriminattoractoo-ciiiie Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference Learning Representations, 2018.
A Kumar, A Levine, and S Feizi. Policy smoothing for provably robust reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
Haoran Li, Zicheng Zhang, Wang Luo, Congying Han, Yudong Hu, Tiande Guo, and Shichen Liao. Towards optimal adversarial robust q-learning with bellman infinity-error. In Proceedings of the 41st International Conference Machine Learning, pp. 29324-29372, 2024a.
Jianhui Li, Bokang Zhang, and Junfeng Wu. Online poisoning attack against reinforcement learning 2024b.
under black-box environments. arXiv preprint
Yongyuan Liang, Yanchao Sun, Ruijic Zheng, and Furong Huang. Efficient adversarial training with- out attacking: Worst-case-awant aware robust reinforcement learning. Advances in Neural Information Processing Systems, 35:22547- 2022.
Yongyuan Liang, Yanchao Sun, Ruijie Zheng, Xiangyu Liu, Benjamin Eysenbach, Tuomas Sandholm, Furong Huang, and Stephen Marcus McAleer. Game-theoretic robust reinforcement learning han- dles camporallycouplees perturbations. In International Conference on Learning Representations, 2024.
Con-Chen Lin, Zhang- Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. In Proceedings of the Twenty- Sixth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, 2017.
Xiangyu Liu, Chenghao Deng, Yanchao Sun, Yongyuan Liang, and Furong Huang. Beyond worst-case attacks: Robust rl with adaptive defense via non-dominated policies. In International Conference Learning Representations, 2024.
Jeremy McMahan, Young Wu, Xiaojin Zhu, and Qiaomin Xie. Optimal attack and defense for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 14332-14340, 2024.
Ronghui Mu, Leandro Soriano Marcolino, Yanghao Zhang, Tianle Zhang, Xiaowei Huang, and Wenjie Ruan. Reward certification for policy smoothed reinforcement learning, In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 21429 21437, 2024.
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui-Wei Weng. Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Processing Systems, 34:26156-261666 2021.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In 17th International Conference on Au- tonomous Agents and Multiagent Systems, AAMAS 2018, pp. 2040-2042. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2018.
You Qiaoben, Chengyang Ying, Xinning Zhou, Hang Su, Jun Zhu, and Bo Zhang. Understanding adversarial attacks observations in deep reinforcement learning. Science China Information Sciences, 67(5):1 2024.
Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time time adversarial attacks against reinforcement learning. In International Conference Machine Learning, pp. 7974-7984. PMLR, 2020.
13

---

Published as a conference paper at ICLR 2026
Anshuka Rangi, Haifeng Xu, Long Than,,hanh, and Massimo Franceschetti. Understanding the limits of poisoning attacks in episodic reinforcement learning. In Proceedings of the 31 International Joint Conference on Artificial Intelligence (IJCAI 2022). pp.
IJCAL, 2022.
Ethan Rathbun, Christopher Amato, and Alina Oprea. Sleepernets: Universal backdoor poisoning
attacks against reinforcement learning agents. In The Thirty-iighth Anmual Conference on Neural Information Processing Systems, 2024. URL. https: https://openreww /wwp/pppeeeeeeeeeeoeee ew net. forum? id=
RkC40Yee3g
Marc Rigter, Brumo Lacerda, and Nick Hawes. Minimax regret optimisation for robust planning in uncertain markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1998- 2021.
Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep reinforcement 2017.
learning framework for autonomous driving. arXiv preprint
Hadi Salman, Greg Yang, Huan Zhang, Cho-Jui Hsich, and Pengchuan Zhang. A convex relaxation barrier to tight robustness verification of neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Lucas Schott, Josephine Delas, Hatem Hajri, Elies Gherbi, Reda Yaich, Nora Boulahia Boulahia-Cuppens. Frederic Cuppens, and Sylvain Lamprier. Robust deep reinforcement learning through adversarial attacks and training: A survey. arXiv preprint arXiv:2403.0042) 2024.
Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pp. 8707-8718. PMLR, 2020.
Chung-En Sun, Sicun Gao, and Tsui- Wei Weng. Breaking the barrier: Enhanced utility and robustness in smoothed DRL agents. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 46957-46987. PMLR, 21-27 Jul 2024.
Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, and Yang Liu. Stealthy and efficient adversarial attacks against deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, pp. 5883-5891, 2020.
Xiaolin Sun and Zizhan Zheng. Belief-enriched pessimistic q-learning against adversarial state perturbations. In The Twelfth International Conference Learning Representations, 2024. URL htttp:://operrett ew net http///oouummim///ppddoeeem
Yanchao Sun, Da Huo, and Furong Huang. Vulnerability-wware poisoning mechanism for online rl with unknown dynamics. In International Conference on Learning Representations, 2021.
Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl. In The Tenth International Conference on Learning Representations (ICLR 2022), 2022.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEFF, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. In ICMI. Workshop on Imitation, Intent, and Interaction (13), 2019.
Edgar Tretschk, Seong Joon Oh, and Mario Fritz. Sequential attacks agents for long- adversarial goals. In 2. ACM Computer Science in Cars Symposium, 2018.
Derui Wang. Kristen Moore, Diksha Goel, Minjune Kim, Gang Li, Yang Li, Robin Doss, Minhui Xue, Bo Li, Seyit Camtepe, et al. Camp in the odyssey: Provably robust reinforcement learning with certified radius maximization. arXiv preprint Xiv::5001.1666 2025.
14

---

Published as a conference paper at ICLR 2026
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning, pp. 0285552869999 PMLR, 2018.
Fan Wu, Linyi Li, Zijian Huang, Yevgeniy Vorobeychik, Ding Zhao, and Bo Li. Crop: Certifying robust policies for reinforcement learning through functional smoothing. In 10th International Conference on Learning Representations, ICLR 2022, 2022.
Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai- Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Prucessing Systems, 33, 2020.
Yinglun Xu and Gagandeep Singh. Black-box targeted reward poisoning attack against online deep reinforcement learning. arXiv preprint arXiv:2305.. 10681, 2023.
Yinglun Xu, Qi Zeng, and Gagandeep Singh. Efficient reward poisoning attacks on online deep reinforcement learning. Transactions on Machine Learning Research, 2023.
Zhihe YANG and Yunjian Xu. DMBP: Diffusion model-based predictor for robust offline rein-
forcement learning against state observation perturbations. The Twelfth International Confer- ence Learning Representations, 2024. URL
2ULjCYL
Chengyang Ying, You Qiaoben, Xinning Zhou, Hang Su, Wenbo Ding, and Jianyong Ai. Consistent attack: Universal adversarial perturbation embodied vision navigation. Pattern Recognition Letters, 168:57-63, 2023.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey, ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
Tianhe Yu, Deirdre Quillen, Zhampeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-woot world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100 PMLR, 2020.
Huan Zhang, Tsui-Wei Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. Advances in neural information processing systems, 31, 2018.
Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho Jui Hsich. Towards stable and efficient training of verifiably robust neural networks. In 8th International Conference Learning Representations, ICLR 2020, 2020a.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:2 1224-22241 2020b.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsich. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representation (ICLR), 2021.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. 2020.
Advances in neural information processing systems,
15

---

Published as a conference paper at ICLR 2026
A
RELATED WORKS
In this section, we discuss prior works on attacks and defenses on DRL, which primarily consider an adversary that perturbs the victim's 's state observations.
A.
ATTACK METHOSS
We classify existing attack methods on fixed reinforcement learning agents into three categories: the wwaardmmiiimization attack, the enchanting attack, and the behavior- ttrreee attack. Furthermore, discuss the poisoning attacks that occur during the victim's 's training phase.
attack. In reward-minimizationn attacks, the adversary's i goal is to minimize the cumulative reward received by the victim. The most basic approaches gradient- gradient-based methods. Huang et al. (2017) proposed an attack method that uses the Fast Gradient Signed Method (FGSM) (Goodfellow al. 2014) compute adversarial perturbations that prevent the victim from choosing optimal actions. Pattanaik et al. (2018) performed a more powerful attack by creating adversarial perturbations to minimize the value estimated by the Q-function. Gleave et al. (2020) proposed an attack that assumes a two-agent environment, This method creates an ad adversarial agent that severely degrades the victim's 's performance. [Sun et al. (2020) point out that previous works lack stealth and proposed the Critical Point Attack and Antagonist Attack, which achieve effective attacks within very few steps. Qiaoben et al.. (2024) classify existing adversarial attacks against RL agents in the function space and propose
attack method based two-stage optimization derived from the theoretical analyses. (Sun et al. (2020) and Qiaoben et al. (004)) are similar to ours in that they attempt to bring the victim's 's policy closer to specific policy. However, these objectives differ from ours, and they implemented via PGD attack that requires white-box access Duan et al. (2025) proposed an attack that targets the distribution of the victim's 's policy. However, their method is reward- minaadddiiiizzaiion attack and differs from ours. Specifically, they intended to induce large deviations from the original victim's 's distribution and indirectly reduce the victim's 's cumulative reward. Because their goal is not to shift the policy toward any specific target distribution, their attack differs from the behavior-targeted attack.
Enchanting attack. The enchanting attack aims to lure the victim agent to reach predetermined target state. [in et al. (2017) first proposed this type of attack. In their approach, the adversary generates a sequence of states and actions that cause the victim to reach the target state. The adversary then crafts sequence of perturbations to make the victim perform the sequence of actions. Tretschk jet al. (2018) proposed an enchanting attack where adversarial perturbations to maximize adversarial rewards are heuristically designed for the attack purpose, thereby leading the victim to the specified target state. Buddarreddygari et al. (2022) proposed a different enchanting attack using visual patterns placed physical objects in the environment so that the victim agent is directed to the target state. Unlike (Lin et al. 2017) and (Tretschk et al., 2018), which perturb the victim's 's state observations, this attack alters the environmental dynamics. Ying et al. (2023) proposed an enchanting attack with a universal adversarial perturbation. When any state observations are modified with this perturbation, the victim agent is forced be guided to the target state. All of these enchanting attacks require white-box access to the victim's 's policy.
Behavior-targeted attack. The behavior-targetee attack aims to manipulate not only the final destination but also the victim's 's behavior in more detailed manner. Hussenot et al. (2020) proposed a behavior-targeted attack that forces the victim to select the same actions as the policy specified by the adversary. More specifically, this attack precomputes a universal perturbation for each action so that the victim who observes the perturbed states takes the same action as the ad adversarially specified policy. One limitation of this attack is that the computational cost of precomputing such universal perturbations can be high when the action space is large or continuous. Since the cost required for pre-computing perturbations significant, applying this attack is challenging. [Blooor et al. (2020) proposed heuristic attack specifically designed for autonomous vehicles. They formulated an objective function with a detailed knowledge of the target task, which is not applicable to behavior- targeted attacks for general tasks. Bai et al. (2025) 2024) investigated victim's 's behavior that follows the adversary ' preference for the behavior. This attack method can be applied to any environment. However, one limitation of this method is that it usually requires
attack to manipulate the
16

---

Published as a conference paper at ICLR 2026
thousands of labeled preference state-action sequences to specify the behavior that the adversary requests to follow. Therefore, this attack is not practical. We remark that all existing behavior-targeted attacks also require white-box access to the victim's 'ss policy.
Poisoning attack. Several studies have proposed poisoning attacks against RL agents, which intervene during the victim's 's training phase. Kiourti et al. (2020) introduced TrojDRL, a framework for evaluating backdoor attacks in DRL, where the adversary intercepts and manipulates states during training to steer the agent toward a predefined target policy. Sun et al. 2021) proposed VA2C-P that adapts without access to environment dynamics and supports both untargeted attacks and targeted attacks that align the victim's 's policy with a target policy. Rakhsha et al. (2020) theoretically analyzed poisoning attacks in both offline planning and online RL with tabular MDPs, while [Rangi al.) (2022) further clarified the fundamental limits of poisoning attacks in episodic reinforcement learning. Other works (Xu al. 2023 Xu & Singh 2023 Li et al.] 2024b) developed more practical black- box poisoning attacks that do not require knowledge of the environment dynamics or the victim's s learning algorithm. More recently, Cui et al. (2024) proposed BadRL, sparse poisoning attack that dynamically generates sample-specing triggers and injects them only at high attack-valt value states. All of these methods aim to steer the victim's policy toward an attacker-specified target policy. However, our threat model explicitly prohibits the adversary from accessing or intervening in the victim's ' training process, whereas poisoning attacks inherently rely on such access. Consequently, these attack settings fall outside the scope of our problem formulation.
A.2 DEFENSE METHODS
One approach to learning robust policies against reward-minimizztion attacks policy smoothing. Shen et al. (2020) introduced regularization term to smooth the policy and demonstrated increased sample efficiency and robustness. Zhang et al. (2020b) formulated the Adversaaalaa Markov Decision Process (SA-MDP) to represent situations where an adversary interferes with the victim's state observations. Based on SA-MDP, they showed that regularization to smooth the policy is effective in resisting wwward-miiimization attacks. However, their theoretical robustness guarantees are limited to reward-minimization attacks and not directly extended to behavior-targeted attacks. Furthermore, they often degrade performance on the victim's original tasks, as it imposes excessive constraints on the policy's representational capacity.
Another approach for defense adversarial training. Zhang et al. (2021) showed that the optimal adversarial policy can be learned as policy in MDP and proposed ATLA, an adversarial training framework that exploits this insight. [Sun et al. 2022) proposed a theoretically optimal attack method that finds the optimal direction of perturbation and proposed PA-ATLA as an extension of ATLA, which is efficient even in large state spaces. Oikarinen al. (202T) proposed framework for training robust RL by incorporating an adversarial loss that accounts for the worst-case input perturbations. Additionally, they introduced a new metric to efficiently evaluate the robustness of the victim. Liang al. (2022) efficiently estimated the lower bound of cumulative rewards under adversarial attacks and performed adversarial learning with partial smoothness regularization. [i et al. (2024a) theoretically motivate minimizing the Bellman infinity-error in adversaavaaaaaal MDPs and propose CAR-DON as a robust value-based algorithm under reward-minimization attacks. However, Korkmaz) (2021) (023) pointed out that the victims trained by adversarial training are still vulnerable to attacks that were not anticipated during training.
McMahan et al. 2024) proposed comprehensive framework for computing optimal attacks and defenses, modeling the attack problem as a meta-MDP and the defense problem as a partially observable turn-based stochastic game. BBkkharin et al. (2024) proposed robust Multi-Agent RL (MARL) framework that uses adversarial regularization to promote Lipschitz continuity of policies, thereby enhancing robustness against environmental changes, observation noise, and malicious agent actions. [Liag et al.) (2024) introduced a new concept called temporally-coupled perturbations, where consecutive perturbations are constrained. Their proposed method, GRAD, demonstrates strong robustness against standard and temporally-coupled perturbations. Some works (Jin et al.) 2018 Rigter et al. 2021 Belaire et al. 2024) proposed a novel defense method based on regret minimization. Additionally, another approach in robust reinforcement learning is certified defense (Wu et al., 2022: Kumar et al. 2022, Mu et al. 2024] Sun et al.) 2024] Wang et al. (025). These studies guarantee lower bound the rewards obtained by the vietim under adversarial attacks. Liu
17

---

Published as a conference paper at ICLR 2026
al. 2024) proposed adaptive defense method robust against multiple types of attacks, not limited to worst-case reward-minimine minimization. their method, the defender prepares multiple robust policies in advance and then selects the policy that maximizes the victim's 's reward under attack based on rewards obtained previous episodes. However, our approach focuses on a single-agent RI. problem, as the adversary usually targets single static victim in stationary environment. As result, their adaptive setting differs from ours.
Recent works (Sun & Zheng, 2024 YANG Xu 2024) have proposed defense methods based on diffusion models. These methods use a diffusion model to denoise perturbed states and recover the original clean state as much as possible. is trained, the same model can be applied to many different victim policies without retraining. However, these approaches require extra computation inference time, because the agent must run the diffusion process before choosing an action. Our regularization method is complementary to these diffusion- based defenses and can be combined with them.
main advantage that, once the diffusion model
B PROOFS
B. PROOF OF THEOREM 5.1
Theorem 5.1. Consider an SA-MDP M = RR, with adversarial policy v. Let x denote the victim's 's policy and Ti the target policy. Assume that the divergence 'D admits the following variational representation:
D(T OU, Tgg) max (g((s, a, 80)) + ( {))]
where f and g are arbitrary convex and concave functions, respectively, and d S x A x S R a discriminator. Let d, be the optimal discrininator equation 3 Under this assumption, define the reward function Ra and the state transition probability p as follows:
Eas alals)p(e')
B(s)
Ra(s, (,5,8,55))
Loca a(als)p(s'(s.0)
(4)
c
otherwise,
p(8'18,8)
E
[a(al3)(661ss,))
(5)
aEA
where C is large negative constant. Then, for the MDP M = -S,S, Ra, 7), it holds that:
Ra,
min D(1 o v, Rig) arg max EMI (R((s,
(6)
Proof. Before proving this theorem, first establish the following lemma:
7), and let be the victim's 's policy. Given the reward function R specified by the adversary, the reward function R and state transition probability p are defined as follows:
Lemma B.1. Consider SA-MDP M
Ensa (sa) Leed a(a (((ss)))))sa)))
i B(s) otherwise,
3.s)) E =
(13)
p(s'1s,8) E <cd(s)p(ss(ssa)
(14)
where C is a large negative constant. Then, for the MDP M (S,S, R. r), the following equality holds:
arg max T Toy [R(s a, s') = arg max BA (f((,,,,'))
(15)
Proof. This proof follows the approach outlined the proof of Lemma in (Zhang et al. (020b). with some modifications to account for the differences in setting.
18

---

Published as a conference paper at ICLR 2026
In the proof of Lemma 1 presented in Zhang al. 2020b), by substituting - a(s),,,s) with
R(s, a, s'), we can derive the subsequent results:
Mss ()), Eo sp(s s)a(aa
(16)
Eap(s' (a, (a(āa
Let M = maxx.a.s' R(s a, s') and = mins,a,s' R(s, a, 8¹). We Define the reward C for when the adversarial policy selects an action à & B as follows:
M. 1
C< min
<<min/
M 7 M
(17)
1-7
From the definition of C' and M, we have for ((),
C' RR(s, â s') S M,
(18)
and, for vā E B(s), according to equation 16
M S R(s, 5 S M.
(19)
MDP has at least one optimal policy, so the M has an optimal adversarial policy v", which satisfies Vrov- (s) 2 (s) for Vs, Vv. From the property of the optimal policy, v* is deterministic. Let H - {v Vs, 3ā E BBs). v(a 8) = 1]. This restricts that the adversarial policy does not take actions & B(s), so E R. v $ 2i state s..
VaOV- (8 = EB
A P24441 8t 8
(20)
k==
= C' + Ep
p Metkt1
8t =
(21)
? M
$ 0 +
(22)
1
M
(23)
7
< Bg
Σ
7 fr4041
|=== = = Vioo (s9).
(24)
k=
The last incquality holds for any // E R. This contradicts the assumption that va is optimal. Hence, the following analysis will only consider policies included in N.
For any policy v E R:
(r) = Ep,r
x Petk+1
11 =
(25)
EEgi
Pr+l +
k Fe4k42 18c
(26)
k=U
Σ v(lls)
Σ
p(s' s,
A((,,ā,5)+ 7Bp,
*fo= 8641 =
(27)
deS
S'ES
Σ v(als) Ei(s'les a) ( + (x)
(28)
aES
e'ES
All policies in Ri are deterministic, we denote the deterministic action à chosen by v E Ri at as v(s). Then for Vv E Ri, we have
(s) = Σ p(s'ss, a) ((ss,,,') + (
(29)
EE
Ea R(s, a, (")ss,)))))) (oo)(
= Σ
n (all))p(s(1s.0)
(30)
e'ES AEA
= Σ m(alu(s))
Σ () [R(s, s²) 7 (To((s))
(31)
AEA
.'ES
19

---

Published as a conference paper at ICLR 2026
Thus, the optimal value function is
VROu (s) P' ())) aEA
Σ (salv((s))
Σ p(s)(ssa) R(s,a,,') V (o)
(32)
s'ES
The Bellman equation for the state value function V (s) of the SA-MIP M = USAABBRR.Py given as follows:
Lemma B.2 (Theorem of (Zhang et al. 2020b)). Given S P(A) and S + S, we have
ViOo(s) (5) Σ
x(alv(s))
Σ
p(s [R(s,a,s)) 7 (oor
(33)
AEA
s'ES
Therefore, if the reward function is R. then ViAw- ViOue. So, we have
ViOuN (s) 1' (*)))((s) aEA
Σ
aTaav (s)) Σ p(s'(s,a) R(s,,,,')) + (ooo.
(34)
s'E8
and V-OW- (8) 2 (8) for Vo, Vu E R. Hence, v² is also the optimal for V.
This lemma shows that an optimal policy on the MDP MI matches the optimal adversarial policy that maximizes the cumulative reward obtained from R specified by the adversary.
We now turn to the proof of the theorem. Under the assumption that the divergence admits variational form, the adversary's is objective can be expressed as follows:
arg min max [row(o(((s,,, Bagl- s')]
(35)
Let d4 the optimal discriminator. Focusing on optimizing the adversarial policy, the problem reduces to:
arg min Main Loo s')), arg max (so,(=9(d4ss,,,,))
(36)
This is regarded as cumulative reward maximization problem in which -g(d+) serves as the reward function. Therefore, for the MDP M- = (S,S, Ra.p.7) the following result holds by Lemma B.1
arg min D( )t)) = arg max M [Ra(s, 8,
(37)
B.2 PROOF OF THEOREM 6.1
Theorem 6.1. Let Reg: S x A The be the adversary's objective reward function, and the discount factor be (0,1). Assume that there exists upper bound Rig TR. for Rig. Then:
1
Boooo (B (Rigr(s, (8))))))))))))))))))) E. Rgg a)))
E. dt (Dec ((-1s))],
V2Rg
1=0
(11)
where (s) Pr(st s|x) represents the state distribution time t.
Proof. We begin by defining the state-action action distribution. The state-action distribution Pa: S x A - [0, is defined by the probability of encountering specific state-action pairs when transitioning according to a policy
(=(3, a) = 71->)
y Pr (st 8, 0, a ~ po(+), as ~ (T-18)), 8141 p(, az)) (38)
t=0
20

---

Published as a conference paper at ICLR 2026
The state-action distribution allows to express the expected cumulative reward under any reward function as:
E [R(s, a)] = E (x(ss,))(ss..
(39)
8,0
Using this representation, rewrite the left-hand side of equation T0 and bound via Pinsker's inequality:
2
1
Ae Regt a)] -EA [Rag (s,
a)])
-(
Σ Ettov(s,)
22
22
(40)
Ragi _ (m (s, a)l
Rig
(41)
(V2Div(Prors
(42)
DEL (ka(ProlllPt)
(43)
where, Dry represents the Total Variation distance, and DKL represents the Kullback-Leible diver- gence.
Next, we introduce the state distribution. The state distribution de S - [0, 1] represents the probability of encountering specific state when transitioning according policy m
(s) = (1 (1- 7)
Σ
r't Pr s so ~ ~po()), ~ T(-ss)), s141 p(-1s),a))) (44)
t=0
Building on definition, we establish the following lemma:
Lemma B.3, Given two policies T, O S + P(A) and their state distribution da drow, the following inequality holds:
72
DxL(dxl/dron -
Σ
EB
(45)
1- 1=1
Proof. This proof is based on Theorem 4.1 of (Belkhale et al. (024). Regarding the distance between the state distributions of and time 1. DRi.dd, (di, d:ow). the following inequality holds for
21

---

Published as a conference paper at ICLR 2026
t M 1 by using KL's joint convexity and Jensen's inequality:
(olddflllfooo) = J. dG(s') log d''')
(46)
di (5')
= - (1.a ydt (omashillll))) log
So ndts (9)
Sa, (
(47)
$ 1/ 5.a
(0 log 1 (mtaaassssp
(48)
$ 1 7d (1) a)
log (s) + log ((als)
(49)
(s)
xov(sss)
18 s,a
(s) $7 { of (s) log (s)
+ log x(dls)
(50)
Toy(als)
$ S. dz (s) log (s)
(s) +7 (s) log T(ass
(51)
rov(als)
S DD (10 + y10 smudt (DLL(((()))
(52)
1-1
S DKb (d0o) + Σ "T sndd [
(53)
1=0
<
E
o u(-1s))]
(54)
Thus, we obtain the following inequality:
DKL PxL(dxlldrow)) = / Σ
Enn (8)
(s)) log
(og
(55)
Eee 2 rd (s)
1=0
20
(s)
<
% (s) log
(56)
7 (d
1=0
00
J. (s) log
d' (5)
(57)
d: (s)
7't ''Di (dtlldl
(58)
t=1
1-1E Eynd (
(59)
1=0
1
<
2t- i E shat (
(60)
[=1
0=
72
<
7 Eanay (De (Da(((()))(oTo
(61)
1 2
1=1
22

---

Published as a conference paper at ICLR 2026
Finally, by using Lemma B.3 the following inequality holds for the distance between state-action distributions:
/
P=(s, a) log P (=(
DKL(Pm) pxllp =
(62)
(s)
8,a
(s))s)dr(s) {(cl8)d=(s)
(63)
(ou(a(s)drou(s)
((a]s)d log Ton(als) I (s) (s(s)
T(ass)
(64)
drow (s)
a,a
[ T(als) (dd [(ass) Toy(als) + +fo dx (5) log (s)
(s)
(65)
(alss)
(s( log Tou(als)
+ DRL(dalldad
(66)
72
00
y "EEndt [Dx(((((s]]TTu(([])) + 1-1²
Σ 1=1
y B Bond [PxL( (()))))))(l((((()))))
(67)
72
Band; [ + 1- 1=0
E endt [DKL (x(18))) o( [s))]
1=0
(68)
1
* Egndy (DKL (P) v(-1s))
(69)
t=0
L
C THEORETICAL ANALYSIS OF THE DISTRIBUTION MATCHING APPROACH IN OUR ADVERSARIAL ATTACK SETTING
In this section, we prove that the distribution matching approach in our problem setting is equivalent to the problem of minimizing the distance between policies through inverse reinforcement learning. This shows that the adversarial policy, which is learned through our distribution matching approach. rigorously mimics the target policy. Our procedures follow the proof of Proposition 3.1 in (Ho & Ermon 2016).
Let set of all stationary stochastic policies as II and set of all stationary stochastic adversarial policies as A. Also, we write R for extended real numbers IR U {oo]. The goal of inverse rein- forcement learning is to find a reward function such that when policy is learned to maximize the rewards obtained from this function, matches the expert's policy. This process aims to derive a reward function from the expert's trajectories. We formulate the adversary's objective as learning an adversarial policy that maximizes the victim's 's cumulative reward from the reward function estimated by IRL:
(Te)) = arg max - (b()) + min BM [c(s, CERSKA
[e(s.a)) all) - E. [c(s, a)],
(70)
PEN
RL(C) = arg min Exov (c(8, a)).
(71)
WEN
where c: S.x.A A R. is cost function, vb SxA_ R convex cost function regularization.
Note that we use a cost function instead of a reward function to represent reinforcement learning as minimization problem. Let e* IRI be the optimal cost function through IRI.. The optimal
adversarial policy is learned with respect to the optimal cost function: v² RL.(c").
For the proof, we define the occupancy measure. The occupancy measure is an unnormalized state-action distribution:
px(s,a) = Σ of (r = 8, a, = also ~ po(-), or ~ T(-1st,, 8t41 ~ rx(((st,at)) (72)
t=0
23

---

Published as a conference paper at ICLR 2026
Therefore, we present the Theorem C.I The Theorem shows that the optimal adversarial policy obtained via inverse reinforcement learning coincides with the optimal adversarial policy obtained through the distribution matching approach:
Theorem C.1. Let C be the set of cost functions, v be a convex function, and x³ be the conjugate of v/. When x. is fixed and C is a compact convex set, the following holds:
RL o (T
arg min v/"" (Prov - VEN
(73)
Proof. Let Drow = (mmovly E W. If Dov is a compact and convex set, equation 73 is valid according to the proof of Theorem 3.1 (Ho Ermon, 2016). Thus, we prove that D noy is compact and convex set.
The mapping from v to л u is linear, and N is compact. Therefore, by (Arkhangel'ski) & Fedorchuk 1990), II, is a compact set. From (Ho & Ermon 2016), when IT is compact, Dxo also compact. Consequently, Dxor is a compact set.
Compactness:
Next, we show that Drov is closed. Policy E II and occupancy measure ý E D have a one-to-one correspondence by Lemma 1 in (Ho & Ermon 2016). Let II, = { y E N,, n' = π o v be the set of all behavior policies. Since II, S II, correspondence. Thus, let {provt>Prous:" Provs:.... } be any cauchy sequence with Proor E D Der- Due to the one-to-one correspondence, there exists corresponding sequence {Ir V/, 1/2... }. Since N, is compact, the sequence (it o V1, π o V2,.... } converges to π o v E [,, Hence, the sequence (prony "arron:PPoogg--.... Proorg..... also converges the occupancy measure Prov corresponding x o v. Therefore, Daow is closed.
E II, and Proev Drow also have one-to-one
Convexity: First, we show that II, is convex set. For Vin E N,, Vuz N and E [0, 1J. we define o / as
TOV= = AT (1- A)T V2.
(74)
Then, have
TOV AT (1 X)TT 1/2
(75)
= Σ (al(s))T(aaa) + (1 - t) Σ (2(()s)T(ala)
(76)
áEs
dEs
Σ (áls) + (1 - ()))
(77)
dES
(A11 +11 (1- -)(2)
(78)
Using the convexity of N. we have tu + (1 t)ue E N. Thus, Tor II, holds for any in EN, D2 E N, and À E [0, and II, convex.
Noting that the one-to-one correspondence of PTOP E DoO and E II, for any mixture policy π o Vm E II, we have Prou.. E &Drov- Consequently, Daw is a convex set.
Based on the above, prove the Theorem following the same procedure the proof of Theorem 3.1 in Ho & Ermon 2016). Let E IRI 7 TODE E RL(2)- = RI. (Tig). The RHS of
73 denoted by
π U E arg min 4 (PrOv PTT)) arg min max -v((c) + / (Prou(s, P (8, ()) sov
say
(79)
We define the RHS of 73 by by L: Drow x C + IR as follow:
[(p.c) +cc)+ + / p(s, p(s,a)c(s,0)- Prww c(s,a,
(80)
We remark that L takes an occupancy measure as its argument.
To prove v V. we utilize the minimax duality of T..
24

---

Published as a conference paper at ICLR 2026
Policy E II and occupancy measure D have one-to-one correspondence Lemma I in (Ho & |rmon,(016), Since II, C II. т o E II, and Prov E Dooy also have a one-to-one correspondence. Thus, the following relationship is established:
PhoPA arg min mig max ()p,)),
(81)
DED+ou GEC
ē arg max min T(p.c). c)
(82)
CEC
PeD
Prop E arg L(p, c).
(83)
pEDnor
Drov is a compact convex set and C is also compact convex set. Since d is a convex function, we have that L(-, c) is convex for all and that L(A,)) -) is liner for all á. L(ū.)) is concave for all Due to minimax duality(Ferniqug) et al. 1983). we have the following equality:
min max L(p,c) max min L(p,c) c). peD CEC
(84)
CEC PED
Therefore, from equation 81 and equation
82 (Prons 2) is saddle point of T,, which implies that PRonA E arg pe [(p c) and so Paop PHOP
The right-hand side of equation 73 represents the optimal adversarial policy that minimizes the distance between occupancy measures as measured by tha Therefore, Theorem C.1 indicates that the optimal adversarial policy obtained through the distribution matching approach is equivalent that via inverse reinforcement learning.
D EXTENSTON OF BIA
In this section, we verify that the assumptions of Theorem 5.1 hold for other imitation learning methods beyond GAIL and demonstrate that BIA can be applied to variety of existing algorithms. As concrete examples, we consider GAIfO (Torabi et al.) 2019). an ILfO extension of GAIL, and AlLBoost (Chang al. 2024), a state-of-the-art ILfD method.
D.1 GAIFO
GAIFO is optimization method applicable in the ILfO setting, where only expert state transitions are provided. It extends GAIL to ILFO and optimizes the policy using a GAN-style algorithm. In GAIfO, discriminator Do: S S - IR that takes state S and next state s' E S input, and reformulate the adversary's objective as:
arg min max J [log Do(s,'))
+ M [log(1 Do
(85)
/ D.
This reformulation leverages that the objective be reduced to distribution matching problem similar to GAIL. By applying the discussion in Appendix|Clas the ILfO setting, we can also convert the adversary's 's objective in our problem setup distribution matching problem. Thus, it ensures that the reformulation of equation 85 holds. Consequently, by Theorem 5.1 it is transformed into the following objective:
arg max E [Ro.]:
(86)
where RD, is the reward function obtained by replacing g(d+) with the optimal discriminator log D in Equation 4 equation 86 can optimized by any reinforcement learning algorithm. Therefore, BIA can also be applied to GAIfO.
D.2 AILBOOST
AlLBoost is a state-of-the-art ILfD method based on gradient boosting, which allows the use of more efficient off-policy algorithms compared to the on-policy methods used in GAIL. In AlLBoost, the objective is expressed in a variational form using a weighted ensemble of policies rather than a single policy:
arg min max V
EMM
a)] + BM [- exp(D(s.a))]
(87)
25

---

Published as a conference paper at ICLR 2026
where {as, i oV,) denotes the weighted ensemble with a, 2 0 and Σ, a; 1. When executing π o v, at the beginning of an episode, a single policy æ v, is sampled with probability a, and then π V2 is executed for the entire episode.
we show that Theorem 5.1 also applies to this ensemble policy. First, given the {ar, π o Ui iteration i, we train discriminator D, on the experiences collected by π (()) Then we optimize the next weak policy л M++1 as
Using Lemma Mammaaa
B.
ensemble π (
π O 1OVI
arg max F.Ad [- Dr(,,, a)]. nov
(88)
After the optimization, the existing weights a, are rescaled by the weighting parameter a to ar a)). and the newly obtained и Mz+1 is added to the ensemble with weight a.
By Lemma B. this equivalent to the following reinforcement learning problem:
14+11 arg max E (s.a).
(89)
where Rp. is the reward function obtained by replacing g(d.) with D.. in equation 4 Since equation 89 defines a standard reinforcement learning problem, the ensemble policy can be learned accordingly. Thus, Theorem 5.1 holds for All Boost as well, demonstrating that BIA is applicable.
E TDRT-PPO ALGORITHM
Algorithm 2 Time-DDscounted Robust Training in PPO (TDRT-PPO)
1: Input: Number iterations T, clipping parameter 6,, Minibatch size M, regularization coeffi- cient X, learning rates (xx; Qv
2: Output: Optimized policy Te
3: Initialize actor network (e(aas) and critic network (s)
4: for = 1 to T do
5: 'D < Collect trajectories using current policy ma
6: for each (st, ar, Tr, St-1) in D do
7:
R, + Ei r'ettt
8:
A, + Rr Vo(ss)
9:
end for
for epochs do
B (18t2) Otp Rep, A ~
M
12:
à - o- -- av Vo V7 E (Vo(s:) R2,
# #Compute the time-discounted regularization term
14:
Rs + Ex,, EB maxs, EB(s.,) ) 7t DKL xo(a,les)
15:
(0) new [a:le]
16:
# Update actor-network with regularization
0 8-0 0 ar Vo
M EM (lip,,r)))) cc, (c)A + XRd)
18:
end for
19:
end for
We present the complete algorithm for TDRT-PPO in Algorithm 2 This algorithm extends the standard PPO algorithm by incorporating time-discounted regularization Ro.
Directly computing the maximum value of the KI. term in line expensive. Following SA-PPO (Zhang et al. (0206). we therefore apply convex- relaxation methods (Zhang et al., 2018 Wong Kolter 2018| Salman et al. 2019 Zhang et al. 2020a) to obtain tight upper bounds. We implemented this using the auto LiRPA toolkit (Xu et al. 2020). reducing the computational cost the regularization term and enabling an efficient implementation
Algorithm is computationally
TDRT-PPO. approach achieves shorter training times compared to adversarial training
This regularization-basee methods that require learning adversarial policies (Liang et al., 2022). A detailed analysis of the training time efficiency is provided in Section H.3
26

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of attack and defense performances in MuloCo environments. Each value repre- sents the average episode reward + standard deviation over 50 episodes. Each parenthesis indicates (Access model, Adversary's 's knowledge). For defense methods, better defense performance achieved with smaller attack rewards. For attack methods, higher attack rewards lead to better performance. Our proposed attack and defense methods are also effective for MuJoCo environments.
Task
Target Reward
Method
Clean Reward
Attack Reward
Turgeted
(SA-RL)
(ours)
Avg.
(Wite bon rage palcy
(Blak lrs wourd fendioe)
(Rl hes, dencesraion
4574
PRO deferse)
3027
3975
4681
4423
4359
TDRT-PPO (Ours)
2944 243
2763
3425
3278 = 13
3155
HallCheetah
3688
PPO (No defense)
2178 #
2823 + 918
4023
3602 :: 15
3482
TDRT-PPO (Ours)
2013
2606
2084
Hupper 3513 + 192
PPO (No defense)
1405 + 54
1673 +
3056
3198 =
2642
TDRT-PPO (Ours)
1310
1524 + 2190
1481
1665
During the KI computation in line 17, gradients are back-propagated only through the output associated with the perturbed state, while the output for the original state is held constant, This design confines the perturbed output to remain close the original output.
For clarity, we denote perturbed states by E (s)). However, since the defender lacks knowledge of the adversary's true ball B, is optimized within defender--peciffeed region instead.
F ADDITIONAL EXPERIMENTS
In this section, we evaluate the generalizability of our proposed method by conducting additional experiments in two distinct environments characterized by diverse state and action spaces. The first environment is MuJoCo [Todorov et al. 2012), which features both continuous states and continuous actions in robotic locomotion task. The second is the MiniGrid environnent(Chevalee Boisverrt jtt al. 2018), which involves a maze task with a discrete action space. For MiniGrid, we conduct experiments under configurations: one where the agent's state is represented as coordinates and another where is represented as images.
F.1 EXPERIMENTS ON MuJoCo ENVIRONMENTS
Setup. As a possible attack in the real world, the adversary aims to increase the victim's 's cumulative reward, which the victim intentionally constrained during training. For instance, an autonomous driving scenario where reaching the destination sooner yields higher rewards, the agent's speed may increase significantly, which may compromise safety by increasing the risk of accidents. To mitigate this, the victim may adjust the reward function during training to maintain balanced level of performance. However, the adversary's 's goal here to force exceptionally high rewards, thereby inducing the victim to adopt excessively fast and potentially unsafe driving behaviors.
We set the adversary's 's objective maximize the victim's reward, while the victim's policy is intentionally constrained to achieve moderate rewards. We define the target policy as one that attains very high rewards on the same tasks. We use the Ant, HalfCheetah, and Hopper tasks, each with a reward function that increases the agent moves to the right. All other experimental settings are the same as those described in Section
Attack and Defense Performance Results. Table[4presenti the results on an attack budget e 0.3. The attack reward represents the reward obtained by the victim under the attack. When evaluating attack methods, a higher attack reward indicates stronger attack performance. Conversely, when evaluating defense methods, a lower attack reward implies greater robustness against attacks. The target reward is an upper bound for the attack reward, the reward obtained by a well-trained target policy. The clean reward denotes the reward obtained by the victim in non-attack. Unlike in the experiments in fection[7] the victim is intentionally not fully trained. As a result, the clean reward does not indicate whether the defense method affects the original performance.
The targeted PGD attack does not exhibit sufficient attack performance, which consistent with the attack performance in the Meta- World experiments, where it also failed manipulate the vietim
27

---

Published as a conference paper at ICLR 2026
AB
herzh
Ikpper
Figure 2: Attack and defense performances under various attack budgets in MuJoCo environments. The horizontal axis represents the attack budget, which indicates the adversary's 's intervention capabil- ity. The vertical axis shows the attack reward, which represents the reward obtained during the attack. Each value represents the average reward over 50 episodes.
Table 5: Comparison of defense performances under the best attack in MuJoCo environments. Each value represents the average episode reward + standard deviation over 50 episodes. Higher clean rewards and lower best attack rewards indicate better defense performance.
Tass
Method
Clean Reward (t)
Best Attack Reward (1)
PPO
4720 1: 255
850 :: 612
Ant
SA-PPO (smoothing w/o lime-discounting)
4268 1 342
3318 .9
TDRT-PPO (ours, smoothing w/ time-discounting)
4612 219
3206 781
PPO
3680 + 220
160 = 540
HalfCheetah
SA-PPO (smoothing w/o time-discounting)
2963 398
2826 513
TDRT-PPO (ours, smoothing w/ time-discounting)
3577 197
2981 462
PPO
3290 1. 150
820 1 430
Hopper
SA-PPO (smoothing w/o time-diccuunting)
3198 1:
1579 672
TDRT-PPO (ours, smoothing time-discounting)
3214
1345
effectively. This suggests that single-step optimization is ineffective manipulating overall behavior in this experimental setting. Comparing the reward maximization attack and BIA, observe that the reward maximization attack demonstrates stronger attack performance. We argue that this occurs because BIA excessively alters the vietim's 's behavior to match that of the target policy. In MuloCo tasks, moving to the right yields higher rewards. Therefore, in the reward maximization attack, since the adversary's 's objective is maximize the victim's reward, no perturbation is applied when the victim is already moving to the right. Thus, the attack does not interfere with the victim's 's movement. On the other hand, in BIA, the attack aims to make the victim's 's behavior close the target policy's behavior, regardless of the reward. Consequently, even if the victim is already moving to the right, perturbations are still applied change the victim's 's behavior. This excessive modification disrupts stable locomotion, ultimately reducing the attack rewards. In evaluation of defense performance, TDRT-PPO results in a lower attack reward than vanilla PPO, indicating that it more robust against behavior-targeted attacks.
Attack Performance on Various Attack Budgets. To further analyze attack performance, we conduct experiments with various attack budgets. The experimental results shown in Figure To standardize the scale across different tasks, the attack reward is normalized so that the target reward set to 1.
In the Ant task, the targeted PGD attack demonstrated high attack performance at large attack budgets. However, in the remaining tasks, failed achieve sufficient attack performance even with large attack budget. Similar to the experiments in Mela-World, this result suggests that there are no falsified states where the victim's 's chosen actions perfectly match those of the target policy. In the HalfCheetah task, under the reward maximization attack, the attack reward for vanilla PPO exceeds the target reward. This occurs because the reward maximization attack only observes the victim's obtained rewards and performs attacks independently of the target policy's 's rewards. As result, the victim under attack achieves higher rewards than the target policy. In contrast, in BIA, the target reward serves the upper bound for the attack, meaning that the attack reward never exceeds this limit.
Defense Performance Comparison. We also compare TDRT with the baseline method in the MuJoCo environment. Since cannot directly compare clean performance in the experimental setup
28

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of attack and defense performances for the setting where the state is represented as coordinates in MiniGrid. Each value represents the average episode reward + standard deviation over 50 episodes. Each parenthesis indicates (Access model, Adversary's knowledge). For defense methods, better defense performance is achieved with smaller attack rewards. For attack methods, higher attack rewards lead better performance. Our proposed attack and defense methods are also effective for discrete action spaces.
Task
Target Reward
Method
Clean Reward
Attack Reward
Rew Max (SA-RL)
Rew Mux (PA-AD)
BIA-ILID (ours)
Avg.
Olladd-bre msont Tractio))
(W tvo, moat frnatoo)
(Blick Noo, desosoratioe)
0.96 +
TDRT-PPO (Ours)
PPO (No defense)
0:44 =
0.91 + 0.68
0.89 =
0.54 was
0.94 + 005
0.91
PPO (No defense)
0.46 1: oz1
0.95 1 use
0.93 1: uuo
0.96 1: wos
0.95
16 16
0.98 a
TDRT-PPO (Ours)
0.38 025
Table 7: Comparison of attack and defense performances for the setting where the state is represented as an image in MiniGrid. Each value represents the average episode reward + standard deviation over 50 episodes. Our proposed defense method is effective for image inputs. However, the ad adversarial policy without white-box access is inelfeetive for image inputs.
Target Reward
Method
Clean
Reward
Max (SA-RI.)
Rew (PA-AD)
BIA-I1.0 (ours)
Asg.
[Hlcc-bon, reveal lancton)
(Whaicho, roverd lanctons
S.K.E (RGB image imput)
0.96 d
PPO (No defenme)
TDRT-
0.37
0.43
0.32
0.50 1:
0.92
0.48
0.40
0.63
(RGB image impur)
16
0.98 1
TRRT-PPO (Ours)
PPO (No defense)
0.38 now
0.44 + 0.7
0:49 d: 1os
0:47 + 0.19
0.94 0.2
0.63 0.1
0.39 + 0.0
0:48 0.24
0.64
0.50
described above, we apply reward- coserld-iiimizatioon setting. Specifically, we define the target policy as the policy trained with a reward function whose sign is flipped from the original task reward. Unlike the previous experiments, the victim policy is sufficiently trained before the attack is applied. The results in Table that TDRT-PPO achieves robustness comparable to SA-PPO while attaining higher clean rewards, which is consistent with the findings in Section 7 the versatility of TDRT across different environments.
These results demonstrate
F.2 EXPERIMENTS ON MINIGRID ENVIRONMENTS
Setup. MiniGrid is 2D grid- environments, where the agent's 's objective is to reach a designated goal coordinate. The agent's actions are defined over discrete space that includes movements and interactions such as picking up keys. We evaluate tasks on both 8 x 8 and 16 x 16 grids. Similar to experiments on MuloCo in Section F. e the adversary's objective maximize the victim's 's reward, while the victim's 's policy is intentionally constrained to achieve moderate rewards. We define the target policy as one that attains very high rewards on the same tasks. We use the Ant, HalfCheetah, and Hopper tasks, each with reward function that increases as the agent moves to the right. All other experimental settings are the same as those described Section
Results in coordinate states. Table shows the results for the setting in which the state is represented as coordinates, with the attack budget set to = 0.3. Our experimental results show that our attack and defense methods are effective even the discrete action spaces. Specifically, BIA-ILiD exhibited higher attack rewards against PPO (No defense), and TDRT-PPO achieved lower attack rewards than PPO (No defense). Since the attack is executed on the victim policy's 's state space, it is as effective in discrete action spaces as it is in continuous ones.
Results in vision-based states. Next, we evaluate the effectiveness of our proposed method in vision-based control tasks. We modify the MiniGrid environment to use RGB image inputs. Table 7 shows the results for the setting in which the state is represented as image, with the attack budget set to e = 255 Our experimental results show that the effectiveness of our attack method decreases with image inputs. When the states are represented as RGB images, the dimensionality of the state space increases significantly. Consequently, the action space of the adversarial policy expands, making learning much more challenging. This problem is not unique our method but is common
3 e
29

---

Published as a conference paper at ICLR 2026
Table 8: Comparison of defense performances under the best attack in MinGrid environments, Each value represents the average episode reward + standard deviation over 50 episodes. Higher clean rewards and lower best attack rewards indicate better defense performance.
Task
Method
Clean Reward (1)
Best Attack Reward (*)
PPO
0.93 +: 0.04
0.12 + 0.32
x 8
SA-PPO (smoothing w/o time-discountingg
0,93 + 0.05
0.78 =
TDRT-PPO (ours, smoothing w/ time- )iscouuti)n))))))
0.92 +
0.74 = 0.15
PPO
0.92 +
0.24 =
x
SA-PPO (smoothing w/o time-discounting)
0.92 +
0.71 +
TDRT-PPO (ours, smoothing time-discounting)
0.93
0.74
among adversarial policy-basse based attacks such SA-RL. PA-AD overcomes this limitation but requires white-box access. Thus, targeted attacks under black-box access in vision-based control tasks remain future work.
Defense Performance Comparison. Analogously to the MuJoCo experiments, we compare TDRT with the baseline method in the MiniGrid environment. Because attack performance is insufficient when using rision-based observations, we restrict our evaluation to coordinate-state settings, while keeping all other experimental configurations identical to the MuJoCo setup. The results are summa- rized in Table TDRT-PPO and SA-PPO exhibit similar robustness and clean performance, which conjecture because MiniGrid is a relatively easy task in which both methods nearly saturate the achievable performance, leaving little room for noticeable differences in clean performance.
G ADDITIONAL ANALYSIS IN ATTACK METHODS
G.1 DEMONSTRATION EFFICIENCY
In this section, conduct additional experiments investigate how the quantity and quality of target policy demonstrations affect the attack performance of BIA-II.D//L.OO. We evaluate the performance of BIA-II AL--I.-//LLL. ff/IL.OO using different amounts of demonstrations: 1, 4, 8, 12, 16, and 20 episodes. The victim is a vanilla PPO agent without any defense method. All experimental settings remain the same as those in Section 7 with the attack budget set 0.3. Figure 3 shows the experimental results. Across all tasks, we observed no significant performance degradation when using up four demonstration episodes. However, performance declined when only one demonstration episode was provided. This decline can be attributed to the variability in initial states, where the discriminator cannot effectively handle such diversity with extremely limited demonstrations.
We argue that the number of demonstrations required for successful tarrgeed attacks depends on the environment's characteristics. In environments with deterministic state transitions and initial state distributions, where the target policy exhibits similar behavior across episodes, fewer demon- strations may suffice. Conversely, environments with more randomness in state transitions and initial state distributions require more demonstrations to ensure proper generalization of the discriminator.
G.2 ATTACK BUDGET EFFICIENCY
In this section, we present a more comprehensive analysis of the attack efficiency of BIA-ILID/ILFO.
We conduct experiments across different attack budgets = 0.05, 0. 0.2, 0.3, 0.4, 0.5 and evaluate the attack rewards. The victim is vanilla PPO agent without any defense methods. All experimental parameters remain consistent with those in Section 7 and the results for 0.3 correspond to those presented in Table [
The experimental results are shown in Figure Across all tasks, we observe that attack rewards increase proportionally the attack budget increases. Notably, in the window-close and drawer-close tasks, the attack rewards nearly match the Target Rewards when given larger attack budgets, indicating that the attack successfully guides the victim to almost perfectly replicate the target behavior.
30

---

Published as a conference paper at ICLR 2026
window--loss--2
window-open-22
4000
800
Attock Reward
3000
Atfack Rew
600
2000
400
1000
BIA-ILID BIA-ILIO
200
BIA-ILID
BIA-ILI0
16
20
12
16
Trajectacics
Irajectorica
drawer-close-22
drawer-open-v2
5000
Attack Roward
Attack Reward
1500
3.00
1000
2000
BIA-ILO ILD
BIA-ILOD ILD
1000
BIA-ILR)
BLA
Trajettries
Trajecuurien
Figure Attack performance of BIA MIA-ILIDIILIO with varying amounts of demonstrations. The x-axis shows the number of demonstration episodes, and the y-axis represents the attack reward. The attack budget 0.3. Each environment name represents an adversarial task. The solid line and shaded area denote the mean and the standard deviation 2 over 50 episodes.
window-close-v2
window-open--2
4200
4000
Attack Reward
Attack Reward
3000
3000
BIA-ILID
BIA-ILID
2000
ILD
Reward Muxmization
BIA-IL8)
10000
Reward Maximizaiion
1000
Turget Rewan!
0.10
drawer-close-22
drawer-upen-r2
50000
BIA ILI
Attack Reward
1000
Attack Reward
BIA-ILIO
Maximization
Target Reward
2000
BIA-IL ID
2000
BIA-IL80
1000
Reward Maximizaion
1000
Turget Reward
0.05
0.10
0.20
0.30
0,40
0.50
0
0.05
0.10
0.20
0,30
0.40
0.30
Afteck Budget
Attadl: Budgel
Figure 4: Attack performance of BIA- with varying attack budget The x-axis shows the value of the attack budget, and the y-axis represents the attack reward. The target reward represents the cumulative reward obtained by the target policy and serves as the upper bound for the attack rewards of BIA-ILiD/ILF.. Each environment name represents an adversarial task. The solid line and shaded area denote the mean and the standard deviation / 2 over 50 episodes.
In the window-close and drawer-close tasks, where the attacks are particularly successful, observed an interesting phenomenon. When the attack budget is small, there are high variances in attack rewards, but this variance decreases the attack budget increases. The high variances indicate that among the 50 evaluation episodes, some attacks achieve perfect success while others completely fail. This finding suggests that the initial state significantly influences the attack success rate. We
31

---

Published as a conference paper at ICLR 2026
Table 9: Comparison of attack and defense performances for the setting where the state is represented as an image in MiniGrid. Each value represents the average episode reward + standard deviation over 50 episodes, A higher attack reward indicates higher attack performance and lower defense effectiveness. Each parenthesis indicates (Access model, Adversary's knowledge).
Task
Target Reward
Attack Reward (BIA-ILID)
803 + 129
777 + 106 3298 +
window-close
3389 =
4543 + 39
3962 + 666
461 +
358 + 318
oin
2076 + 1599
585 + 439
4508
566 + 523
1029 864
1470 + 1016
drawer-close
3480 + 1744
4252 +
4868 +
4760 640
1175 = 510
1278 + 637
drawer-open
2544 +
1169 + 645
4713 + 16
1556 + 607
Table 10: Comparison of attack performances. Each value represents the average attack success rate (ASR) over 50 episodes. Each parenthesis indicates (Access model, Adversary's knowledge). We set the attack budget 0.3.
Targeted PGD
Rew Max (PA-AD)
Rew Max (SA-RL)
BIA-ILID (ours)
BIA-ILIO (ours)
Random [ao box
Task
(wiitee box)
(black b0x)
(black box)
(black box)
(no box)
window-close
0.08
0.82
1.00
0.72
0.74
0.00
window-open
0.08
0.06
0.12
0.06
0.00
drawer-close
0.68
0.68
1,00
1,00
1,00
0.22
drawer-open
0.00
faucet-close
0.00
0.04
0.68
0.68
0.60
0.00
fancet-open
0.00
0.00
0.00
0.12
0.08
0.00
mundle-press--tee
0.16
0.98
1.00
1,00
0.02
mundle-pull-side
0.06
0.70
0.92
1.00
1.00
0.00
door-Be
0.04
0.46
0.50
0.42
door unlock
0.00
0.62
0.68
0.64
0.62
0.00
hypothesize that this occurs because certain initial states require the victim to perform actions that are rarely selected in their normal behavior, making the attack more challenging in these scenarios.
G.3 RELATION BETWEEN THE PERFORMANCE OF THESE DEMONSTRATIONS AND THE ATTACKING PERFORMANCE.
In this section, we evaluate how the performance of the target policy influences the attack performance of BIA. We use three types of target policies. Each target policy achieved different target reward. All other settings remain the same described in the experimental section.
Table shows the results. We confirm that lower-perforning target policies lead to lower attack performance. This is because the adversarial policy is not trained to maximize the attack reward but rather to mimic the target policy. However, even if the target policy is suboptimal, selecting one that closely resembles the victim's 's original behavior may still lead to strong attack performance.
G.4 EVALUATING ATTACK PERFORMANCE WITH ASR
In this section, we evaluate the performance of the attack methods using the attack success rate (ASR). Success is determined by the task success flag provided by the Meta- World environment. For each Meta-World setting, we run 50 episodes and report ASR as the fraction of episodes marked as successful. All other experimental settings follow the main text.
32

---

Published as a conference paper at ICLR 2026
The results are shown in Table 10 Consistent with the evaluation based episode reward, the results indicate that BIA is effective, supporting that the attack performance of our proposed method generalizes across evaluation metrics.
G.5 COMPUTATIONAL COST OF TRAINING ADVERSARIAL POLICIES
In this section, compare the computational cost of different adversarial attack methods. We divide computational cost into two components: training-ting cost and test-time cost. This distinction is important because different attack families place the computational burden different phases. For example, BIA requires training an adversarial policy, but its test-time execution is simply a single forward pass. contrast, Targeted PGD requires no training, but incurs optimization overhead at every timestep during test-time rollout. Table I summarizes the computational cost measured single NVIDIA H100 GPU.
Table 11: Comparison of computational cost.
Targeted PGD
Rew Max (PA-AD)
Rew Max (SA-RL)
BIA-ILID (ours)
BIA-ILFO (ours)
Training time Test time
3.6 h
4.3 h
6.2
6.1
37 sec / episode
BIA vs. reward-based white-box attacks (Rew Max). BIA has higher training cost than Rew Max (PA-AD). This is mainly because imitation learning-based BIA must train both a policy and diseriminator, while Rew Max only optimizes the adversarial policy. On the other hand, BIA only requires demonstrations of the target behavior, whereas Rew Max assumes access to the target reward function, which is often harder to obtain in practice.
BIA vs. Targeted PGD. As described above, Targeted PGD has no training-time cost, but incurs test-time optimization cost. When many episodes need be attacked (e.g.. long deployments or large-scale evaluation), BIA's training cost is spread out over many episodes, and BIA becomes more computationally efficient overall than repeatedly running Targeted PGD.
H ADDITIONAL ANALYSIS IN DEFENSE METHODS
H.1 FULL RESULTS IN DEFENSE PERFORMANCE EVALUATION
In our evaluation of defense methods in Section 7 we only provide the results of the best attack. In this section, present the results of all attacks in Table 12 Targeted PGD is ineffective against robustly trained victims. We observed a trend where the Reward Maximization Attack tended to be slightly more effective than BIA when attacking highly robust victims. When the vietim's robustness is high, it becomes difficult to make the victim behave like the target policy, which may cause the learning process BIA fail or collapse. On the other hand, in the Reward Maximization Attack, the reward serves as a good guidepost, allowing learning to proceed even when the victim's 's robustness high.
H.2 IMPACT OF REGULARIZATION COEFFICIENT
We analyze how policy smoothing affects both robustness and original performance. The strength of the regularization is determined by the coefficient X. Accordingly, we evaluate the clean rewards and best attack rewards for SA-PPO and TDRT-PPO with E {0.03, 0.1, 0.2, 0.3, 0.5). The results are shown in Table [3 and all other settings follow those in Section 7 (where X=0.3).
Robustness. For X=0.5, the best attack rewards remain unchanged compared to 7=0.3 for both TDRT-PPO and SA-PPO, indicating that X=0.3 provides sufficient regularization, and that further increasing it does not enhance robustness. Conversely, as À decreases, the best attack rewards decline. Notably, TDRT-PPO fails to maintain robustness when A=0.03, highlighting the need for a certain level of regularization achieve robust performance.
33

---

Published as a conference paper at ICLR 2026
Table 12: Comparison of Defense Methods, Each value is the average episode rewards + standard deviation over 50 episodes. Clean Rewards are the rewards for the victim's 's tasks (no attack). The best attack reward is the highest reward among the five types of adversarial attacks. The attack budget is set to 0.3.
Adv Task
Methods
Clean Rewards
Attacki Rewards (1)
PSO
4305 2: 12)
947 2: 529
1605 133
4505 2:65
4255 300
5962 655
4206 510
4505 65
XTLA-FPO
4169 -00
1706
2008
4:
6378 319
3065 1115
2564 157
4370
PA-ATLA-PPO
4:3
RAD-FPO
513
1157
P90
2579 0 1256
400
400 2:5
575 2: 135
481
381
440 20
515 2: 135
SA-PI1
690 10
478.1
477
181 1 6J
4788
21 03
13 1:2
685 1:1 6.
4112
432
009
175
172 45
482
ATLA
4550 2:
354 + 257
310 0: 250
541 2:
400 340
556 640
532 :: 444
586 +: 649
wistne-open
RAD-FITO
BN-ATLA-PDD
4369
6331 109
335
299 90
234 63
301
671 540
112
455 492
116
531 1.112
534 1. 673
671 889
131
SA PPO
0092
150
156
272
200
272 37
TPRT (OEES)
4:53
213 6: 54
213 54
229 4 54
217 4:
254 = 214
229 +: 137
254 214
PPO
4714 16
1000 #: 1585
2001 =: 150
4658 # 147
3168 1788
4760 640
4636 6: 191
4700 + (40)
PA-ATT AITTA-PPY)
4511 103
1354
1858
0966
4855
RAT-SPP
4354 1-1
166 1 134
958 4T6
2011 L 7165
1106 3176
4526 913
7704 +: 7765
1998 1 971
WocLR, PPO
4103 204
562 = 1335
834 1635
3654 1976
1738 2001
4857 2
4835 22
4807 =
56. PPO
=
1345
1620
1965
PEO
4855
841 3: 357
953 450
1499 5: 536
1607 4
1555 2: 607
1445 610
1556 607
ATLA-PPO
6363 7
#: 270
+ 129
1158 1026
650 21 518
831 = 655
741 #: 561
11.5 1026
957
Waz
4304 634
410
405 2.8
579 0131..0 15
515
445
442 12
579
SA-P1O
4.81 1537
365 3.9
165 2.9
358.2 2.9
368 3.9
403 09
415 49
403 49
TPRT: (000))
4800 3:37
37% 10
778 10
378 10
375.8 10
=:4
378 10
ATLA-
4756 2:
1425 115
1723 137
3872 732
3907 726
4105 790
4058 791
4108 + 700
fooetelloe
PA-AILA PPO
RAD-FTI
5715 9022 802
4110 46
1275 210
1562 137
4012 8: 123
3292 833
1955
1745 165
1789 48
1927 72
1949
4012 4: 123
SA-P11
4106 1:42
1263
1599
1301 967
457
465 11
1599
TDRT-PP (00.5)
4740 17
1297 2: 743
1432 907
1789 6 510
1615 +
1169 243
1318 362
1789 + 610
PSO
4754 15
1577 81
2514
1418
K5
1495
2718 1741
9131 1493
Gueet--pon
PA-ATEA PPO
RAD-PRO
4713-111 111
3767
1556 2: 1045
1358 97
4254 5- K25
3546 1130
1545 206
2356
3133
695
2934 4: 11.45
1674
1358
4254 4 (25
WixcR
3756
2101 2 1106
2605
288 1340
2415
1196
2507
1300
3013 9: 1301
3013 5 1301
5A PPO
199
257
PPO
4042 2 732
1866 : 1340
1994 + 1225
4625 + 175
4726 à: 175
4031 408
4027 550
4726 175
XTLA-FIN
481
1961
797
1437
4107
PA.
2211=
1385
RAD-FRO
814
2375
855
997
2575
Woc.B. PPO
3724 2
1073 5 911
1784 983
5042 = 1095
2895
1742
2184 891
1964 = 2133
3042 =: 1195
SA-111
5235 305
817 1541
506 1054
1051 1627
915 1 1527
1819 2209
1838 L. 11.59
1858 L. 11.9
TDRTU-19 (IXEn)
515
438
4067
1356
5115
1925 176
PPO
3617
ATLA-POO
4606 2.58
482 2: 426
534 2: 438
531 534
157 658
482 1059
228 2: 969
532 S34
badle pill-
RAD-FFO
PA-AIAA PPO
3534 = 1903
4800
492 2: 763
342
493 54
466 1044
425 324
252 514
512 982
1156
392 2 993
1035-2
$12 952
SA.P
0041 790
10:5-0
1-6:1
1-1
6404 176
30-2-4
30 8
T
6:3.1
5:6:1
7
PPO
4000 4:33
169 494
640 2: 664
1951 &: 1186
765 769
2045 :: 1229
1906 +: 1045
2041 5: 1239
doas-ook
PA-ATLA--P1
RAI--1TO
2015 1328
2185 1111
455
458 10
457
459 16
611 205
221 966
112 302
517
589
091
135
563 131
992
713. 963
140 19
WooeR PPO
2430 = 1415
461 7
461 1
561 =7
561 4.7
502
562 14
562 +: 14
PPO
TDRT
457
PRO
3845 79
391 3:59
331 61
3421 + 904
3395 2 1111
3335 992
3123 =: 1123
3421 + 974
XTLA-PP RAD-PPO
6501 761
665
1166
3371
1550 1325
3165 1358
3163 4: 1186
3771
dotrsnieck
1543
2743
Wocdl.PP PPO
2545 201
125 0: 171
761 161
1073 161
1044 165
855 2: 130
929 97
1073 4: 161
5% PPO
2017
505 2: 123
515 130
514-2 133
550
713 1135
757 1001
757 1001
TIXET-I1O (senn)
1600
312
711
450
402.
360
34

---

Published as a conference paper at ICLR 2026
Table 13: Comparison between TDRT-PPO and SA-PPO with different values. Each value the average episode rewards =\t standard deviation over 50 episodes. Clean Rewards are the rewards for the victim's tasks (no attack). Best attack rewards represent the highest attack reward among the five types of adversarial attacks. The attack budget is set to = 0.3. In SA-PPO, which does not apply time discounting, ensuring sufficient robustness leads to significant drop in performance on the original task. In contrast, TDRT-PPO, which applies time discounting, achieves high robustness while preserving original-task performance.
Adv Task
TDRT-PPO (ours)
SA-PPO
Clean Rewards (1)
Best Attack Rewards ())
Clean Rewards (1)
Best Attack Rewards ())
0.03
4500 + 20
1853 + 241
4482 + 21
1452 = 512
0.1
4512 12
712 1 46
4324 1: 76
987 + 62
window-close
0.2
4403 34
512
4218 129
1:
0.3
4412
482
4367
485 + 61
0.5
4351 +: 130
495
4041 + 293
499 1:
0.03
4512 38
489
4483 23
401
0.1
4430 47
397 219
4219 +: 76
253 +
window-open
0,2
4403 1: 52
253 321
4198 1: 87
284 +: 21
0.3
254
4092
0.5
4313
263
4015 1: 212
268 d:
0.03
4660
4709
4792 +:
0.1
4398 72
4592 +
4129 +:
4809 +:
drawer-close
0.2
4442 + 44
4890 +: 4
2183 + 572
5 + 3
0.3
4237
93
4860 +
2156 + 453
0.5
4184 +
4592
1952 +
4 4
0,03
4818
1098 +: 192
4799 42
809 210
0.1
4860
792 :
4801
823
drawer-open
0.2
4843
394
4766 31
670 +
0.3
4802 1: 27
378 + 10
4161 = 1537
403 + 49
0.5
4839 25
413 -
3984 76
405 +:
Original Performance. Both TDRT-PPO and SA-PPO experience slight drop in performance as À increases, likely because stronger regularization reduces the expressiveness of the policy. For SA-PPO, lowering À improves performance, suggesting that A=0.3 may be overly stringent and that reducing it helps recover the policy's representational capacity.
Crucially, while SA-PPO avoids performance degradation at weaker regularization levels, it fails to achieve sufficient robustness in those settings. In contrast, TDRT-PPO can remain robust with- out compromising original performance. These findings indicate that time discounting in TDRT effectively curtails behavioral shifts throughout the entire episode.
H.3 TRAINING TIME EFFICIENCY
We compare the training time of TDRT-PPO in the window-close and window-open tasks with ATLA-PPO, PA-ATLA-PPO. RAD-PPO, WocaR-PPO, and SA-PPO. For all methods, training is conducted for about 3,000,000 steps. To ensure a fair comparison, we use an Nvidia H100 Tensor Core GPU for the training of all methods.
Table 14: Comparison of training time on window-closs and window-open tasks. For all methods, training conducted for 3,000,000 steps. Each value represents the training time in hours.
Method
Task
Vanilla PPO
ATLAA-PPO
PA-ATLA--PPO
RAD-PPO
WocaR-PPO
SA-PPO
TDRT-PPO (ours)
window-closs
2.2h
7.8h
8.0h
4.9h
6.0h
3.8h
4.5h
window-open
2.0h
7.7h
8.5h
5.0h
7.
4,2h
Table 14 shows the training time of each method for each task. TDRT-PPO shows superior time efficiency compared to other methods. However, its training cost is higher than that of SA-PPO. This difference is probably due to the fact that TDRT-PPO requires recording the time step of each state in collecting experience.
35

---

Published as a conference paper at ICLR 2026
ATLA-PPO and PA-ATLA-PPO use adversarial training methods. As a result, their training process requires learning not only a robust agent but also an adversarial agent, leading to significantly higher training costs. RAD-PPO incurs higher training costs than vanilla PPO due to the need to compute an approximate minimum reward for regret computation.
In addition, WocaR-PPO must estimate both the regularization term for policy smoothness and the worst-case value. While these computations are not excessively costly, the training cost increases compared SA-PPO and TDRT-PPO, which only compute the policy smoothness regularization term. SA-PPO achieves a lower training cost because it focuses only on calculating the regularization term for policy smoothness.
I IMPLEMENTATION DETAILS
In this section, we provide a detailed explanation of the implementation.
1.1 ENVIRONMENT DETAILS
We conducted the experiments described in Section Z Jusing Meta- World et al. 2020), a benchmark that simulates robotic arm manipulation. All tasks continuous state space and a 4-dimensional continuous action space. In our experiments, we use four tasks: window-close, window-open, drawer-close, drawer-open, faucet-close, faucet-open, handle- press-side, aandle-pull-side door-lock, and door-unlock. The objective of each task is to move specific object to a designated position.
using
share common 39-dimensional
Reward Design The reward design is specific to each task in Meta- World. For example, the reward functions for window-close and window-open tasks are designed independently. Therefore, when attacking victim trained on the window-closs task to perform window-opor open task, this differs from an untargeted attack since does not simply minimize the victim's reward.
1.2 DEFENSE BASELINE DETAILS
This section provides detailed explanation of the baseline defense methods used in our experiments.
(i) SA-PPO Zhang et al. 2020b): This method aims to increase the smoothness of the policy's action outputs regularizer. The difference between TDRT-PPO and SA-PPO that SA-PPO does not apply time discounting in the regularization.
(ii) ATLA-PPO (Zhang et al. 202T): ATLA-PPO is an adversarial training method that alternates between training victim and an adversary. The adversary is trained to generate perturbations that produce the worst-case cumulative reward for the victim by leveraging the SA-MDP framework. The agent is then trained to optimize its policy against this strong adversary, resulting in improved robustness to adversarial attacks.
(iii) PA-ATLA-PPO (Sun al. 2022): PA-ATIA-PPOO extends ATLA--PO by integrating the Policy Adversarial Actor Director (PA-AD) framework, which improves adversarial attack generation. PA-AD utilizes "director" to determine optimal policy perturbation directions and an "actor" to generate corresponding state perturbations, ensuring efficient and theoretically optimal adversarial attacks. Unlike ATLA-PPO, which trains adversary directly using reinforcement learning, PA- ATLA-PPO's decoupled approach enhances efficiency and scalability in environments with large state spaces.
(iv) WocaR-PPO (Liang al., 2022): WocaR-PPO trains the policy to maximize the worst-case cumulative reward under adversarial attacks. Unlike adversarial training methods such as ATLA- PPO, which involve learning alongside adversary, WocaR-PPO uses computationally feasible approach to estimate the worst-case cumulative reward without requiring additional interaction with the environment. Additionally, regularization is applied to improve the smoothness of the policy, focusing specifically on critical states where significant reward drops are likely to occur based on state importance weights.
36

---

Published as a conference paper at ICLR 2026
(v) RAD-PPO (Belaire al. 2024): RAD-PPO is regret-based defense approach. Regret represents the difference between the value without an attack and the value under an attack. RAD- PPO aims to achieve robustness against adversarial attacks by learning policy that minimizes regret at each step. Since regret is defined based on the rewards obtained by the vietim, regret-based defense methods primarily assume untargeted adversaries. Therefore, the robustness of this approach cannot be fully guaranteed against the Sehavior-targetee attack. It is worth noting that the implementation code for RAD-PPO is not publicly available, we implemented ourselves based the details provided in the paper.
1.3 HYPERPARAMETER DETAILS
In this section, we discuss the hyperparameters used in our experiments. In general, our hyperparam- eter settings follow (Zhang et al. 2021).
Architecture. For TDRT-PPO and all defense baselines, we used 3-layer ML.ss with hidden layer sizes of [256, 256] the policy network. Similarly, 3-layer MLPs with [256, 256] were used the Discriminator network for training BIA-ILID/ILO,, This configuration is commonly employed in imitation learning.
Parameter Search. We conducted hyperparameter tuning for the victim agents using grid search. Specifically, we explored the following parameter ranges and selected the models that achieved the highest clean reward (cumulative reward for the original task in the absence of attacks); policy learning rate: (1e-3, 3e-4, le-4, 3e-5), value function learning rate: (le-3, 3e-4, le-4, 3e-5), entropy coefficient: le-5, O. For RAD-PPO and WocaR-PPOC which require training Q-function, we also searched the Q-function learning rate within the range [0.0004, 0.004, 0.00004). Regarding attack methods, during the training of BIA-ILIDE IAIILID,,,,, we explored the following ranges: adversarial policy learning rate: (le-5, 3e-5, le-4, 2e-4, 3e-4), discriminator learning rate: (le-5, 3e-5, 5e-5, le-4, 2e-4, 3e-4). We observed that if the balance between policy learning and discriminator learning deteriorates, attack performance significantly decreases. Thus, hyperparameter tuning for BIA-ILiD/ILfO is crucial for achieving high attack performance. Furthermore, for the Target Reward Maxinization Attack, explored the following ranges for adversarial policy training: policy learning rate: (1e-3, 3e-4, le-4, 3e-5), value function learning rate: [1e-3, 3e-4, Ic-4, 3c-5). We select the models that recorded the highest attack reward (cumulative reward for adversarial tasks under attack).
1.4 THE DETAILS OF TARGETED PGD ATTACK
In this section, we provide a detailed explanation of targeted PGD attacks and present additional experiments. In Section I.4.] show the pseudo-code for targeted PGD attacks and provide specific details about their implementation. In Section I.4.2 we conduct experiments with various attack budgets and analyze the results gain deeper insights into targeted PGD attacks.
1.4.1 IMPLEMENTATION DETAILS OF THE TARGETED PGD ATTACK
We present the algorithm used optimize the false state at each step for the targeted PGD attack in Algorithm 3 The attack aims to find optimal false states that minimize the difference between the victim's ' action and the target policy's action at each step. The algorithm performs T iterations, during which it updates states using FGSM in each iteration. Specifically, it uses the I.2 distance between current victim actions and target actions as the loss function and updates states by scaling in the direction of gradient signs by Estep In all experiments, we set T 30. We also implement random initialization for stable optimization.
1.4.2 PERFORMANCE ANALYSIS OF TARGETED PGD ATTACKS UNDER DIFFERENT ATTACK BUDGETS
We evaluate the performance of targeted PGD attacks across various attack budgets We con- duct attacks against victims trained with PPO without any defense method, using e values of [0.3, 0.5, 1.0, 5.0, 10.0). The adversary's objectives are the same in Section
37

---

Published as a conference paper at ICLR 2026
windarn-loa--1
winlow-npem--2
300
Towardd
700
7000
close-12
drawer-open-v2
4000
Attack Reward
Altack
300
Bedget
Dedget.
Eanceé-close-42
Gucct--pen-122
argns//
EXA k.ftx=00h
Roward
2000
2000
Attack Bedget
Autcc Budget
pde-s-
handle
Reward
10
1.0
100
doer-lock-22
dooe-unlocker2
Meppaai/ airk E.h
Attack Rowardd
-200
100
1000
Bedget
Artacc Dudget
Figure Performance of targeted PGD attacks under different attack budgets The x-axis represents the attack budget and the y-axis represents the attack reward. Each environment name represents an adversarial task. The solid line and shaded area denote the mean and the standard deviation over 50 episodes.
38

---

Published as a conference paper at ICLR 2026
Algorithm 3 Optimization of adversarial states at each step in the targeted PGD attack
Input: Initial state s, target policy network Tig,, victim's policy network æ, perturbation bound number of steps 'I'
2: Output: Perturbed state
3: Caee & c/T
4: 8min f- 8 6, 8max f- 8 +
5: 8 +-
Uniform( (efform((estepppp
Esteps Estep)
6: Agt ~ -8-)
7: for t = 1 do
8:
Ogurret ~ (
9:
L Acurrent Ogg 1
10:
$+ sign( V L) Gstep
$ + clip(,, Smin Smax)
12: end for
Figure shows the experimental results. The environment names in the graphs represent adversarial tasks. most tasks, attack performance increased as the attack budget increased. We hypothesize that this is because, with a larger attack budget, it becomes easier to find fictitious states where the victim's s chosen actions perfectly match those of the target policy. However, in the drawer-open task, attack performance did not improve even with larger attack budget.
Comparing BIA with other attack methods at 0.3, we find that in some tasks, even with a large attack budget, the attack performance did not surpass that of BIA. This strongly suggests that single-step optimization ineffective for behavior-targetee attacks. Therefore, we argue that in behavior-targer tarrettd attacks, it is crucial to train adversarial policy that considers future behavior when performing attacks.
39


Reviews:
[{"review_id": "p4ofHY56Rj", "reviewer": "Reviewer_185Q", "summary": "This paper proposes TDRT, an adversarial training method for mitigating adversarial behavior manipulation, along with a novel black-box adversarial behavior manipulation method, BIA. BIA leverages adversarial imitation learning to train the adversarial policy using demonstrations of the adversary's target behaviors under the State-Adversarial MDP (SA-MDP) setting. TDRT then integrates BIA into the adversarial training process to learn a robust policy against behavior-targeted attack. Experiments show BIA and TDRT are comparable with previous works, SA-RL and SA-PPO, separately.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is well written and easy to follow.\n2. The motivation for developing black-box behavior-targeted attacks is clearly analyzed.\n3. Experiments demonstrate the effectiveness of BIA and TDRT.", "weaknesses": "1. The novelty of this paper is limited. Firstly, regarding BIA, it is clear that BIA is equivalent to SA-RL; both approaches involve adversarial policy learning within the SA-MDP framework. The primary distinction is that BIA requires the adversary to provide several demonstrations of the target behavior, whereas SA-RL necessitates the adversary to develop a reward model for that behavior. Moreover, Theorem 5.1 in BIA closely resembles Lemma 1 from Zhang et al. (2020b), with the only difference being that the adversary's reward function in BIA is derived from the demonstration samples.\nSecondly, concerning TDRT, it is also evident that TDRT is equivalent to SA-PPO. Theorem 6.1 is nearly identical to Theorem 5 from Zhang et al. (2020b), differing primarily in that TDRT employs a discounted form while SA-PPO uses a maximum form.\n\n2. Due to the limited novelty, the performance of BIA and SA-RL, and the performance of TDRT and SA-PPO, are almost the same. Although the authors compare the clean task performance of TDRT and SA-PPO and show that TDRT achieves better clean task performance, it may be mainly due to the different hyperparameter sets. The authors should provide a clearer and fair comparison of the best performance of both SA-PPO and TDRT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aay0Hq3HCO", "reviewer": "Reviewer_vKMc", "summary": "In this paper, the authors propose behavior imitation attack (BIA), a new class of behavior-targeted attack in DRL (where the adversary's goal is to manipulate the victim policy in such a way that it benefits the adversary instead of pure reward minimization) via adversarial manipulations to state observations. Unlike prior work, BIA leverages imitation learning and replaces the state observations of the victim with the falsified state generated by the adversary policy and works under black-box or no-box settings. The authors also present a defense strategy, Time-Dependent Robust Training (TDRT), against such attacks. TDRT introduces a regularization term to suppress the sensitivity of action outputs to state changes. Experiments on Meta-World, MuJoCo, and MiniGrid show that BIA can effectively manipulate victim policies, and TDRT outperforms existing adversarial training baselines in robustness-performance trade-offs.\n\nI believe that the theoretical insights combined with the novelty is enough for acceptance.", "questions": "Q1. What is the computational cost of BIA compared to white-box attack baselines?\n\nQ2. Is TDRT (or policy smoothing) vulnerable to reward minimization attacks? Can we combine adversarial training with policy smoothing to equip agents with a defense that can cover different attack types?\n\nQ3.  When I click to repository in Appendix I, I get the following error: \"The repository is expired\". The authors should ensure that the code is accessible.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "S1. First black-box/no-box behavior-targeted attack: The introduction of BIA allows for attack generation with extremely limited victim access, which aligns with realistic threat models. \n\nS2. Theoretical analysis: The authors provide a theoretical basis for both the attack and defense strategies proposed in the paper. I also find the motivation behind TDRT simple and elegant. \n\nS3. Strong empirical evaluation: Although most of the experiments and valuable ablation studies are pushed to the appendix, results are comprehensive and multiple baselines are compared using different environment settings.", "weaknesses": "W1. The defense method is referred to as both Time-Discounted Robust Training (in introduction) and Time-Discounted Regularization Training (in later sections). The naming should be unified throughout the paper.\n\nW2. The proposed attack may not scale to high-dimensional state spaces, which was also admitted by the authors in the limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eXMi49zx4W", "reviewer": "Reviewer_kHJs", "summary": "This paper introduces behavior-targeted attacks, which steer a victim RL agent’s policy toward an adversary-desired behavior rather than simply reducing reward. It proposes (1) Behavior Imitation Attack (BIA) — an imitation-learning–based attack operable in black-box or no-box settings, and (2) Time-Discounted Regularization Training (TDRT) — a defense that suppresses early-trajectory policy sensitivity via time-weighted regularization.", "questions": "1. I would like to see a discussion between the behavior imitation attack and the backdoor attack. This should be an important related work in this work.\n\n2. I would like to see some results on diffusion model-based defenses against behavior imitation attacks.\n\n3. Could the proposed methods extend to a large state space environment, such as images based environment, such as Atari games? I know it is a challenging extension, but do the authors have any potential ideas on this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Innovative attack under limited access: BIA elegantly converts the problem into an imitation-learning formulation, removing dependence on victim parameters.\n\n2. TDRT’s time-discounted regularization is motivated by a provable bound (Theorem 6.1) and empirically validated.\n\n3. Experiments span multiple continuous-control and grid environments, with clear comparisons against strong baselines (ATLA-PPO, SA-PPO, RAD-PPO, WocaR-PPO).\n\n4. The paper is well-structured, with a precise threat model, proofs in appendices, and reproducibility statements including algorithm pseudocode.", "weaknesses": "1. Diffusion model based defenses[1][2] have been proposed recently to fight against state adversarial perturbations. Could the author compare them with the proposed defense?\n\n2. The proposed method currently do not scale to image based input RL environments such as Atari games.\n\n[1] Z. Yang and Y. Xu. DMBP: Diffusion Model–Based Predictor for Robust Offline Reinforcement Learning against State Observation Perturbations. ICLR, 2024.\n\n[2] X. Sun and Z. Zheng. Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations. ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6HTGNaLK7", "reviewer": "Reviewer_Yovk", "summary": "This paper proposes a new state-perturbation attack against a trained RL agent, aiming to manipulate the agent into behaving according to a target policy chosen by the attacker. The main idea is to use imitation learning techniques, such as GAIL, to learn the attack policy from demonstrations of a target policy, so that the attacker does not need to access the agent's policy directly. Further, the paper proposes a new defense adapted from the SA-PPO algorithm, using time-discounted regularization to train a smoothed policy.", "questions": "1. Algorithm 1 is a straightforward adaptation of GAIL, and it can be directly derived from the objective (2), without Theorem 5.1 and the discussions in Section 5. Why are those discussions useful? \n2. Why should the agent consider target behavior manipulation? It seems that the agent should always consider the worst-case scenario to achieve robust defense, and there is no need to distinguish target behavior manipulation and reward minimization from the agent's perspective. \n3. Is time-discounted regularization still useful when gamma is close to 1, as typically considered in previous work?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This problem of adversarial behavior maniplation through state perturbation has been considered recently. The main contribution of the paper is showing that the attacker's problem can be viewed as a MDP with carefully defined reward and transition functions derived from the agent's MDP, and utilizing behavior imitation to avoid white-box access to agent's policy. Another contribution of the paper is the obsevation that the state changes in the early stages of trajectories have a great influence on the agent's overall performane in continuing tasks and the time-discounted regularization scheme derived from that.", "weaknesses": "1. While the paper does not need white-box access to the agent's policy, it requires demonstrations from the target policy, which might not be easy to get in practice without strong domain knowledge, including the environment and the agent's policy. In particular, behavior manipulation includes reward minimization as a special case, treating a reward minimization policy as the target policy. \n2. Algorithm 1 is a straightforward adaptation of GAIL, and it can be directly derived from the objective (2), without Theorem 5.1 and the discussions in Section 5. Why are those discussions useful? \n3. The defense objective (10) does not make sense to me. From the agent's perspective, it should just try to maximize its own reward. Why should it try to minimize the attacker's gain? This only makes sense in the zero-sum setting. Further, since behavior manipulation includes reward minimization as a special case, the agent should consider the worst-case scenario to achieve robust defense. In that sense, there is no need to distinguish target behavior manipulation and reward minimization from the agent's perspective. \n4. The observation that the agent's policy is more sensitive to early stages of trajectories looks like an easy consequence of using a discount factor in continuing tasks, and this should apply to the reward minimization case as well. However, this result does not apply to episodic tasks and when the discount factor is very close to 1, which are commonly considered in the previous work, such as SA-PPO.  \n5. The evaluation is not very convincing. The paper does provide baseline results for other defenses except in the Meta-World environment. Further, it ignores important recent defenses such as:\n- Li et al., Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error, ICML 2024.\n- Sun et al., Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations, ICLR 2024.\n- Yang et al., DMBP: Diffusion Model-Based Predictor for Robust Offline Reinforcement Learning against State Observation Perturbations, ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)