On Minimizing Adversarial Counterfactual Error in Adversarial Reinforcement Learning
Authors: Roman Belaire
Arunesh Sinha
Pradeep Varakantham
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement learning
robust reinforcement learning
adversarial robustness
partially observable markov decision problems
Primary area: reinforcement learning
TL;DR: Adversarially robust reinforcement learning for observation-level perturbations. We provide novel robust strategies by explicitly considering the partially-observable nature of adversarial RL.

Abstract: Deep Reinforcement Learning (DRL) policies are highly susceptible to adversarial noise in observations, which poses significant risks in safety-critical scenarios. The challenge inherent to adversarial perturbations is that by altering the information observed by the agent, the state becomes only partially observable. Existing approaches address this by either enforcing consistent actions across nearby states or maximizing the worst-case value within adversarially perturbed observations. However, the former suffers from performance degradation when attacks succeed, while the latter tends to be overly conservative, leading to suboptimal performance in benign settings. We hypothesize that these limitations stem from their failing to account for partial observability directly. To this end, we introduce a novel objective called Adversarial Counterfactual Error (ACoE), defined on the beliefs about the true state and balancing value optimization with robustness. To make ACoE scalable in model-free settings, we propose the theoretically-grounded surrogate objective Cumulative-ACoE (C-ACoE). Our empirical evaluations on standard benchmarks (MuJoCo, Atari, and Highway) demonstrate that our method significantly outperforms current state-of-the-art approaches for addressing adversarial RL challenges, offering a promising direction for improving robustness in DRL under adversarial conditions. Our code is available at https://github.com/romanbelaire/acoe-robust-rl.

Published as a conference paper at ICLR 2025
ON MINIMIZING ADVERSARIAL COUNTERFACTUAL ER- ROR IN ADVERSARIAL RL
Roman Belaire
Arunesh Sinha
Singapore Management University
Rutgers University New Brunswick, NJ a arunesh, s nhe rutggrrrrrrrr..... edu
Singapore
rbelaire. 2021 @ @phdcs smu, edu.sg
Pradeep Varakantham
Singapore Management University
Singapore
pradeepvesee smu, edu. sg
ABSTRACT
Deep Reinforcement Learning (DRL) policies are highly susceptible to adver- sarial noise in observations, which poses significant risks in safety-critical sce- narios. The challenge inherent to adversarial perturbations is that by altering the information observed by the agent, the state becomes only partially observ- able. Existing approaches address this by either enforcing consistent actions across nearby states or maximizing the worst-case value within adversarially per- turbed observations. However, the former suffers from performance degradation when attacks succeed, while the latter tends to be overly conservative, leading to suboptimal performance in benign settings. We hypothesize that these limi- tations stem from their failing to account for partial observability directly. To this end, we introduce novel objective called Adversarial Counterfactual Error (ACoE), defined on the beliefs about the true state and balancing value optimiza- tion with robustness. To make ACoE scalable in model-free settings, we propose the theoretically- grouudddd surrogate objective Cumulative-ACof (C-ACoE). Our empirical evaluations on standard benchmarks (MuJoCo, Atari, and Highway) demonstrate that our method significantly outperforms current state-of-the-art approaches for addressing adversarial RL challenges, offering a promising direc- tion for improving robustness in DRL under adversarial conditions. Our code is available http:://gggttbbb.coo/mmaababaaap robust-rl.
INTRODUCTION
The susceptibility of Deep Neural Networks (DNNs) adversarial attacks their inputs is well-documented phenomenon in machine learning (Goodfellow et al. 2014 |Mdrry et af. (017). Consequently, Deep Reinforcement Learning (DRL) models are also vulnerable input perturbations, even when the environment remains unchanged (Gleave et al. 2019 Sun et al. 2020: Pattanaik al. 2017). As DRL becomes increasingly relevant real-world applications such as self-driving cars, developing robust policies is of paramount importance (Spielberg et al., 2019 Kiran et al. 202T). An example highlighted by Chen al. (2018) successfully alters stop sign both digitally and physically to deceive an object recognition model, demonstrating the ease and potential dangers of such adversarial attacks.
Adversarial retraining, which entails inserting adversarial perturbations to the replay buffer during training, effectively enhances the robustness of deep reinforcement learning (DRL) against known adversaries (Gleave al. 2019 Goodfellow et al. 2014 Pattanaik et al. 2017 Sun al.) 2023). However, this approach often fails to generalize well to out-of-sample adversaries (Gleave et al. 2019 Guo et al. 2023). More importantly, it is well-known that stronger adversaries
always be found (Madry et al. (2017) and that the high-ding dice-smnnniinal observation spaces real problems offer overwhelming number of adversarial directions (Korkmaz 2023
Liu et al. 2023 Furthermore,

---

Published as a conference paper at ICLR 2025
due to issues such as catastrophic forgetting, naive adversarial retraining in reinforcement learning can result in unstable training processes and diminished agent performance (Zhang et al., 2020) This highlights the need for algorithms that not tailored to niche adversarial perturbations but generally robust. Rather than develop a policy that is value-optimal for a set of known adversarial examples, our goal is identify and mitigate behaviors and states that introduce unnecessary risk. A widely-recognized method to achieve general robustness is the maximin optimization, which seeks to maximize the minimum reward of policy (Everett al., 2020; Liang et al., 2022). While this approach does enhance safety, it often sacrifices the quality of the unperturbed solution to improve the worst-case scenario.
Another prevalent robustness mechanism strengthens "non-adversarial value" optimizing policies (i.e. vanilla policies) by incorporating adversarial loss regularization terms, ensuring robust policies are close to the tron-adversarial value" optimal policies. This aims to ensure that actions remain consistent across similar observations, thereby reducing the likelihood of successful adversarial attacks (Oikarinen et al. 2021 Zhang et al. 2020; Liang et al. 2022). However, prior empirical findings indicate that these methods still leave policies vulnerable when attacks do succeed (Belaire et al. 2024), the observation space is high- gigh-dimessionll it is not feasible to ensure all similar observations have similar actions.
Adversarial perturbations make the ground truth partially observable and this aspect-thougl acknowledged-has not been explicitly reasoned within existing work, except recently in Liu et al. (2024); McMahan et al. (2024), the best-performing of which is called Protected (Liu et al. (0224). However, the Protected framework requires multiple adaptation runs at test time to achieve better performance than existing work. The requirement for multiple execution runs in the presence of an adversary at test time is not viable in self-driving cars and other real-world scenarios. To that end, we introduce a novel objective called Adversarial Counterfactual Error (ACoE), which calculates the error due to adversarial perturbations by explicitly considering the belief distribution over the underlying true state.
Contributions:
In a significant departure from previous research, we address the partial observability present in adversarial RL problems (due to adversarial perturbations) by introducing the concept of Adversarial Counterfactual Error (ACoE), which is defined based on beliefs about the underlying true state rather than the observable state only.
We introduce scalable surrogate for ACoE called Cumulative ACoE (C-ACoE) and estab- lish its fundamental theoretical properties, which aid in developing strong solution methods.
We develop mechanisms to minimize C-ACoE while maximizing expected value by leverag- ing established techniques from Deep Reinforcement Learning (e.g.. DQN, PPO).
Finally, we present comprehensive experimental results on benchmark problems (MuJoCo, Atari, Highway) employed in adversarial RL area to demonstrate the effectiveness of our approaches compared to leading methods (e.g., Protected, RADIAL, RAD, WOCAR) for adversarial reinforcement learning. We test against potent myopic attacks (such as MAD, PGD) and more advances macro-strategic adversaries such as PA-AD (Sun et al. 2023).
RELATED WORK
Adversarial attacks in RL: Deep RL is vulnerable to attacks on the input, ranging from methods
targeting the underlying DNNs such as an FGSM attack (Huang et al. 2017 Goodfellow et al., 2214)) tailored attacks against the value function (Kos and Song. 2017 Sun et 2020), or adversarial
behavior learned by an opposing policy (Gleave et 20.1 F Everett et a 2020) Oikarinen et al.
2021: Zhang et al. 2020). We compile attacks on RL loosely into two groups of learned adversarial
policies: observation poisonings (Gleave et al.) 2019 Sun et al. 2020; Lin et al. 2017 Guo et al..
2023) and direct ego-state disruptions (Pinto al. 2017 Rajeswaran et al. 2017). Each category has
white-bor counterparts that leverage the victim's 's network gradients to generate attacks (Goodfellow et al. 2014; Oikarinen et al. 2021: Huang et al. 2017; Everett et al. 2020h. In this work (similar to existing works highlighted in this section), we focus on defending against the former group, observation poisonings, with both white-box and black-box scenarios.
2

---

Published as a conference paper at ICLR 2025
Adversarial Retraining and Adversary Agnostic Approaches: In adversarial retraining, adversarial
examples are found or generated and integrated into the set of training inputs (Shafahi et al.
2019
Ganin et al.; [016 Wong et al. 2020: Madry et al.] 2017: Andrusshhhoo and Flammarion 2020
Shafahi et al. 2020). For a comprehensive review, we refer readers to Bai et al. (2021). In RI,
research efforts have demonstrated the viability of training RL agents against adversarial examples
(Gleave et al. 2019 Bai et al. 2019 Pinno et al. [017]] Tan et al. 22200] Kamalaruban et al. 2020
Sun et al. 2023). Training RL agents against known adversaries is a sufficient defense against known attacks; there are effective adversarial retraining methods grounded in many disciplines such as
curriculum learning (Wu and Vorobeychik] [022). policy- colicy-adversary training (Sun al., (023) and
behavior cloning NNie et al. (022). However, novel or more general adversaries remain effective against this class of defense (Gleave et al., 2019 Kang et al. 2019). Furthermore, they often take longer to train (needing to train both victim and adversary policies). The adversarial retraining technique PA-ATLA-PPO (Sun et al., 2023) reports needing 2 million training frames for MuJoCo- Halfcheetah. For comparison, both RAD (Belaire et al. (024) and WocaR-PPO (Liang et al. 2022) are adversary-agnostic methods, and require less than 40% of the training frames. This paper focuses on adversary-agnostie defenses that do not train against specific adversaries in the environment.
Robust Regularization: Regularization approaches (Zhang et al., 2020 Oikarinen et al., 2021 Everett et al., (020)) take vanilla value-optimized policies and robustify them to minimize the loss due to adversarial perturbations. These approaches utilize certifiable robustness bounds computed for neural networks when evaluating adversarial loss and ensure the probability an attacker successfully changes the agent's ' actions is reduced using these lower bounds. Despite lowering the likelihood of a successful attack, a successful attack (i.e., two close states have different actions creates vulnerability) is still just effective. Previous works suggest the need to learn safe trajectories via robustness- specific objectives, rather than robust decision classifier only (Belaire et al.] 2024 Liang et al., 2022
Li et al.) 2024), such that successful attacks (if any) are also less effective.
Measuring and optimizing a regret value to improve robustness has been studied previously in uncertain Markov Decision Processes (MDPs) et al. [203]] Rigter et al. 202T
Robust Control:
Adulyasak et al. 2. In RL, Jin et al. (2018) establish Advantage-Like Regret Minimization (ARM) policy gradient solution for agents robust to partially observable environments. In continuous time control. Yang et al. (023) study the composition of robust control algorithms with a robust predictor of perturbed system dynamics. In contrast to policy regret, we form beliefs about true states and minimize the cumulative adversarial counterfactual error (a novel notion of action-regrett to ensure robust policy is computed, also recognizing the partial observability present in the problem.
Game Theoretic Approaches: A thread of approaches (McMahan et al. 2024 Liang et al. 2024) have employed partially observable stochastic games represent problems of interest. A key advantage of game theoretic approaches is their ability to reason about adversaries. However, they assume that an adversary is always present-this can result in conservative solutions-and typically are computationally heavy. We do not use equilibrium concepts to ensure there is good balance between robustness and anvesrrlera value" maximization. Instead, our rewk-reward balance is computed based on the empirical belief about the adversary obtained from observations.
Partially Observable Adversaries: Several prior works (Jin et al. 2018 Zhang et al., 2020 Liu
et al. 2024) have acknowledged and consider that adversarial observation perturbations make the underlying state partially observable. This has resulted in improved results. However, there are a few fundamental differences on how partial observability is considered in the most recent work (Liu et al.) 2024) and our contributions:
Partial observability is captured using history of observations that does not consider that this partial observability is being driven by an adversary (i.c., with intention). The partial observability present in adversarial RL is not the same as in Partial Observable MDPs, where partial observability is facet of the agent sensor (that is only stochastic, not adversarial). In our work, our belief state computation (to account for partial observability) explicitly considers that an adversary is driving the observation.
In training, they compute a set of non-dominated policies to execute at test time. Then, they do test time adaptation, performing regret minimization over multiple (800) complete runs of the policy against the adversary. This is effective, though unfortunately impractical in domains such as autonomous vehicle control, where adapting adversary after
3

---

Published as a conference paper at ICLR 2025
catastrophe is not acceptable. Thus, such test time adaptation has not been utilized in any of the existing works, ours included.
They do not adapt at every time step (which is feasible in RL settings based on observations), rather waiting until the end of each episode to adapt their policy meta-weights Because time step-wise interaction and adaptation fits within RL settings, we consider the adversarial susceptibility of actions at every time step based on estimated belief and act accordingly.
ADVERSARIAL COUNTERFACTUAL ERROR (ACoE)
In this section, we define the ACoE objective for the Adversarial Reinforcement Learning (RL) problem. Intuitively, ACoE refers to the difference in the expected value obtained by a defender in the absence of adversarial perturbations versus in the presence of an adversary. It should be noted that in the case of adversarial perturbations, the defender only receives the altered state, and no information that is verified be uncorrupted. By minimizing the ACoE objective in conjunction with maximizing expected value, we aim to derive policy that provides a good trade-off between robustness (against adversary perturbations) and effectiveness (accumulating reward).
Expected value without adversarial perturbations, V(s):
In the case without adversarial perturbations, the defender's problem is one of an infinite horizon MDP. Formally, we define the MDP (S, T, R, where S is the state space, A is the action space, T(s') 8, a) is transition probability, R(s a) is the immediate reward, and 7 is the discount factor. Without loss of generality, we assume R(s, a) E [0, 1]. For case of presentation, state and actions in the mathematical sections. The aim in the MDP is to choose actions at every time step (specified as a policy :)) that maximize the value function V. In infinite horizon MDPs, the optimal policy is memoryless and stationary, i.e. a function of only the current state. However, to be more general and keep consistent notation with the case where there is an adversarial partially observable case below, we use / as the current information state, i.e., I is the sequence of observed states and actions up the present, and the policy computes the action as a function of I, (().. Note that this is without loss of generality, as the optimal policy in an MDP will simply ignore the history preceding the current state. Then, the value for a policy is given by
assume discrete
(ss) R(s, (1)) +
(V
Expected value with adversarial perturbations, (b):
In the case of an adversarial perturbation, the defender only receives an altered observation, providing only partial information about the underlying true state (i.e., the true state near the perturbed state). Formally, we define the adversary's 's policy possible distributions over S; we also abuse notation slightly to indicate the perturbed random state as vs)). We follow the standard assumption in adversarial learning that the perturbed state is close to the true underlying state, i.c., [s)) 8 20 c. This is an example of a one-sided Partially Observable Stochastic game (POSG) (Horák et al.) 2023) in which the adversary has full observability while the defender does not observe the underlying state and only observes the perturbed state. It is well known (Horák et al. 2023) that with fixed adversarial perturbation policy (possibly randomized), the defender's 's problem reduces to Partially Observable Markov Decision Process (POMDP).
function, v S + A(S), where (SS) denotes all
POMDP is MDP where the state is only partially observed. This partial observability is captured using an observation space O and observation probability Pa(o s', a) that specifies the probability of observing o given true state s' obtained on taking action a. Further, a POMDP is known to be equivalent to belief' state MDP (Kaclbling et al. 1998) where states are beliefs over the underlying states in the POMDP. belief state, b is probability distribution over underlying states, s, where Σ, b(s) = 1. On taking actions, this belief state changes and is computed by using standard Bayesian update:
B'(s)) Po 8 E, T(s' ss))((s) where Polo b,a) = Po b.a)
Σ Poloo s(,)) T(S (sss
We will employ a short form to represent the above update, 8' = SE((,, o, a). As the belief update requires knowledge of the model (transition function), our initial mathematical analysis is in
4

---

Published as a conference paper at ICLR 2025
model-based framework. An optimal policy in POMDP can be function of the belief. However, it is known that for POMDPs belief is a sufficient statistic for information state I, so we can consider the more general policy that depends on I, without any loss of generality. We denote by U the value function of this POMDP for policy π.
(b) R(6, x (1)) + Polo ( (((())(((E(600..
The partial observability exhibited in adversarial RL has a particular structure in which the observation space O is the same the state space S, and the observation probability function Po(o a) is governed by the adversary's s perturbation policy. More specifically, in our problem, the observation probability depends only on the true state and not the defender action, thus, we write P% s), but note that 6' = (b, a) still depends on due to the use of transition T. Note that the non- adversarial case can be considered special case where the adversary policy is the identity function id, and then Pid (o| s) = I[(o s) for indicator function 1. As the observation space O = S, we will often use the notation s, to refer to an observation as s, E S where the subscript is used to denote that this is an observation. In particular, any distribution over the observation space is a distribution over the state space.
Ad Adversarial Counterfactual Error, ACoE: We analyze the difference in return V U obtained in the non-adversary case (denoted by V) and adversary case (denoted by U) using common policy i in each case. We term V - U as Adversarial Counterfactual Error (ACoE). As the optimal policy depends on different information structures in these two cases, to compare these cases with the same policy we have already chosen to generalize the policy function of the information state I. We write the value functions starting with the currently observed belief, where the non-adversarial case is the true state itself. For notational ease in the later sections, we will write s, to represent the current observation, which particularly emphasizes that in our problem, the observations are themselves part of the state space. Further, in particular domain, o S, thus, P. b, π (()) specifies probability distribution over states. Thus, by renaming variables and dropping the dependence of observations on actions, we rewrite Σ. (o (SE)) o, x x{))) as E.g-Pot (SS s .T ([))]] Then, for both the adv-adveryr and adversary scenarios, following standard MDP and POMDP facts, we have a recursive form below:
(sa.) = R(s0, क (T)) F (t) (V(.)
(b) R(b, x(1)) + Pol. (UE s/ 7 (())]
ACoE is defined as V(s.) - U(b).
We also use an additional shorthand notation of To b, a) denote the joint probability distribution of S and b' specified by the sampling process: s. ~ Po b b, = SE((, s., a). We define the following important quantity:
Definition 3.1 (Cumulative Adversarial Counterfactual Error (C-ACoE)). Define C-ACoE as
8(s0, b) R(so, x()) R(b, (T)) + E.Ee, (x)())00(660
(1)
any observed Theorem 3.2. Let K = max (s) and assume TV a), Po(: b,))) an
state 8,, belieft, b, and action a in the same time step, then
7 KE
[V<.)) - ((() 5(s..
The above result shows that there are two parts to ACoE, the uncontrollable part with the TV distance captures structural differences in the transition without attack and transition induced by the attack, while the controllable part, C-ACoE term 6(s.. b) captures long term return difference due to the adversarially induced transition. In the appendix, we delve more into the structural difference in transitions by utilizing Wasserstein distance instead of Total Variation, TV distance. The above results also suggest that apart from the inherent structural differences, minimizing C-ACoE 8(s,, b) can be effective in ensuring that returns in the adversarial scenario are close to the non-adversarial scenario, which we explore in the next section.
Since the structural differences in transition are not controllable by the defender agent, we focus on minimizing the C-ACoE for the defender. Furthermore. ensure that the effectiveness of the policy in accumulating rewards is high, we minimize C-ACoE while simultancously maximizing the non-adversarial expected reward.
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: 8-PPO
Initialize policy network weights 01. value network weights Φ1, and 8-network weights t/ X
Set boousttess-hyperppraaaeee
for iteration k E {1..... } do
Collect of trajectories D; by running policy ne, multiple times for T steps
Estimate rewards-to-go fG and C-ACof-to-go & at all time steps for all trajectory in Dj.
Compute advantage estimates A using Generalized Advantage Estimator (Schulman et al. 2016), based on R and Vax
Compute C-ACoR Advantage Ac,t A, Ad;
Update policy parameters to 0x+1 by maximizing the PPO-clipped Schulman et al. 2017 form of Ac,t
Update Øk+1 argmin 1De ErED, (v R1)²
Update v = argming D.IT ΣrED, Ei=oldp(se) 8,)2 10
OPTIMIZING C-ACOE ALONG WITH NON-ADVERSARIAL EXPECTED REWARD IN ADVERSARIAL RL
In RL settings, we do not have the model and hence the transition dynamics T are not available. Thus, computing 5(s.. b) exactly not possible, as the belief depends on knowledge of transition probabilities. However, our problem presents a structured scenario where the observation depends only on the current true state and uncertainty entirely due to adversarial perturbation. has been stated in literature and is also intuitive that adversarial perturbations effective in causing harm when they induce a large enough change in the defender's action distribution (Oikarinen et al. 2021 Zhang et al. 2020). Thus, we propose to derive surrogate belief based on the observed state s, in conjunction with reasoning about how the adversary might have forced this observation to arise. We present couple of such belief constructions here.
Using the full history of observations and actions (represented as the information state, I) as an input to the policy is computationally expensive implement. Prior approaches have used variety of approximations (Azizzadenesheli et al. 2018); we adopt a simple measure (Müller and Montufar 2021; Kober et al. (2013) where we restrict solutions to the set of policies that depend just on the current observation. Next, note that if 6 depends on so only, then 8(so, b) is function of s, only. Hence, we redefine the C-ACoE as
6(s.) = R(s,, π (s.)) ((b(so),((sb)) + (s)) (=)(=)))((({.))
(2)
We note that the underlying true state s' is not observed, but estimating the second term on the RHS above requires only samples of observation s which are available from the simulator. In this form, C-ACoE also satisfies the Bellman optimality structure (as stated formally in the following proposition) and hence allows for incorporating the minimization of (ss)) in standard RL techniques.
Proposition 4.1. Let 8* (so) be the minimum C-ACoE value from observation s Then, 8t (so) = min (ain(R(so.,) (s.)) R(b(so): a) + "Ess s.a) (s()))
Algorithm [shows our adaptation of PPO for optimizing 8 along with maximizing V. The steps for maximizing V follow standard steps in PPO leading to the standard advantage Å, in line 7. We also compute the C-ACoE-to-gg from the sampled trajectories (line 5) and use it to augment the standard advantage A, in line 7 (we need to minimize C-ACoE, hence the negative sign before d.). Line a standard PPO step to update the V network and we do so similarly for the 8 network in line 10. We found that computing an advantage-like term for 5 did not improve performance, thus used only C-ACoE-to-gee describe two possible belief constructions given the observed state s.-
similar adaptation is also done for DQN, presented in the appendix. Next, we
Adversary-Awar Belief Estimation (A2B): We aim to assign a belief to states in neighborhood N (ss)) of observation, 8, where Ns.)) {s ||s soll S c). N(s.) is restricted bound

---

Published as a conference paper at ICLR 2025
given established adversarial perturbation practices. We know that adversarial perturbation from state s to state so is an effective attack when the action distribution (s)) and o ($x) are quite different. Based on this fact, we form belief:
(
b(s)
E ENE (DI(((())I(O))))
Addeessar-AAttack--aar Belief Estimation (A3B): Different from A2B, we assign scores to states in (s.) based on assumptions about adversarial preference. These scores depend surrogate attack v, for which we use a 50-step PGD attack; quick empirical checks show this to find the worst- case bound of the Lo -nom ball in nearly every state. We assign score z(s) to state (s.) the action distributions at possibly perturbed observation 8, and the state s) (the KL divergence of actions distribution at (()) and s). Then, belief is assigned to state s' depending on the score 2 by softmax operation:
that is a ratio of: (the KL divergence
c=(+)
(xL(T((o)((((s))
b(s) EvEN(O.) c=(x") where z(s) = (xL(((v(s)]]
The intuition for the above formulation of score is that if the true state 8, the adversary should prefer to provide v(s) with a high KL divergence between action distributions at v(s) and s, but since we observed so, the ratio of KL divergences in score =(s) measures how effective the change s to 8, is, compared to the change s to v(s). Any candidate true state s has low score if 8, is not an effective attack from state s. Thus, A3B reduces the scores (weights) of states that are unlikely adversarial choices based on the policy л. Then, optimizing C-ACoE using A3B beliefs coupled with non-adversarial value maximization allows balancing unperturbed performance with robustness, as highlighted earlier in the introduction.
For visual explanation of the logic of A3B, con- sider Figure [ This figure shows two neighbor- hood states 81 and 82 which could potentially be the underlying true state, given the observed state so. Subsequently, (s1) contains a worst-PGD per- turbation s v((1) and ((22) similarly contains s/2 = v(81). Even though sl2 may be close in Eu- clidean distance to So. it is possible that
N(s2)
N(s.)
((1))
$2
S1
DKL (T<s))
>> (xz(((s))](((ss))
leading to a small score 2nz (closer to 0) for 82- This is intuitive,
an adversary will likely not perturb $2 to So, due to the existence of the more disruptive attack szz. Similarly, the score 2s for Ro DeL (=T(s)]]=(( (s1)) )), which is intuitive as 8, results in same amount of change in action distribution as 8/.
Figure 1: A3B belief construction. Let the dotted line 8,8, have magnitude representing the damage when perturbing 8; - 83. In this example, our method should discount the pos- sibility that v($2)= = 80. and lessen the score z(s2).
81 can be close to due to (=))))((((()))))
Continuous State Sampling: One issue to consider above is when the state space is continuous. In such scenario, still form finite set N(s.) by uniformly sampling given number (hyperparameter) of samples from the continuous set C' = {s=
- < e}. From the definition of d (Eq. 2. we use b estimate R(b, a). Our true value of this is R = R(b, a) Ssec R(s, a)p(ss where the probability density p(s) = e-(*)/ Seec e=(s) ds. In contrast, we sample states from uniform distribution U with probability density given by u(s) = (C) where vol is the volume of set C' and estimate Å EveN(aa) R(e' R(e',a)) a)eeta))
EveNGo) en(e)) We show a result in the appendix that justifies the
estimate R by showing that the expected value of this estimate is close to the true required value R.
Recurrent State History: A3B is primarily proposed as an adversary-aware method of deriving belief about true states based on the current observation. However, this approach can be adapted to consider history of observations, albeit with higher computational burden. We provide extended A3B definition with multistep observations and additional evaluations of this extended A3B using an L.TTM network in the Appendix.
7

---

Published as a conference paper at ICLR 2025
Table 1: Experimental results versus myopic adversaries. Each row shows the mean scores of each RL method against different attacks. The most robust scores are in bold. Our approaches are A2B and A3B, which are highlighted.
Method
Unperturbed
MAD
PGD
Unperturbed
MAD
PGD
Seghwa-fast-vee
merge-v0
PPO
24.8::5,42
13.63+19.85
15.21
14.94
10.2:10.02
10.42:10.95
CARRL
24.4 15.4
12.43
12.6:0.011
12.6+
12.02:40.01
RADIAL
28.55=00001
1.4
14.97=3.1
14.86=0.01
11.2990.01
11,04#0.91
WocaR
21.49:1.001
6.15+00.3
6.19=0.4
14.91=0.04
12.01=0.28
11.71#0.221
RAD
21.01+00.01
20.59+4.1
20.02++0.01
13.91+0.01
13.90+0.01
11.72+0001
A2B
24.8+00011
23.11+0.01
20.8++12.6
14.91+0.01
14.23++088
12.92+H0.13
A3B
23.8+0.01
23.2110.01
22.61114.1
14.9110.17
14.88.1.117
14.8910.17
EXPERIMENTS
We provide empirical evidence show the effectiveness of our proposed method. particular, we want to investigate whether A2B and A3B improve over leading adversarial robustness methods on established baselines, and what aspects of C-ACoE contribute to a viable defense against strategic adversaries.
5.1 EXPERIMENT SETUP
We evaluate C-ACoE methods on the standard Atari (Bellemare et al. 2013) and MuJoCo (Todorov eet al. 2012) domains, and additionally the Highway simulators (Leurent 2018) to demonstrate real problems interest. In the Mujoco and Highway tasks, the agent earns score by traversing distance without incurring critical collisions. Atari tasks are game-dependent. We use standard training setup seen in (Oikarinen et al. 2021 Liang et al. 2022 Belaire et al. 2024), and detailed in Appendix C.
We compare C-ACoE optimization methods (A2B, A3B) the following baselines: PPO (Schulman 2017): CARRL, a simple but robust minimax method (Everett et al. 2020); RADIAL, a leading regularization approach (Oikarinen al.) (021);: WocaR, worst-case aware value maximization (Liang
et al.
et al.
2022): ; RAD, method minimizing notion of regret (Belaire et al. 2024); and Protected (Liu
2024). We test all methods against two greedy attack approaches of reward-ninimizing policy al.
adversaries and gradient attacks. We evaluate each method's 's PPO implementation in the Highway and Mujoco domains, and DQN implementations in Atari tasks. Additional comparisons including to
few more baselines, namely BCL (Wu and Vorobeychik 2022) and CAR-DQN (Li et al. 2024),
in the Appendix.
We wish to specifically address the comparison with Protected (Liu al. 2024). Protect does regret minimization (EXP3) over multiple rounds (each round full policy episode) and the weights are updated at test time based empirical return in each round. As stated earlier, this has a major advantage against all other approaches the literature, which do not do any test time adaptation, and unfortunately makes Protected impractical for safe RL applications. To indicate this, the results of the original Protected are presented but grayed out (and not compared to when highlighting best result) in Table 3 The test time adaptation also results in Protected having significantly higher unperturbed score in some of the domains (e.g., HalfCheetah, Walker2d, Ant) even when compared PPO. Therefore, for a fair comparison to all the adversarial RL approaches, we also provide comparison against a variant of Protected, referred to as Protected!, where there no test time adaptation. Further details of Protected and additional comparison presented in Appendix.
Protected Baseline:
Myopic Adversaries: We test the adversarial robustness of each method against adversaries that we term as "greedy" or myopic, meaning that they compute worst-case attacks for given time step. Following the set up employed in existing works, we measure a 10-step PGD attack (Madry et al. 2017) with 0.1, and MAD attack (Zhang al.] (2200) with 0.15. We evaluate both MAD and PGD attacks as they represent two distinct attack directions (MAD is eoward-based while PGD gradient-based).
8

---

Published as a conference paper at ICLR 2025
Table Experimental results versus myopic adversaries in Atari domains, formatted the same as Table Methods are evaluated as their corresponding DQN implementations.
Method
Unperturbed
MAD
PGD
Unperturbed
MAD
PGD
Pong
Freeway
PPO
21.0:10
-2.00+0.077
-19.0+1.00
29 + 3.0
4 + 2.31
2::2.0
CARRL
13.0 112
11.0+00.010
6.0+1.22
18.5+0.0
19.1 1:220
15.4-00.22
RADIAL
21.0:10
11.0#2.9
21.0=
33.240.19
29.0
24.0=:0.10
WocaR
21.01
18.7 #0.10
0.21
31.24:0.1
19.8#3.81
28.143.24
RAD
21.0:10
14.0 0.04
= 2.40
33.2+0.18
30.0++.23
27.7+
A2B
21.0=
20.1
21.0
33.2+0.18
30.1=0043
30.8+1.511
A3B
21.0:10
20.810077
21.0
33.210.18
31.0.10.87
31.111
Table 3: Experimental results versus myopic adversaries in Mujoco domains, formatted the same as Methods are evaluated as their corresponding PPO implementations. Note: the Protected method requires test time adaptation rounds to achieve full results. The Protected method without test time adaptation labelled as Protected".
Table I
Method
Unperturbed
MAD
PGD
Unperturbed
MAD
PGD
Hopper
Walker2d
PPO
4128 + 56
1110#:32
128+1005
5002 + 20
680+:1570
730+:262
RADIAL
3737+75
2401#13
3070+31
5251=10
3895=128
3480+33.1
WocaR
3136=463
1510 519
2647 #310
4594=974
3928+11305
3944
Protected
3652-1108
2512:1392
22211:775
6319-131
5148:1111
4720:E 1508
Protected
35734-81
6398+1665
22154-98
5019 + 87
3887 492
3613
487
RAD
34734223
27831325
3110=30
4743178
3922+426
41361139
A2B
3710#11
3240141
3299428
4760116
46361187
4708 184
A3B
3766++23
3370+275
3465+117
5341+60
5025+994
239
HalfCheetah
Ant
PPO
5794 + 12
1491+-20
-27+
5620+;29
1288+
1844+330
RADIAL
4724+17
4008+4500
3911+129
5841133
3210+380
1221+121
WocaR
5120-112
3530
34751610
5421192
155
4004
Protected
7095-188
1492
1200
576911200
4450:11053
4228 484
Protected
4777=360
4531+883
3997+2255
46201:32
4264=166
4368+1473
RAD
4426:154
4240
4022+881
4780+11
3647#32
3921
A2B
5192 +.56
4855+ 120
4722+-33
5511+13
3824+2188
4102+315
A3B
5538++20
4986+441
5110+22
5580+44
4071+242
290
Long-Horizon Adversaries: We also assess adversarial robustness of each method versus more strategic, long-horizon adversaries that compute worst-case trajectories to deceive an RL agent. We evaluate agents against PA-AD (Sun et al. 2023), the ttat-offthheart attack, as well as the Critical Point Attack (Liang et al. 2022) and Strategically Timed Attack (Lin et al. 2017). We evaluate the adversarial robustness of the target policies the depth of strategy increases for the han-hoorzon adversaries. In the context of the Critical Point attack, a higher depth of strategy increases the length and number of trajectories sampled to find the worst-case future outcome, and a stronger Strategically Timed attacker has a larger perturbation budget.
Seversarially-dirette policy
5.2 RESULTS
In Tables T 2 and report the mean result over policies initialized with random seeds, with 50 test episodes each. The variance reported (#o) is the standard deviation from the mean for each method. The most robust score is shown in boldface.
Myopic attacks: As seen Table 3 C-ACoE methods A2B and A3B achieve state of the art robust performance against standard greedy attacker strategies, well nominal performance similar the

---

Published as a conference paper at ICLR 2025
Table 4: Robust performance against the PA-AD attacker (Sun al. 2023). We train the attacker with the PA-AD framework against the completed victim policies for 500 episodes, the same for each victim and environment. As the Protected method has several PA-AD attackers (for each non-dominated policy), we instead use the sampling schema outlined in their work.
PA-AD Perturbed Scores
Method
HalfCheetah
Walker2d
Hopper
Ant
PPO
-388 820
427 + 32
167 93
-121 1255
Radial
3441 + 42
3703 + 202
2288 + 74
2567 = 41
Wocar
4148 + 68
3895 + 126
2387 =
2779 + 170
Protected
4411+4718
5803--857
2896-4-723
4312
Protected
2331 + 277
4480 + 492
2210 +:355
3103++96
RAD
4233 +
3864 67
2403 = 129
2756 =
A2B
4393 +
3997 214
2441 31
2821 312
A3B
4478 67
4931 166
2580 92
3205 + 275
best observed malliminxii methods such as PPO. We attribute this success to the parts of ACoE: framing the adversarial robustness problem as a POMDP and the simultaneous maximization of value and minimization of ACoE error brings increased performance over maximin methods and higher robustness overall. Our approaches perform better than Protected with test time adaptation and also and Protected in all the cases, except Ant.
attacks: We also test our methods against attackers with longer planning horizon (and not only the myopic attackers from above). In Figure 4 and Table we test the performance of our approaches in the presence of the SOTA attack, referred to as the PA-AD policy attack (Sun) al. 2023). We also include experiments evaluating robust methods against the Strategically Timed attack (Lin et al., 2017) and the Critical Point attack(Sun et al. (0200) in the appendix. We find that across domains, C-ACoE agents maintain robustness even against long-horizon attacks. This is one philosophy, as
of the main advantages of our proposed methods following the C-ACoE-mininue
the crror-robust policies seek stable trajectories rather than robust single-step action distributions.
observe qualitative differences between PPO, A3B, and WocaR. The WocaR agent adopts more stable motion, minimizing the worst-case, and PPO optimizes for speed, only using the back leg. A3B balances the two approaches, using both legs to keep stability while still retaining a wide range of motion. Full videos of the behaviors described in Figure 5 can be viewed from DropBox at tinyurl.com/ašb-gif, where the extent of robust behavior can be better observed.
Robust Behavior: In Appendix Figure
6 DISCUSSION AND LIMITATIONS
We introduce the novel concept of ACoE based on beliefs about true state. We propose a scalable approximation of ACoE, C-ACoF, and demonstrate its usefulness in proactive adversarial defense, achieving state of the art robustness against strong observation attacks from both greedy and strategic adversaries on a variety of benchmarks. More importantly, we find that recognizing the partially observable nature of the defender agent in adversarial RL problems and optimizing ACoE can be used increase the robustness of RL to adversarial observations, even against stronger or previously unseen attackers. In this paper, we focused on the estimation of belief states from single step perturbed observations: It may be beneficial to further estimate belief based on observations over multiple time steps. Some preliminary results this are in the appendix, and addressing the computational complexity of multistep observation based belief construction makes for promising future work. We also note that the efficacy of the belief construct that Divergence as a notion attack strength. We find our measures to be empirically the strongest, compared to notions such as Euclidean state distance, other F-divergences, minimum reward, however, and leave other more complex measures to future work.
is reliant on the accuracy of using KL
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
By trying to understand how to produce robust and safe RL policies, we unavoidably create knowledge on the destruction of prior policies. While this pursuit yields net positive result by far, still important to acknowledge the risks associated with this field of research. In this paper specifically, we acknowledge the information asymmetry between the attacker and defender in the problem, as well as the insight that adversary is, in general, considering attacks that change the victim's behavior to the greatest extent. These insights are formal definitions of existing dynamics, and while their acknowledgement may yield some tools to bad actors, also provide formal and explicit tools to mitigate those harms.
REPRODUCIBILITY
We have uploaded code as part of our submission, showcasing the implementation of our ACoE- optimizing PPO methods, well the computation of A3B and A2B. Additionally, Algorithm T provide pseudocode-levs instructions on the implementation of our methods. We have listed hyperparameter values and additional details in the appendix. All proofs in our paper are also present in the appendix.
ACKNOWLEDGMENTS
This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-017) and the grant INF-24-1110 from the US Army Research Office.
REFERENCES
Yossiri Adulyasak, Pradeep Varakantham, Asrar Ahmed, and Patrick Jaillet. Solving uncertain mdps with objectives that are separable over instantiations of model uncertainty. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), Mar. 2015. doi: 10. 10.1609/aaii v291l 16655. URL
https://ojs.co org/ wndex.iiex...... php/ AAA.AAAAAA//...........A.
Asrar Ahmed, Pradeep Varakantham, Yossiri Adulyasak, and Patrick Jaillet. Regret based robust solutions for uncertain markov decision processes. Advances in neural information processing systems, 26, 2013.
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33: 13:48-86-16,5, 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 214-223 JMLR.org, 2017.
Kamyar Azizzadenesheli, Yisong Yue, and Animashree Anandkumar. Policy gradient in partially observable environments: Approximation and convergence. arXiv preprint 1XX1::8100009000 2018.
Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. arXiv preprint arXiv:2102.01356, 2021.
Xueying Bai, Jian Guan, and Hongning Wang. A model-based reinforcement learning with adversarial training for online recommendation. Advances in Neural Information Processing Systems, 32, 2019.
Roman Belaire, Thanh Hong Nguyen, David Lo, and Pradeep Varakantham. Regret-based defense in adversarial reinforcement learning. In Proceedings of the 23rd International Conference Autonomous Agents and Multiagent Systems, AAMAS 2024, pages 2633 2640. ACM, 2024. doi: 10.5555/5635637.366322 URL ht tpt:ps://// // acm, org dot: 555133..5555555333333
3663250
11

---

Published as a conference paper at ICLR 2025
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013. doi: 10. 10.1613/jaiie 10.1613/jair.3912 htttps:////doo http://///oo..oooo/...1111111111
3912
Shang- Tse Chen, Cory Comelius, Jason Martin, and Duen Horng Chau. Robust physical adversarial attack on faster R-CNN object detector. CoRR, abs/1804.058 10, 2018. URL http://azxiv
org/abs/1804 05810
Michael Everett, Bjöm Lütjens, and Jonathan How. Certified adversarial robustness for deep reinforcement learning. CoRR, abs/2004.06496, 2020. URL https://arxiv..rrr
2604.06496
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. aaversiaavvesarialll training of neural networks. The journal of machine learning reseurch, 17(1):2096 2016.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. 2019. doi: 10.48550/ARXIV. 19t....99.......... URL
https://arxiy/:
org/ http://xxxivv.orra//////////////
10615
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. 2014. doi: 10.48550ARRXIV.44122.0
6512
Wenbo Guo, Xian Wu, Lun Wang, Xinyu Xing, and Dawn Song. PATROL: Provable defense against adversarial policy in two-player games. In 32nd USENIX Security Symposium (USENIX Security 23), pages 3943-3960, Anaheim, CA. August 2023, USENIX Association. ISBN 978-1-
939133-37-3. URL https: /www. www.usenni..... http://rrorrooooeeeeeeeeeeeeeeee htt.://rrennneueerrsssserrrryy/e presentation/ www.oonnnuuuubb-uoe
Karel Horák, Branislav Bošanský, Vojtëch Kovařík, and Christopher Kiekintveld. Solving zero-sum one-sided partially observable stochastic games. Artificial Intelligence, 316:103838, 2023.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv: 17t.::22.22.22282 2017. doi: 10:48550/ARXIV17022.
02284. URL http::tts://att /ttp:v////aiivvaa/a org/ abs /tw.a////110 1702. 02284
Peter Jin, Kurt Keutzer, and Sergey Levine. Regret minimization for partially observable deep reinforcement learning. pages 2342-2351, 2018.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dynamics, Advances in Neural Information Processing Systems, 33:8127-8133 2020.
Daniel Kang, Yi Sun, Tom Brown, Dan Hendrycks, and Jacob Steinhardt, Transfer of adversarial robustness between perturbation types. arXiv preprint urXiv:1905.01034. 2019.
Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 2021.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, 32(11):1288-1274 2013.
Ezgi Korkmaz. Adversarial robust deep reinforcement learning requires redefining robustness. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference Artificial Intelligence, AAAI 2023, Thirty-Fifthl Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 8369 8377, AAAI Press, 2023, doi: URL httpt:ttpp..... /oi org /0tp:/............ wtw.aaa..v.7.......
12

---

Published as a conference paper at ICLR 2025
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. 2017. doi: 10.48550/ ARXIV.1705.0645..
06452
Edouard Leurent. An environment for autonomous driving decision-making GitHub repository, 2018.
Haoran Li, Zicheng Zhang, Wang Luo, Congying Han, Yudong Hu, Tiande Guo, and Shichen Liao. Towards optimal adversarial robust q-learning with bellman infinity-error. In First International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: opwnnevveeeeeeeeeee ew. net. d 99ng2Ny
Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial training with- out attacking: Worst-case-aware robust reinforcement learning, Advances in Neural Information Processing Systems, 35:22547-22561, 2022.
Yongyuan Liang, Yanchao Sun, Ruijic Zheng, Xiangyu Liu, Benjamin Eysenbach, Tuomas Sandholm, Furong Huang, and Stephen Marcus McAleer. Game-theoretie robust reinforcement learning handles temporally-coupled perturbations. In The Twelfth International Conference on Learning Representationss ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httpps://ppenee
net/ forum?rrmmmmmmm W2-2UuuUASS
Yen-Chen Lin, Zhang-Wei Hong,
Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, and Min Sun. 2017.
Tactics of adversarial attack on deep reinforcement learning agents. CoRR,
URL http:ttpp/aax...rttteee wwgp://iiiorrooooooooo abs / 10t.s/6773007788888
Xiangyu Liu, Chenghao Deng, Yanchao Sun, Yongyuan Liang, and Furong Huang. Beyond worst-case attacks: Robust RL with adaptive defense via non-dominated policies. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. net www./eeet//nneeee Dww.ddd-DDWWyy-111 ott.::/www..............ooa..aaa
OvenRevie 2024.
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang. Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: /wwp:///epeeeeeeeeeeeeee net www.formmmii.oooooo aamniddjbIyffdai
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. 2017. doi: 10.48550/ARXIV.17666 06083. URL Ut..ntpps///axxx..rrr//////
org.
06083
Jeremy McMahan, Young Wu, Xiaojin Zhu, and Qiaomin Xie. Optimal attack and defense for reinforcement learning, In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Tight AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 14332-14340 AAAI Press, 2024. doi: 10.1609/AAAL V URL https://dox org/10. 110.016609aaaaa. v w3..1311.... 29346
Johannes Müller and Guido Montufar. The geometry of memoryless stochastic policy optimization in infinite-horizon pomdps. In International Conference on Learning Representations, 2021.
Buqing Nie. Jingtian Ji, Yangqing Fu, and Yue Gao. Improve robustness of reinforcement learning against observation perturbations via loc lipschitz policy networks. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eight AAAI Conference on Artificial Intelli- gence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 14457-14465. AAAI Press, 2024. doi: 10. 1609/ org MAAL.33113.299333 URL https:tpp///m 2ttp://.....0.11..////..........
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui- Weng. Robust deep reinforcement learning through adversarial loss. 2021. URL https: / /ttp::::///ppeeeie////
httt/forummi==eale bdw0Q
13

---

Published as a conference paper at ICLR 2025
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632, 2017.
Lerrel Pinto, James Davidson, Rabul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference Machine Learning, pages 2817-2826. PMLR, 2017.
Aravind Rajeswaran, Sarvject Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. 2017.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Minimax regret optimisation for robust planning in uncertain markov decision processes. Proceedings of the AAAI Conference on Artificial Intelligence. 35(13):11930-1199 May 2021. doi: 10. wtl.1669/aai.vv3551... v35i 174177 URL wwt.thttttssss.... hteg/index..hp//AA//arrttt
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063441 2017, doi: 10.48550/ARXIV.1707.
06347. URL http:tt::: /ectts://arxiae /ttp://aaavvv/.. org/ abs /wt.////110.0.. 06347
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems, 32, 2019.
Ali Shafahi, Mahyar Najibi, Zheng Xu, John Dickerson, Larry S Davis, and Tom Goldstein. Universal adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5636-5643, 2020.
Steven Spielberg, Aditya Tulsyan, Nathan P Lawrence, Philip D Loewen, and R Bhushan Gopalumi. Toward self-driving processes: deep reinforcement learning approach control. AIChE journal, 65(10):e 6689, 2019.
Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, and Yang Liu. Stealthy and efficient adversarial attacks against deep reinforcement learning. Proceedings of the AAAI
(00 Apr. 2020. doi: 10.1600911 1..11609aaiiiv http///aaa.v3444....o v34404.6047. URL httph:tt////otto //oj/ aaaii...........................
Conference on Artificial ort.iorrg//eeeeee php/ApA /art i lllllee le/view////eeeee 6047
Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who the strongest enemy?
towards optimal and efficient evasion attacks in deep rl. 2023.
Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, Soumik Sarkar, al. Robustifying reinforcement learning agents via action space adversarial training. In 2020 American control conference (ACC), pages 3959-3964. IEEE, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. doi: 10.1 20.10990000.22202386800000
Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprini arXiv:2001 -039944, 2020.
Junlin Wu and Yevgeniy Vorobeychik. Robust deep reinforcement learning through bootstrapped opportunistic curriculum. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, (CML2022. 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 24177-24211 PMLR, 2022. URL htllttsss/pprrceednngg mir .tt.ppoeeeednsss......essse press/ v1 62. w22kw22kk... html
Jinghan Yang, Hunmin Kim, Wenbin Wan. Naira Hovakimyan, and Yevgeniy Vorobeychik. Certified robust control under adversarial perturbations. In 2023 American Control Conference (ACC), pages 4090-4095. IEFF, 2023.
14

---

Published as a conference paper at ICLR 2025
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsich. Robust deep reinforcement learning against adversarial perturbations on state observations. 2020. doi: 1048550AARX1.2200300833 URL //arivv org/ abs 2atss/2033.oo 08938
15

---

Published as a conference paper at ICLR 2025
A PROOFS AND ADDITIONAL THEORY RESULTS
Proof of Theorem 3.2
Subtracting U from V, and adding and subtracting 7.ug ())()) (s())
get
V(s.) U(b)
R(So: (T)) R(b, ( + t 1 (ba())) (66) y =( [V(s')] - PPP (ba(t) (V(6.)]
Note that by definition of To, we have that Eag=Po (a()))(((s)) U(AI = Feg. D.g.b (x(())(((s))) U(K)]
Next, from Holder's inequality, get that ( [V()) Eg b,.)).)))))))) (V6)Vl max (Vx(V (T), Pol: b, n (T))
Thus, for one side of the inequality above (i.e., using a S from the shown |a| S b, the other side -b S a)
V(sa) U(b) R(8., T()) (((b,((1)) 7 (, - U(6) +
For notation simplicity, let R(s,, x R(b, m(1) = oR(s,b). We use I' as the updated information -()))
state obtained by concatenating with π (1), s. Applying the above recursively, we get
V(s.) (b)
s ōR( b) +
óR(so, b) + 7 (. (6R(s, + /ra -((")) > + tkK
S ...
?K
(dR(s 7ōR(s' b') + 28)((" b²) + ...] + 1-7
We note that El D..,,, (on 7 20 b() + ...] 5(80, b), where
b(8,,b) R(so, - R(b, x (T)) + Ee , Tob: (()())))(0))))0)))))))))))))))) 83)
Thus,
V(s.) - (b)) S 6(s. b) + Tバ
1 1-7
By symmetric argument using other side of Eq. 3 we get
<k KE
8(s0. b)
S V(s.) (bb)
1 -
These last equations led to the statement in the theorem.
The result above uses total variation distance (other work literature also do (Zhang al. 20200) but, total variation is not as informative distance measure as Wasserstein distance. For example, it easy to see that TV Q) = I whenever the support of and Q do not overlap, but does not distinguish whether the non-overlapping supports are near or far apart. As shown in prior work on WGAN (Arjovsky et al. 2017), Wasserstein distance provides more fine- ane-ggained distinctions. Also, the assumed bound E above hides the effect of the nature of the underlying transition T on the bound. Hence, we prove the next result using Wasserstein distance, which reveals these facets of the problem.
Theorem A.I. Assume that (1) V is L-Lipschitz and (2) for any ||s s'lloo and any action have Wi(' (T(-),, a), T(-1s', a))) & &. Then,
Vs*.) - U(b) -((s.. b){ < 7L(5 +c)
1 1-7
16

---

Published as a conference paper at ICLR 2025
Proof of Theorem A.7 The overall proof follows the same structure Theorem difference is in the bound for
3.2 The only
(4)
For a simpler presentation, we use a, a' to denote action taken in current and next time step. As V I-Lipschitz, by duality of W1 Wasserstein distance, have
(ggTT() (+)) Pag P (0m(())(()))))) / s Wi (T()))),, Pal: b,
multiplying by L
(x (s()) Es (=())]((s))] LW1(((()so.
bb,
(s' s, a,a))(s). First, because the restriction on ad adversarial perturbation, we know that if b(s) > then ||s soll S e. Then, based on our assumption
Next, we bound Wi (T-ss, a). Note that Po(. b, E. (- 8 E.
W1 WT(T(-lso a). T()s, S & for any such that b(s) > 0
(5)
First, note that W1 a convex function of its argument. This can be seen easily; we show it for the first argument below. Recall that definition of W1 (n,, v) (p..)) I d(x, (dr,,, dy) for couplings (joint distribution) set that have marginal as μ. V. Choose 7f as a minimizer in W (H1, v) and 7/2 minimizer in (1(,, Let ari (1 a) it easy see that 7* E T(,.)). Then,
inf Wi(om + (1 v) *ET(au+ ((1o)))))())))))) /
d(s, d(s, ()d7()))))))
d(z, v)d
/ d((x, () (2, + (1 a1) / d(x, ((x.5))) -ai v) (1 (1-
Let (7.11,,00) = E, T( |s, a))(s)). Using the above convexity of W1, we get that
WT a). a() - ()) a)) SE
(6)
where the last inequality follows from Eq. 5
Next, we bound W, (Po(- TT
b, a)). First, by definition of b, a) we get that Pol. b,
Eg (- s s') b, a). Consider the joint distribution 7- over the space easy to check that 7" is a coupling, i.e., 7" T(Po(-Ib, a), T(-lb, a)). We show this and for this we drop the dependency on b, for ease of notation.
S given by (s', sy)
sampled as 8o
T(- T(-1b,a), a), (-
First, 7* B) = -AxB (s' (8',6) = SA P (B)s'))dTT( Thus, 7 (A, = SA ds' = T(A)
and (S, B) = Ss P (B)s)))(((ss)
P.(B). Also, note that ||s' - -sollo I for d as the infinity norm because of the bound adversarial perturbation implicit in PV. Then,
/ 118' %olbody (8'
inf (er(r. WP(P(. b. a), a) T(: (b,))
< / ||s' 5% (ooo
<E
(7)
Combining Eq. 5q.5 5 and Eq. 7 by triangle inequality we get
W1(T(- so. a), Po( b,))) S E
17

---

Published as a conference paper at ICLR 2025
The above results show that some basic structural properties needed from the underlying system for bounding ACoE. One is that the value function should not change by a large amount due to small changes in state and another that the distribution of the next state should not be very different for two close by states. Clearly, an adversary can exploit systems that lack these properties.
Proof of Proposition 4.1 The proof is observed from the fact that C-ACoE can be viewed as an infinite horizon MDP with observations so states, immediate cost as R(so, ((s))) - R(b(s.), a). and transition to next state s' described by s', v(s'), ~ T(: s,a).
B ADAPTATION FOR DQN
Algorithm 2: 6-DQN
Initialize network &w with random weights and target network 8..- with weights
Initialize network Qo with random weights 0 and target network Qo with weights # 0
Initialize replay buffer B
Set robustness temperature À
for episode (1,.... M) do
for t 0 - H do
With prob. 1 e, select a² E argmax, Qo a) A6 (s% a), else select a' at random
Sample & states in N(s)), compute b(s) for each 8 E N (s.))
Compute C-ACoE: 6n R(s", a²) Em.N(..)) b(s) a')
Execute action at, get observed state st+ store transition B = B U (s st, 1t)),8)) Sample mini-batch M D;
for each (s), a², g++1 81)) in mini-batch M do
12
h. if episode terminates at step
Set target ys = 8 + mina -w- (x d'), otherwise
[(s'' a²), episode terminates step i +
Set target qi [(s)) a²) + y min Qo (s a'), otherwise
15
Perform gradient descent to update w using loss:
|
y: 8m u+)
Perform gradient descent update using loss: Σ b=1
M
4: Qo (s a²)
Every K steps reset w and 0- 0:
C ESTIMATION OF BELIEF FOR CONTINUOUS STATE SPACE
Lemma C.I. Assume 2(8) < for some constant B. Consider n. uniformly random samples
from C stored in N(so). Let R and R be defined above. Then, (1/n) EvEN(s.)) c=(d') is
an unbiased estimate of (1/vol(C)) Seec ea(s) ds. There exists n large enough so that 1 + (L)vol(C)) J.ce e(i) de
with probubility for given small e, And then, R(1 + e)
(1/n) Ex'NNO ex(ely
E R(1 c) with probability 5.
Proof. Note that Ex (e) = (I)vol(C)) Jaec (s) which gives us the first unbiasedness result. The second result comes from straightforward application of Hoeffding's concentration
18

---

Published as a conference paper at ICLR 2025
inequality where the bound B is used. Then, that
E R
JAEE
I
Σ, C, u(s1) ... u(sn)ds) ......dss dsp
JenEC
JnECc J JnnEC
Sesc c
E u(s1). ... u(8n)do don
Seec es(t)ds
(1 (1+e)vol(C)
JnEC I J.EEC
a(s u(81) ... u(sn)ds) ..... den
n
(1 (C+
El S.EC (e u(s;)ds;
n
(1 ())ol(((C)
vol(C) J( J..ec R(s,, a)(ssdd
n
+1+c)
similar argument holds for the lower bound, thereby, leading the required result,
D DEFINING ACOE BELIEF METHODS WITH STATE HISTORIES
As mentioned in the paper, our methods are amenable to LSTM state histories as well, although empirically find it to be not necessary (Table 9.. Below, we define A2B and A3B when considering a state history of length 2.
A2B: Consider time window of two with the current observation as $,,1 and the previous observation as 8,,0-
808
E E ((s.)))) (
and
b(s1)
Σ b(81,80)
(x,,))
For the initial timestep Sa,u should be fixed to some constant, i.e. using the single-state A2B formula. This formulation does scale exponentially with the size of the neighborhoods, however we can scale down the previous state's neighborhood by considering subset so E (8...)) that had the highest belief.
A3B:
(0)
b(81,50) = E (iN. (+11)))
and
b(s1)= Σ b(s1,
.oEN(snD)
Here,
2(31, so) = Deu(
E ADDITIONAL EXPERIMENTAL RESULTS
We provide empirical investigations into number of specifics that were cut from the main paper for space. Namely, fine-grained evaluations against long-horizon attack strategies in Figures 2 and 4 and further empirical comparison to Protected-PPe Liu et al. 2024). We also provide an extended version of the results tables in the main paper in Table 5 and 6 which include few more baselines, namely CARRL (Everett et al., 2020). BCL (Wu and Vorobeychik 2022), and CAR-DON (Li et al.
2024
19

---

Published as a conference paper at ICLR 2025
E. LONG-HORIZON ADVERSARIES
In prior works published before 2023, robust RL methods had been evaluated against myopic adversaries (i.e. adversaries give perturbations based on the current observation and victim policy, independent of future states and actions), and long-horizon adversarial actors were not considered. In more recent works PA-AD (Sun et al. 2023) is considered, however there are a variety of approaches each with distinct targeting strategies that can be evaluated. In our additional experiments, we include assessments of robust RL methods against the Strategically Timed attack (Lin et al. (017). where the attacker computes the most effective attack intervals, and the Critical Point attack (Sun et al.) 2020), in which the attacker delivers perturbations after computing the score reduction N steps into the future.
We omit Protected-PPO from these granular long-horizon adversary experiments because these adversaries learn to attack a fixed victim policy at test time, and as the Protected-PPO method adapts over multiple episodes at test time, fair comparative methodology is unclear. For worst-case PA-AD results with Protected-PPO, we refer to Table 8 and the PA-AD experiments table in the main paper.
E.2 EMPIRICAL EVALUATIONS WITH PROTECTED-PPO
Online Adaptations: The most up-to-date robust RL method in this space is Protected-PPO (Liu al. 2024), which computes set of non-dominated policies during training. A key part of this method is the test time adaptation step in which regret minimization algorithm (EXP3) with the set of policies is run for multiple rounds (each round is full policy episode) and the weights are updated at test time based on empirical performance against a fixed adversary. over T = 800 rounds of EXP3 ((Liu et al. 2024) reports 800, but we find the actual convergence to faster in most environments). Because the evaluation setup for this method is quite different from all existing literature, we provide an empirical investigation into how the method performs under standard test setups as is helpful to understand how it fits into the robust RI. landscape.
The applications of interest for safe and robust RL such as autonomous vehicle or industrial control realistically do not accommodate any margin for error within one episode, let alone adaptation of policy over multiple episodes.
To this end, we test the performance of Protected-PPO without any test time adaptation (T which denoted with i in the main paper) and with limited test time adaptation (T = 10). In Table g we find the unadapted policy performs poorly compared the weakly-adapted counterpart, which is more uniformly robust. We also note that the weakly-adapter threshold of (T' = 10) adaptation rounds doesn't It improve performance uniformly across domains, as Ant and Hopper both become robust in that short time while Walker does not.
LSTM History Length: In Table 7 we also perform an investigation into the importance of an LSTM history for the Protected framework. We provide results for a Protected- PPtO model using only linear hidden layers, labeled #f We find that the state history quite integral the performance of the method, which functions as the belief about the adversary for the method. This supports the ideas that the partially-observall nature of adversarial RL is the main challenge and must be addressed.
HalfCheetah
Hopper
Walker2d
333
180
370
99
150
350
96
342
75
330
21
330
352
300
Figure 2: Robust agents vs. a Critical Point strategic adversary search sizes.
Sun et al. 2020 with increasing
20

---

Published as a conference paper at ICLR 2025
Table 5: Experimental results versus myopic adversaries. Most robust scores are in bold. Methods evaluated DQN implementations in Atari and Highway, with adversarial perturbation bounds permitted as c=0.1 for PGD, and 0.15 for MAD. *CAR-DQN results are reported directly from their publication, which only uses PGD e00.2
Method
Unperturbed
MAD
PGD Unperturbed
MAD
PGD
highway-fast-v0
merge-v0
PPO
24.8:1542
19.83
16.
14.94=0.01
10.210.02
10.42:10995
CARRL
24.4+1.10
4.86+15.4
12.4343.4
12.6=0.01
12.6+0.01
12.02+0.01
RADIAL
28.55=+0.01
2.42#1.3
14.97+3.1
14.86=#0.01
11.29
11.04+0.91
WocaR
21.49+10.01
6.15+0.3
6.1940.4
14.91=0.044
12.01+0.28
11.71+0021
RAD
21.01++0.01
20.59+4.1
20.02-0.011
13.91+0.01
13.90+0.01
11.72+0.01
A2B
24.8+1001
23.1140.01
20.8=12.6
14.91=0.01
14.2340.8
12.92::.113
A3B
23.810.01
23.2110.01
22.61-14.1
14.91-0.17
14.7
14.89
roundabout-v0
intersection-v0)
PPO
10.33:00.00
7.4110069
3.92=1.35
9.2647.66
3.62
6.75+12.99
CARRL
9.75+10.1
9.75+0.01
5.92=0.12
8.0+0
7.5+
9.0+0.11
RADIAL
10.29+10001
5.33+0.01
8.77+2.4
10.0#0
2.4+5.1
9.61
WocaR
6.75+22.
6.05:10.14
6.48+2.7
10.0
9.47+0.33
3.26
RAD
9.22-10.3
8.98+10.3
9.1140..3
9.85+11.2
9.71422.3
9.62-H0.1
A2B
10.5+-0.0
10.1+0.11
10.0
10.0+0
10.0
9.88
A3B
10.5+0.01
10.33
10.18
10.010
9.68-0
9.8
Method
Unperturbed
MAD
PGD
Unperturbed
MAD
PGD
Pong
Freeway
PPO
21.0+10
-20.0=0.07
110.0+1.0
29 3.0
+ 2.31
2::2.0
CARRL
11.2
11.0=0.010
6.0.1.22
18.5+0.0
19.1 1.20
15.4:0.22
BCL
2110
21:
34.0=0
21.2+ 0.5
CAR-DQN
21
21 0
34.0=0
0.1
RADIAL
21.0=10
11.0+2.9
21.0
0.01
33.2-0.19
29.0
24.0+:010
WocaR
21.0++0
18.7 +0.10
20.0 0.21
31.2++041
19.8+3.81
28.14:3.4
RAD
21.0+
14.0 + 0.04
14.0
33.2-40.88
30.0++0233
27.7+0.2
A2B
21.010
20.1
21.0-0.01
33.240.18
30.1
30.8
A3B
21.010
20.8
21.0-0.01
33.210.18
31.0
31.111
BankHeist
RoadRunner
PPO
1350-10.1
680-1419
0#116
42970-1210
18309
10003+552
CARRL
84940
830+32
790+110
26510+20
24480++200
22100+1370
BCL
2215::8
894.1 9.2
42490+1100
132
CAR-DON
1349
1347:3.66
1015
43286-1801
RADIAL
1349+00
9974+3
1130+6
44501+1360
23119+11100
24300+1315
WocaR
1220+-0
1207+339
1154+194
44156+-2270
25570++390
12750+400
RAD
1340-0
1170:142
12111:56
40200+11200
2909011400
571501505
A2B
1350-00
1230-L42
1240+:56
14000=11020
38205-1440
400151505
A3B
1350-10
1230+11
1250:130
44290+1250
41001-1610
426451458
Method
Unperturbed
MAD
PGD
Unperturbed
MAD
PGD
21

---

Published as a conference paper at ICLR 2025
Table 6: Experimental results versus myopic adversaries. Most robust scores are in bold. Methods are evaluated on PPO implementations in Mujoco, with adversarial perturbation bounds permitted c=0.1 for PGD, and 0.15 for MAD. Protected- -PO is grayed out due differences in evaluation methodology as outlined in the main paper. For fine-gras ime-grained comparisons, see Tables and
Hopper
Walker2d
PPO
4128 + 56
1110#32
128+105
5002 - 20
680:1570
730=262
RADIAL
3737+75
2401
30704:31
5251+10
3895+1125 128
3480++3.1
Wocak
3136-1463
1510 + 519
2647 4:10
45944-974
1908+1305
3944--508
Protected
3652+-108
2512++392
2221=
6319+331
5148+11416
4720+ 1508
RAD
3473+223
2783+1325
31101130
4743=78
3922+1226
4136=1399
A2B
3710111
32401141
3299128
4760-61
4636187
4708
A3B
3766+23
3370:t275
3465+11
5341=60
9025+94
$292-223
HalfCheetah
Ant
PPO
5794 + 12
1491+220
-27+1288
5620+:29
1288+491
1844+330
RADIAL
4724+76
4008+1450
3911#129
5841+34
3210+380
3821=121
WocaR
5220+t112
3530-1458
34751610
5421192
355
4004
Protected
7095-188
4792-1148
4680:E1203
5769:1290
1050
4228 484
RAD
4426=55
4240
4022:851
10
3647#32
3921
A2B
5192 #56
4855+ 120
4722+#33
5511=13
38241218
4102#315
A3B
5538+220
4986+-41
5110+-22
5580+=41
4071+242
4418 290
Table 7: Comparison to the Protected framework Liu et al. (0044) with a history of only one state. Here, we demonstrate superior robust performance when information limited.
Method
Unperturbed
MAD
Unperturbed
MAD
Hopper
Walker2d
PPO
4128 + 56
1110:332
5002 + 20
680
WocaR
3136::463
1510 = 519
4594+1974
3928+1305
Protected
2451+88
2198=2333
3509#332
3410=41
A2B
3710+-11
3240+141
4760+161
4636+-87
A3B
3766+-23
3370+275
5341 +-60
5025 94
HalfCheetah
Ant
PPO
5794 + 12
1491+:20
5620:129
1288-1491
WocaR
5220+112
3530-458
5421192
3550+1155
#-
3210118
2241+392
3997+228
23311:277
A2B
5192 +55
4855+ 120
5511=13
3824-1218
A3B
5538+:20
4986-141
5580-41
4071+:242
22

---

Published as a conference paper at ICLR 2025
Table 8: Comparison to the Protected framework Liu et al. (024) with zero test time adaptation (labelled 7' = 1), for apples-to-apples evaluation comparison to existing baselines. Without the online adaptation part of the Protected framework, we find robust performance (i.e. low drop in score) but not high nominal scores. T 10 allows Protected to adapt for limited number of rounds.
Method
Unperturbed
MAD
PA-AD
Unperturbed
MAD
PA-AD
Hopper
Walker2d
Protected ==
3573=81
2398+666
2210=385
5019 + 87
3887 + 492
4480 + 492
Protected T= 10
3691=81
3314=391
3221=222
6001 + 24
3410 = 558
5520 + 31
A2B
3710+111
3240+441
2441 #:31
4760-161
3997+-214 4636+187
A3B
3766:1-3
3370-+275
2580
5341+-60
5025
4931+166
HalfCheetah
Ant
Protected
4777+-360
3997--285
23314-277
4620+:32
4264+1166
3103+ 96
Protected""=1
5722.158
529611111
4522=450
4747
4688+1201
418611
A2B
5192 1:56
4855 120
4393:19
5511=13
3824-1218
2821 = 312
A3B
5538+220
4986::41
4478+167
5580=14
4071+:242
3205++275
Table 9: Empirical analysis between single-state ACoE and LSTM-ACoE on discrete-action domains (top, highway-env) and contiuous-action domains (bottom, Mujoco). Single-state PPO included as point of reference.
Method
Unperturbed
MAD
PGD
Unperturbed
MAD
PGD
highway-fast
merge-v0
PPO
28.8+:5.42
13.63++19.85
15.21+11.1
14.94++0.01
10.2+0002
10.42++0.95
A3B
25.8++0.01
24.21+0.01
22.61+14.11
14.91+0.117 14.96:1..1
14.88+00117
14.89
A3B-LSTM
28.8:10.11
25.21
23.03
14,88
14.9010.15
Halfcheetah
Hopper
PPO
5794=12
14911:20
5620:129
4128 + 56
1110#33
5002 + 20
A3B
5538=20
4986:141
5110:122
3766123
3370-1275
34
A3B-LSTM
5641+33
5002+:67
5171+188
3729
3411#137
3453 21
Table 10: Ablation study: relaxing test-time attacker constraint shows lower score degradation in ACoE agents than SOTA Protected agents.
Method
MAD attack e 0.15
= 0..175
= 0.2
= 0.3
Halfcheetah
A3B
4986 41
5008 + 259
4907 + 200
1477 2410 + 1880
3896
Protected
4551 + 843
4391 + 729
3855 + 1718
Hopper
A3B
3512=112
3470 66
3367 208
3023 + 348
Protected
3484113
3312+119
3290 249
270511396
23

---

Published as a conference paper at ICLR 2025
HalfCheetah
Hopper
330
430
22
330
22
3.5
Viceol
2215 2.16
2.3 Tudill 11cca 150
Figure 3: Robust agents vs. a Strategically Timed Attack adversary (Lin et al. 2017). , as the length of perturbation increases. We find that the level of strategy increases from Long-horizon attackers, C-ACoE minimization improves robust performance, relative to other methods.
HalfCheetah
Hopper
Walker2d
Ant
4900
37000
4500
20000
3000
20000
25400
A2B
A3B
Radial
WocaR
RAD
Figure 4: Robust agents vs. PA-AD attacker (Sun et al. 2023). as the optimality of the attacker policy increases. To represent levels of optimality, we save PA-AD model weights at 5 evenly distributed points across the training epochs. We find that the level of strategy increases from Long-horizon attackers, C-ACoE minimization achieves more robust performance, relative to other methods.
E.3 ABLATION STUDIES HYYPERPARAMETERS
In Tables TO and we examine sensitivities different training parameters used in the ACoE framework. We train several different ACoE models in Mujoco-halfiheetahl varying the denoted parameters. We determine that while the jobustness-sensitivit parameter À does have some effect on the robustness/value tradeoff, it is not sensitive small changes. We find significant impact of the neighborhood sample size on performance, due to the use of Softmax which favors extreme values.
In Table we observe the improvements made ACoE when including two-state LSTM history as the Protected framework uses, and find that while the performance does marginally increase the unperturbed score. However, the trade-off is expensive, applying ACoE each state in history is combinatorially complex.
Table 11: Ablation study: training parameters. We train several different ACoE models in Mujoco- halfcheetah, varying the denoted parameters. We determine that the robustness-eensitivity is not sensitive to small changes. We find no significant impact of the neighborhood sample size on performance.
sensitivity parameter
value:
0.1
0.19
0.2
0.21
0.3
0.5
ACoE unperturbed:
5620 40
5578
5538
20
5557
19
4994 +
4286
23
ACoE vs. MAD:
4897 + 62
4971
47
4986 + 41
5002 + 48
4731 + 28
4021 + 30
# Nbhd samples:
10
20
ACoE unperturbed: ACoE vs. MAD:
5521+223 4981
5528-L20 4986+14
5535+11
4990-138
24

---

Published as a conference paper at ICLR 2025
F
SUBJECTIVE ANALYSIS
PPO
A3B
WocaR
Figure 5: Last 5 frames of PPO, A3B, and WocaR agents (top to bottom), on MuloCo-HallCheetuh PPO deviates the least from the dashed center-mass line, and has the least balanced gait. WocaR has arguably the most stable posture when noting the faster front leg recovery of A3B, but our empirical results suggest optimizing maximum stability is not always necessary. Full GIFs: tinyurl.com/asb comg/
In Figure 5 we show the visual differences frame-by-frame between PPO, A3B, and Wocar-trained models. A3B and Wocar agents exhibit visually similar behavior, which are distinctly more stable than the PPO-learned behavior. Subjectively speaking, the robust behavior is more realistic and accurately depicts how one would expect the agent move, while the PPO behavior is more of an exploitation of the MuJoCo physics engine than a realistic behavior. Under adversary this becomes relevant: the niche value-optimal exploitative movement of the PPO agent turn exploited by an adversary, while the robust models can retain their stability.
G TRAINING DETAILS AND HYPERPARAMETERS
G.1 MODEL. ARCHITECTURE
Our DQN and PPO models follow settings common to the current lineage of robust RL work (SA- MDP, Radial, WocaR, RAD). For C-ACoE estimator functions, a single linear output layer, congruent to the CCER estimator in RAD and Worst-value estimator in WocaR. For Atari image domains, we
two 64x hidden layers with
convolutional layer with 8x8 kernel, stride of and 32 channels, a convolutional layer with a 4x4 kernel, stride of 2 and 64 channels, and a final convolutional layer with 3x3 kernel, stride of and 64 channels. Each layer followed by a ReLU activation, and finally feeds into fully connected output.
The LSTM models use 64x64 hidden layer size with linear layers for input and output.
G.2 TRAINING HYPERPARAMETERS
We train our methods for 900 episodes for all MuJoCo environments, using annealed (Adam) learning rate of 0.005. The robustness hyperparameter À is set to 0.2 for all of our models, which is the same the robustness hyperparameters found in prior works Oikarinen et al. (2021); Liang ct al. )2022); Belaire et al. (2024); Zhang et al. (2020). The attack neighborhood sample size is set to 10, and the training attack neighborhood radius is set to 0.1. both tuned from sets in the range +100% All other hyperparameters are the same as those used in Liang et al. (0022). which is open-sourced at http://:tttuu..coommmmmm whw.-haaunnnanaaaaRRa..
G.3 HARDWARE
We train our linear models on an NVIDIA Tesla V100 with 16gb of memory, and LSTM models on an NVIDIA L40 32gb GPU.
25


Reviews:
[{"review_id": "5s7qVGDhG3", "reviewer": "Reviewer_BY2g", "summary": "The paper introduced a new perspective into adversarial RL problems by viewing the state-perturbing adversary as inducing a POMDP. The authors then proposed the concept of Adversarial Counterfactual Error (ACoE) and developed a surrogate of it for applicability in general RL problems. Extensive experiments in various environments demonstrate the effectiveness of the proposed ACoE objective and the practical algorithms.", "questions": "1. The authors \"restrict solutions to the set of policies that depend just on the current observation\" (line 302). Can the authors discuss the trade-off here, i.e., how much benefit can there be if we consider using policies that can depend on the history, and what's the downside of the increased complexity (e.g. computational cost).\n2. It's encouraging to see that the proposed approach performs well even against long-horizon adversaries, given that the design of the brief construction algorithms only involve the consideration for the myopic adversary. It would be interesting to see whether A3C can be hacked and to which extent--suppose we have a long-horizon adversary who's aware of the A3C heuristic (an adaptive attacker), how robust is the proposed algorithm in this case.\n3. The authors focused on the state-perturbing adversary in this work. Can the authors discuss the extension of the framework to other types of RL adversary?\n\nMinor\n\n- Line 362: should be $\\pi(s_1)$ instead of $\\pi(s_1')$.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The newly introduced objective ACoE provides a theoretically grounded approach to handling the POMDP induced by state-perturbing adversaries. This is novel and a valuable contribution to the adversarial RL literature. \n2. The evaluation is comprehensive. The authors experimented with various RL environments and considered different types of attacks, which consistently demonstrated the effectiveness of the proposed approach.\n3. The paper is overall well-written and pleasant to read. The authors have done a great job setting up the background, comparing with the literature, motivating the new objective and algorithm on top of that.", "weaknesses": "I overall enjoy reading the paper and only have a few minor questions; see \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1tMhWx7mhD", "reviewer": "Reviewer_xpL3", "summary": "This paper proposes a novel method for adversarial reinforcement learning which balances between natural policy value and adversarial robustness. By revisiting the relation between policy value without perturbation and with perturbation, the paper proposes a new objective that minimizes the controllable difference of the return between attacked policy and unattacked policy. Based on the theoretical insights, the paper introduces a model-free adversarial training method with two algorithms, A2B and A3B, to estimate the belief states. Experiments on variaous environments and attack methods show that the proposed methods are effective, achieveing better balance between clean reward and attacked rewards.", "questions": "See my questions in the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes an interesting idea of balancing policy unperturbed value and perturbed value through the optimization of Adversarial Counterfactual Error. Different from prior work which considers the worst-case performance, the proposed method estimates the attacked value via belief state estimation. \n- The paper is in general well-written and easy to follow. \n- The paper presents experiments in several different scenarios and shows better performance than SOTA baselines. The method can be adapted for both PPO and DQN, both of which achieve good empirical results.", "weaknesses": "Both A2B and A3B rely on some knowledge of the attacker, such as attack radius and attack perference. Can the authors provide more justification on whether this is a realistic assumption? For the experiments, how does the attack radius and surrogate attack set for different attack scenarios? If the surrogate attack radius or attack preference is very different from the actual attack, how will the attack performance drop compared to baselines which consider the worse-case attack?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z4joYcHy44", "reviewer": "Reviewer_Ui4m", "summary": "This paper focus on the adversarial defense in the context of Deep Reinforcement Learning (DRL). Specifically, this paper introduces Adversarial Counterfactual Error (ACoE) defined based on the beliefs (uncertainty) about the underlying true state of the target model, thus balancing value optimization for improved robustness. In addition to extensive experimental evaluations, the authors also provide systematic analyses in the theoretical part to justify the effectiveness and generalization ability of the proposed method.", "questions": "1. Is it possible to provide an analysis of the trade-off between natural performance and adversarial robustness?\n2. Is it possible to extend the proposed method to high-dimensional continuous state spaces?\n3. The authors can try to run sensitivity analyses on key hyperparameters, such as the\nrobustness-weight factor \\lambda and neighbourhood size", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is novel with a clear motivation. It's also scalable for model-free environments.\n2. Extensive experiments and analyses are conducted to demonstrate the efficacy of the proposed method.\n3. Theoretical analyses are detailed and solid.", "weaknesses": "1. This paper has limited (experimental) justifications for surrogate belief construction choices. Comparisons with alternative constructions would be better to support the motivation.\n2. The method's adaptability to diverse or higher-dimensional continuous state spaces is underexplored. Instead of theoretical analyses, further explanations or experimental results would be preferred.\n3. I feel the performance of the proposed C-ACoE would be likely sensitive to hyper- parameters. Further analyses are required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FtIye27yM9", "reviewer": "Reviewer_X3rZ", "summary": "This paper introduces the Adversarial Counterfactual Error (ACoE) as a metric for quantifying robustness in adversarial deep reinforcement learning (DRL). Current methods for adversarial robustness often struggle to balance performance and robustness, especially in cases of partial observability due to adversarial noise. The authors hypothesise that explicitly accounting for partial observability in adversarial settings can lead to more balanced results. To address this, they propose Cumulative-ACoE (C-ACoE), a scalable surrogate objective designed to improve model-free DRL methods. C-ACoE allows for simultaneous robustness and performance improvements achieved through adaptations of popular DRL algorithms such as PPO and DQN. Their experimental results on various benchmarks (MuJoCo, Atari, and Highway) show that C-ACoE methods outperform current state-of-the-art approaches in adversarial environments.", "questions": "1. Do we care about unperturbed performance since basically all real-world applications have states being perturbed. Can you list any applications?\n2. Both A2B and A3B require KL divergence between policies, which should be computational costly. Did the authors address this anywhere?\n3. Can you provide any intuition about algorithmically, why this method can have such good empirical performance?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper proposes an original approach to managing adversarial robustness in DRL by framing it as a partial observability problem. This novel perspective on adversarially-induced partial observability, operationalized by ACoE and its scalable form C-ACoE, is a significant theoretical contribution. With ACoE as the objective, the methods A2B and A3B show superior performance compared to baselines.", "weaknesses": "- The Abstract is overly long.\n- Missing $)$ in Eq. 2.\n- It is unclear to me how C-ACoE $delta$-network is learned, and how it's aligned with the theory especially Eq. (2). Specifically, in the paper the authors claim that $delta$ is no longer dependent on $b$, then what's the use of $b(s)$ in A2B and A3B? And if $b$ is not needed then in Eq. (2) RHS the second term still has $b$, then how do you learn C-ACoE correctly? The authors should put more details about C-ACoE in algorithm 1, and omit the common knowledge like GAE and PPO-clipped.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)