On the Tension Between Optimality and Adversarial Robustness in Policy Optimization
Authors: Haoran Li
Jiayu Lv
Congying Han
Zicheng Zhang
Anqi Li
Yan Liu
Tiande Guo
Nan Jiang
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
adversarial robustness
policy optimization
theory-practice gap
bilevel optimization
Primary area: reinforcement learning
TL;DR: This paper demystifies the gap between theory and practice in adversarially robust reinforcement learning and address this by proposing BARPO, a novel framework that bridges adversarial robustness and optimality.

Abstract: Achieving optimality and adversarial robustness in deep reinforcement learning has long been regarded as conflicting goals. Nonetheless, recent theoretical insights presented in CAR suggest a potential alignment, raising the important question of how to realize this in practice.
This paper first identifies a key gap between theory and practice by comparing standard policy optimization (SPO) and adversarially robust policy optimization (ARPO). Although they share theoretical consistency, *a fundamental tension between robustness and optimality arises in practical policy gradient methods*. SPO tends toward convergence to vulnerable first-order stationary policies (FOSPs) with strong natural performance, whereas ARPO typically favors more robust FOSPs at the expense of reduced returns. Furthermore, we attribute this tradeoff to the *reshaping effect of the strongest adversaries* in ARPO, which significantly complicates the global landscape by inducing *deceptive sticky FOSPs*. This improves robustness but makes navigation more challenging. To alleviate this, we develop the *BARPO*, a bilevel framework unifying SPO and ARPO by modulating adversary strength, thereby facilitating navigability while preserving global optima. Extensive empirical results demonstrate that BARPO consistently outperforms vanilla ARPO, providing a practical approach to reconcile theoretical and empirical performance.

Published as a conference paper at ICLR 2026
ON THE TENSION BETWEEN OPTIMALITY AND AD- VERSARIAL ROBUSTNESS IN POLICY OPTIMIZATION
Haoran Li
Jiayu Ly
School Mathematical Sciences University of Chinese Academy of Sciences Beijing, China
School of Advanced Interdisciplinary Sciences University of Chinese Academy of Sciences Beijing, China 1vjiayu240mar 1s. ucas acas.ac.cm cn
lihaoran2 coemm21@malls.. ucas ac.co on
Congying Han
Zicheng Zhang
School of Mathematical Sciences University of Chinese Academy of Sciences Beijing, China
JD.com Beijing, China
zhangzicheng6@@jd. com
hancy@ucas.accoom
Anqi
Yan Liu
School of Mathematical Sciences
School of Statistics and Data Science
Nankai University
Nankai University
Tianjin, China angil iigili@nankali edu. en
Tianjin, China
1iuyan233nannken eduu en
Tiande Guo
Nan Jiang
School Mathematical Sciences
Siebel School of Computing and Data Science University of Illinois Krbana-Champaige Urbana, IL 68001, USA
University of Chinese Academy of Sciences
Beijing, China
tdguo@ucas cn
nan nanjiang iichiggilli ailiiniiniiiiiii.....mm edu
ABSTRACT
Achieving optimality and adversarial robustness deep reinforcement learning has long been regarded as conflicting goals. Nonetheless, recent theoretical in- sights presented CAR (Li al., (024) suggest a potential alignment, raising the important question of how to realize this in practice. This paper first iden- tifies a key gap between theory and practice by comparing standard policy opti- mization (SPO) and adversarially robust policy optimization (ARPO). Although they share theoretical consistency, a fundamental tension between robustness and optimality arises in practical policy gradient methods. SPO tends toward con- vergence to vulnerable first-order stationary policies (FOSPs) with strong natural performance, whereas ARPO typically favors more robust FOSPs the expense of reduced returns. Furthermore, we attribute this tradeoff to the reshaping ef- fect of the strongest adversaries in ARPO, which significantly complicates the global landscape by inducing deceptive sticky FOSPs. This improves robustness but makes navigation more challenging. To alleviate this, we develop the BARPO, bilevel franework unifying SPO and ARPO by modulating adversary strength, thereby facilitating navigability while preserving global optima. Extensive em- pirical results demonstrate that BARPO consistently outperforms vanilla ARPO, providing a practical approach to reconcile theoretical and empirical performance. The code is available at httpps//gghubb.oo/////aaaaaaa//s
"Equal contribution Corresponning author

---

Published as a conference paper at ICLR 2026
(a) SPO
(b) ARPO
(c) BARPO
(d) Overall
Figure 1: Schematic illustration of the optimization landscapes under SPO, ARPO, and BARPO. (a) SPO ascents along fragile directions, leading to vulnerable FOSPs with high natural value. (b) ARPO becomes trapped in robust regions but is limited to low-return solutions. (c) BARPO reshapes the landscape by lifting robust but low-return regions, enabling convergence robust FOSPs with high returns. (d) Overall comparison of the three paradigms: contour lines represent natural returns, while background color indicates robustness, with darker red denoting lower robustness.
INTRODUCTION
Deep reinforcement learning (DRL) has demonstrated remarkable success across a variety of com- plex tasks (Mnih et al. 2015 Lillierap et al., 2016; Silver et al. 2016) and real-world applications,
including robotics (lbarz et a..
Liu et al. [0222] lang et al. (2025) autonomous driving (Kiran 2024 and large model training (Xin et al. 2025 Kumar al.. 2025
2021
et al. 202 Chen et al.; 2022
Guo et al.j 2025). Despite such substantial progress, DRL agents remain highly vulnerable to im- perceptible attacks on their state observations, leading severe performance degradation and even
complete system failure (Huang et al. 2017: Behzadan & Munir 2017a Lin et al.) 2017 Kos & al. (018) Weng al. [2019 Inkawhich al. [2020] llahi et |l.] 1). This vulnerability threatens the reliable deployment of DRL in safety-critical environments, highlighting
Song 2017
the urgent need for robust agents capable of withstanding subtle perturbations and malicious attacks.
Motivated by these challenges, Zhang et al. (2020c) introduced the suate-adversariaa Markov deci- sion process (SA-MDP) framework, providing a theoretical foundation for adversarial robustness in reinforcement learning. Building on this framework, they demonstrated that an optimal robust policy (ORP) may not always exist, indicating potential conflict between the objectives of optimality and robustness. Recently, [i et al. (2024) 2025) further explored the concept of ORP in practical settings and proposed the ISA-MDP formulation, a refinement that formally establishes the existence of an ORP that coincides with the Bellman optimality policy. These theoretical results suggest that the objectives adversarial robustness and optimality are, in principle, aligned in most practical tasks.
This insight drives the pursuit of DRL agents that not only maintain strong performance in clean vironments but also exhibit resilience to adversarial attacks. This theoretical consistency is critical for deploying DRL agents in real-world applications, where strong malicious attacks are compar- atively rare. However. in practice, achieving this alignment remains challenging. Despite sharing this theoretical consistency, conventional policy optimization methods struggle under adversarial settings (Zhang et al., 2021 Sun et al. 2022 2024). It remains an open problem whether this theo- retical alignment can be technically realized, leaving a critical gap between theory and practice.
In this paper, we delve into the optimization behavior and learning dynamics of two train- ing paradigms: standard policy optimization (SPO) and adversarially robust policy optimization (ARPO). SPO aims to maximize the standard value function, i.e., max, V* (s). ) V while ARPO optimizes the worst-case adversarial value function, i.e., max, min, Vnou(s) (s), ; 8. By conducting a systematic comparison, reveal novel optimality-robusteess robustness tension in policy gradient methods.
In our analysis, we first prove that both SPO and ARPO converge to first-order stationary policies (FOSPs) rather than global optima. Next, by examining the learning behaviors around these FOSPs, we uncover a key discrepancy: ARPO consistently produces policies with markedly higher robust- ness to state perturbations, whereas SPO-derived FOSPs remain vulnerable. These findings help explain the empirical vulnerability of SPO, despite the theoretical existence of an ORP, and empha- size the importance of ARPO in building robust DRL agents. Finally, reveal that this robustness
2

---

Published as a conference paper at ICLR 2026
gain comes significant cost: ARPO-trainet policies exhibit substantially lower natural returns compared to SPO. Altogether, these results expose a fundamental robustness perforssnpp tradeoff in goliie-gradient methods, highlighting a practical challenge for closing the suooryppacctic gap.
To understand this tradeoff, we analyze the optimization landscape and value geometry induced by SPO and ARPO. We attribute the tension the reshaping effect induced by the strongest adver- saries: although they help guide the agent toward robust FOSPs, they often impede optimization by compromising the navigability of the landscape. As illustrated in Figure [ ARPO creates robust peaks but introduces additional valleys, distorting the global landscape to a more rugged terrain. These new valleys act as traps for the optimization path, hindering gradient-buu baased methods from ascending towards the global or better local peaks. Instead, the optimization process stalls at these deceptive sticky FOSPs, which often result in lower natural returns compared to those achieved by SPO. This reshaping effect presents a core barrier to learning both optimality and robustness.
To address this challenge, we propose bilevel optimization framework that bridges SPO and ARPO, which adjusts the adversary strength to promote traversable optimization paths while maintaining robustness. This modulation prevents the optimization landscape from becoming overly rugged, helping to avoid local traps while still encouraging robustness. Importantly, it preserves the glob- ally optimal robust policy shared by both SPO and ARPO. To enhance convergence and efficiency, incorporate SPO dynamics into the bilevel framework, further smoothing the optimization land- scape. The resulting approach, Bilevel ARPO (BARPO), consistently achieves superior natural and robust returns, offering promising approach bridging the gap between theoretical alignment and practical performance. To summarize, this paper makes the following cohesive contributions:
We explain the nature of vulnerability in standard policy optimization (SPO) within the theoreti- cally aligned framework. While an optimal robust policy exists, SPO typically converges to fragile first-order stationary policies (FOSPs), which offer strong natural performance but lack robustness.
We uncover a new form of optimality-robustenn prior notions of conflicting objectives, adversarially robust policy optimization (ARPO) shares the same global optima with SPO but prefers more robust FOSPs that typically yield lower returns.
tension arising in policy optimization. Unlike
We elucidate the underlying mechanism behind this trade-off. The strongest adversaries in ARPO reshape the optimization landscape by introducing numerous sticky FOSPs, which improve robust- ness but hinder effective policy improvement by making the landscape more difficult to navigate.
We propose a unified bilevel framework that connects SPO and ARPO, and instantiate it as Bilevel ARPO (BARPO). a novel method that reshapes the landscape to reduce the prevalence of poor sticky FOSPs while retaining access to robust global optima. Empirical results demonstrate that BARPO significantly improves practical performance, narrowing the gap between theory and practice.
2 PRELIMINARIES AND NOTATIONS
MDP and SPO. A Markov Decision Process (MDP) is formulated as a tuple P,, Mo), where S is the state space. is the action space. The reward function r : S E assigns scalar reward to each state-action pair. The transition dynamics IP: S x A + A (S) define the probability distribution over next states, and No A (S) denotes the initial state distribution. The discount factor y E [0. 1) controls the trade-off between immediate and future rewards. A stationary policy S - A (A) maps each state to probability space over actions. Under given policy æ, the state value function is V* (x) = E EE 7'r(st, (x0 8). and the action-value function (or Q- function) is (T a) = Er,, E 7 ("r(s,, as)lsp = s, ag = a]. A fundamental property of MDPs is the existence of stationary and deterministic optimal policy π² termed Bellman optimality policy (BOP), that maximizes all V (s) and Q" a) for all states and actions. In practice, particularly for large-scale state and action spaces, standard policy optimization (SPO) methods are widely used for their efficiency (Silver et al. 2014; Scchulman et al. 2016 2015 Lillicrap et al., 2016) Mnihh et al., 2016 Schulman al. 2017 Haarnoja et al.) 2018). For the parameterized тe. SPO aims to solve:
max (s),,,, (s), V where I1: := {Tel (o (-(s) E A (A), VSES). 011.E11
(SPO)
To simplify notations, we take the expectation over the distribution Mo: V (Ho) = Egnpal VV (s)).
ISA-MDP and ARPO. An Intrinsic State- adversarvll Markoy Decision Process (ISA-MDP) is de- fined by the tuple (8, A,r, r, P. Po; B²). This formulation extends the standard MDP by incor-
3

---

Published as a conference paper at ICLR 2026
porating structured neighborhoods of allowable state perturbations. For each state E S. the set-value function (s)s) S S specifies admissible perturbations which cannot alter the optimal action. An adversary is represented by mapping v S - S that perturbs the observed state 8 to a nearby state 8y == v(s) E (s). Under this adversary, the policy operates on the per- turbed state, denoted by (st o (a]s)) == (a)v(s))) Under this setting, the adversarial value func- tion is given by (Tou(s) = E.o. 12 7)r(ss, = s], and the corresponding adversarial Q-function is (s, a) = Evor, 1-== 7'r(s,,a)) ai]lso = s, du = a]. For any policy т. there ex- ists the strongest adversary v" (TT) that minimizes the value function across all states, defined by v" ("()) = arg min, (s). Vs. Within this framework, an optimal robust policy (ORP) x ex- ists, which simultaneously maximizes performance against its strongest adversary for all states, i.e., max, (s) for all s S and x² aligns with Bellman optimality policy.
We refer to the robust optimization problem in ISA-MDP as Adversarial Robust Policy Optimization (ARPO). Despite its centrality to robust decission-maknng ARPO has not been thoroughly explored. In this work, we adopt the direct parameterized adversary, resulting in the following formulation:
max min (Taora(s). s E S, where V == {volvo(s) = s 02 E (s)s), s} (ARPO) Olr AugE
Similar to SPO, we consider the expected objective: V* (po) E V (s)]- In particu- lar, SPO corresponds to solving the outer maximization while fixing the inner solution to O.
FOSP refers stationary point in the optimization sense, where the policy gradient vanishes. Formally, we define No (or 0) as a first-order stationary policy (FOSP) for SPO if it satisfies:
V V (mo) = O, Via VT (Ho) = 0.
(FOSP in SPO)
Similarly, me (or 0) is a FOSP for ARPO if it satisfies the following conditions:
V. VOVTOoNT (TO) (50) (mo) = O,
V Vaoov" (T7) (1o) 3
(FOSP in ARPO)
We further define (Tg, y² (TT)) (or (0, 0 (0)) as a first-order stationary policy-adversary for ARPO
Terminology Clarification. To avoid ambiguity, we define the following terms used throughout the paper: 1) Natural returns (or performance): the estimate of standard value function V' (po). ). 2) Robust returns (against v/): the approximation of adversarial value function )To)( 3) Robustness
(against v): an estimation of the relative degradation in performance V V***((u)) (Ho) (mo)
S 0.
V (NO)
OPTIMALIY- TENSION IN POLICY OPTIMIZATION
This section identifies the inherent tradeoff between optimality and robustness in policy gradient methods. Although SSOO and ARPO share the consistent globally optimal robust policy, their con- vergence behaviors diverge significantly. In practice, SPO often tends to vulnerable policies with high natural performance, while ARPO prefers more robust policies, typically at the cost of reduced returns. Notably, we explore the value geometry and optimization landscape, attributing this diver- gence to the adversarial reshaping of ARPO. The strongest adversary distorts the global terrain by introducing numerous local minima, which trap the navigation in robust but suboptimal policies.
3.1 ROBUST CONVERGENCE BEHAVIOR OF ARPO
We begin with rigorous convergence analysis of ARPO showing that it generally converges to first-order stationary policy (FOSP) rather than the global optimum. To better understand this, we examine the nature of FOSPs in both ARPO and (SPO by analyzing empirical behaviors and local learning dynamics. Our findings reveal that ARPO promotes stronger robustness near convergence, even without reaching the globally optimal policy. In contrast, SPO achieves high natural returns but lacks robustness against perturbations. Furthermore, this analysis uncovers a connection between robustness and generalization, suggesting that sufficient robustness may improve generalization.
3.1.1 CONVERGENCE TO FIRST-ORDER STATIONARY POLICIES
To enable formal gradient- based analysis of ARPO, first derive the policy gradient expression for the adversary. This formulation facilitates sample-bar based estimation and numerical computation within the inner optimization. The complete derivation and proof are provided in Appendix C.1
4

---

Published as a conference paper at ICLR 2026
Table I: Natural and robust performance of SPO and ARPO for continuous control tasks in MuloCo.
Enrtronment
Method
Return
Ramãor
Critic
MAD
RS
under
SA-RL
PA-AD
Worst
Worst-case Robustness
Hopper
SPO
3081
2923
2035
1763
-0.974
nemo
ARPO
sux
0:528
8:30
.0.524
45%
Walker2d
4662
23
4628 a:21
1:15
4907 0:65
115
336 2:200
AXPO
2095
983 50%
1815 230
2300 à 1821
1944 9.95
1360 52
1270 8:503
1460 737
1270 a:500
0.394 1 58%
Hallcheetah
SPO
50MW
1M
4463 2:50
3281 à 1991
918 541
1049
-213 1:
.69 4:21
-213 m:
-1 1.042
m 0,13)
ARPO
1412
99. 72%
1363 263
1359
1402 04
1230 75
1079 5:45
1216 4:
1079 46
+0.236 77%
And
SPO
5381
5329
4696
1768
1097 2-10
-1398
-3107
-3107 10N
-1 3.5
ARPO
1709
2026
1648
1648
40,034
58%
Theorem 3.1 (Policy Gradient for Adversary). Given a policy TG, for all state s E S, consider the direct parameterization representation for adversary vo - S, 8 3 s E B(s). Then, for any state 8, E S, we have the state-wise policy gradient for the adversary as follows:
1 V De Vioovo (s) - I- (s (a)demoon [Quoors (5 (d.'''Vo, log Tog'e(a'ls' + + 1ls' = s:)]
where d is the state-action visitation distribution under v, and I(-) is the indicator function.
Based on this analytical expression, we further characterize the convergence behavior of ARPO. The detailed analysis and proofs are provided in Appendix CC22 Justifications and potential relaxations of assumptions, and the connection with core contributions are further discussed in Appendix H.1
Theorem 3.2 (Convergence of ARPO). Denote A == maxa ming VToeve (40)-ming) TT oMe (po). Assume that the sampled policy gradient is Lipschitz continuous and bounded by Ma. the sampled estimation of the value function is locally p-strongly convex, the state-action visitation distribution Lipschitz continuous, and the variance of the stochastic gradient is bounded by 0². Set the step size as Tik- = V (
Then, the ARPO with 8 5-approximate adversary and K 2 of satisfiex:
EE k=
E
(TOP (HO)
S 40
AL K
2L +
K
1
where Lgg and I, Leo + Lgg + Me Lucataee + T.d0) are the Lipschitz constants.
Theorem 3.2 shows that, instead of converging to the globally optimal robust policy, ARPO ap- proximates an FOSP of the value function under the strongest adversary. The convergence rate is O(K-1/2), and the approximation eтror depends on the strength of the adversary. Similarly, SPO achieves convergence the FOSP of the standard value function the same rate (Agarwal et al., 2019 2021). These underscore the gap between theoretical optimum and practical convergence.
3.1.2 LEARNING BEHAVIORS NEAR FIRST-ORDER STATIONARY POLICIES
Based on the above convergence results, we further explore the characteristics of SPO and ARPO around their FOSPs, highlighting the critical difference between robustness and natural performance.
Empirical Behaviors Near FOSPs. We empirically examine the properties of FOSPs obtained by ARPO across both simple and complex environments. Specifically, we analyze toy ISA-MDP with two states and two actions using directly parameterized policies, alongside continuous control tasks in MuJoCo employing neural network policies. In both settings, we observe that FOSPs discovered by ARPO tend to exhibit robustness but generally achieve substantially reduced returns than those achieved by SPO. The detailed analysis and complete proofs are provided in Appendix D
Proposition 3.1. There exists an ISA-MDP such that the following statement holds. Let ag be FOSP under SPO, and let A be FOSP under ARPO. Then, TA is a robust policy with V* V*A(Ho) (HO) (V (Ho) V (Ho)), where V (Ho) min V (po) denotes the worst-case value.
Furthermore, shown in Table T in complex digh-dimensional tasks, SPO achieves strong natu- ral performance but suffers substantial degradation under adversarial attacks. In contrast, ARPO exhibits substantially greater robustness, particularly in more challenging environments such as
5

---

Published as a conference paper at ICLR 2026
HalfCheetah and Ant. However, its natural performance remains consistently and noticeably lower, often reaching less than half the returns achieved by SPO. These empirical results underscore the practical disparity between the FOSPs obtained by SPO and ARPO, revealing an inherent tradeoff between robustness and optimality in policy gradient methods.
Learning Dynamics Near FOSPs. To better understand the practical performance of SPO and ARPO, we analyze their learning dynamics around their FOSPs. This sheds light on how these paradigms behave near convergence and reveals critical insights into how robustness impacts gener- alization. Specifically, we characterize the Hessian structures at the FOSPs for both SPO and ARPO, which reflect the local curvature of the value landscape and provide a measure of policy flatness.
) be a first-order stationary policy-adversary in ARPO, satisfying the second-order optimality condition. Denote X-in(-) and >max(-) (-) as the smallest and largest eigenvalues, and let d be the state space dimension. Suppose (0", (*) is locally stable, the initialization (0o, Ua) satisfies V (Ho)
Theorem 3.3 (Flatness Bound of FOSPs). Let (0", (0))
Vig- (Ho), the pol- icy gradient noise is coercive with coefficient k. Then, for single-loop ARPO with step size 1 2/22min ( (-V2g V ova (Ho)), attack budget e S 2/ mmxx V VRo-oko (Ha)), and batch size 00
B 2 dee V2 V ove (HO))?, ,, the following inequality between Hessians holds:
(1 V owr (no))?++
+ konp?
B
(1-
(1
2ko B
V2
20
V
ouS. (un))l
(1)
KOT?
Similarly, for a locally stable FOSP 0* in SPO, if the initialization do satisfies V (po) V (po), and the policy gradient noise is coercive with Kg, then VV V (Ho)ll S mon
<
The detailed derivations are in Appendix C.3 From inequality (.. observe that for FOSP in ARPO, if the adversary-side curvature is bounded as V/ Vala b0g ((ug)]l S B/(2kg)²), then the policy-side curvature in ARPO matches that of SPO, i.e., V V oue (Ho)ll? S B/(Ron²). ) This implies that if ARPO achieves sufficiently strong robustness, i.e., flat adversarial curvature, can preserve generalization comparable to SPO. Moreover, greater robustness may further enhance generalization. However, if robustness is insufficient, generalization may be significantly degraded.
3.2 RESHAPING EFFECT OF STRONGEST ADVERSARIES HINDERING POLICY IMPROVEMENT
In this subsection, we investigate how the strongest adversary in ARPO reshapes the optimization landscape and value space geometry, shedding light key factor behind the optimality-robustnnn tension. We show that although the strongest adversary promotes convergence to robust FOSPs, also severely distorts the global landscape, thereby impeding effective policy improvement.
Adversarial Reshaping Promotes Robust FOSPs. As illustrated in Figure 2 com- pare ARPO with SPO. It follows directly from the definition that the robust objective V (= (mo) is always less than equal to the standard objective (po). In blue re- gions where ne exhibits strong robustness, the remains small, leading to only a slight shift in the location of 0 within the landscape relative to that under SPO. In contrast, in red regions where Tae more vulnerable, the gap becomes substantial, resulting in sharp decline in the objective value at the corresponding location. This adversarial reshaping significantly increases the ruggedness of the optimization landscape, thereby steering the learning process toward robust policies as FOSPs.
optizal palcy
sulazable
desuptin FUSP ARARI
tzibal paticy
Inttal policy
valtrratle
nitrad
gap (*a) Vagou"
(xx) (=
(a)
(b) ARPO
Figure 2: Intuitive examples of reshaping.
Distorted Landscape Impairs Policy Navigation. However, the robust policy space typically ex- hibits vulnerable connectivity. Adversarial reshaping introduces deep valleys that fragment the ter- rain, isolate different FOSPs, and obstruct gradient-based traversal. The proof in Appendix [
Proposition 3.2 (Vulnerable Connectivity Leading to Separated FOSPs). There exists an ISA-MDP where the robust policy space Robust contains cut point; that is, there exists policy n such that Il Robust {T) is disconnected. Therefore, ARPO tends to yield more isolated FOSPs than SPO.

---

Published as a conference paper at ICLR 2026
As illustrated in Figure b). the valleys intro- duced by the adversary create barriers that ob- struct ARPO from reaching better FOSPs the globally optimal robust policy. This fragmenta- tion increases the difficulty of learning and of- len traps the agent in suboptimal FOSPs. As further shown in Figure even in simple ISA- MDP, approximately one-third of initial poli- cies converge to deceptive FOSP with low value, highlighting the stickiness of these de- ceptive solutions and how the reshaped terrain can mislead the learning dynamies. These results reveal a core optimization challenge in adversarially robust DRL: adversarial reshaping not only fosters robustness but also introduces deceptive FOSPs that impede consistent policy improvement.
opainirrtte sola
10-
CSP Cor
Paice
0eoppia 135"
-00
-5
-10
-13
Figure 3: Adversarially robust value geometry.
BRIDGING OPTIMALITY AND ROBUSTNESS VIA A BILEVEL FRAMEWORK
The above analysis highlights the strong influence of worst-case adversaries on the optimization landscape. In this section, we seek to mitigate the distorting effects of such adversarial reshap- ing while preserving strong robustness. To this end, we propose a bilevel optimization framework that interpolates between standard and adversarially robust policy optimization by modulating the strength of the adversary. Furthermore, we derive a surrogate for the strongest adversary to construct practical algorithm that smooths the distorted landscape while maintaining robust performance.
4.1 UNIFYING STANDARD AND ADVERSARIALL ROBUST POLICY OPTIMIZATION
To bridge standard and adversarially robust policy training, we introduce the following bilevel op- timization framework that relaxes the inner optimization to allow non-strongest adversaries. This relaxation enables a better balance between adversarial robustness and landscape smoothness:
max (TTOOE(()) (uo), s.t. 1² (0) arg max G(TT, ma).
(General Bilevel ARPO)
Here, the inner objective G(T, v) defines the adversary behavior, extending robustness beyond worst-case scenarios. This formulation generalizes both SPO and ARPO. Specifically, when (s; 0) s, this framework reduces to and when G( v) = - -V (μo)), it recovers ARPO Importantly, within the ISA-MDP setting, this unified framework preserves the same optimal robust policy as both [SPO and ARPO preserving the theoretical alignment of optimality and robustness.
4.2 BILEVEL ADVERSARIALLY ROBUST POLICY OPTIMIZATION WITH KL-SURROGATE
To make the proposed bilevel framework applicable in practice, we need to specify a suitable inner objective G that both promotes policy learning and maintains strong robustness. To this end, consider adversaries practically derived via policy gradient methods, and identify the KL divergence between the original and perturbed policies as an effective surrogate for the strongest adversary.
Theorem 4.1 (Surrogate Adversary). For any policy ne, state s E S, and K > 0, let the adversary be computed via K- -step gradient descent on the adversarial value function, i.e., 0a,k 0a,k-1 7s.k Va, Vmoone (s)la,=a,,, -0,, 1.1 < k < K. Assume that step sizes Tlskk I 1/((>, + 8) for some > O, and G(s;T,V) = ( v-(ss << 1. Then, we have the lower bound of robustness:
(T VTOOMO (s) -TTOS)
26 Xmax (Fs,0)
G(8; no, vs) o (((ss
where Fgg 8 is the Fisher information matrix of log Ro uo(als) with respect to 0, 0, defined as F..0 F(0,s) = Bunnol)))) (V,, log To + 0.) o,)) (Vo, log To(als + 02)lo, max Amax V2 Vroove (8) and Xmmox (.) is the largest eigenvalue of matrix.
The detailed proof is provided in Appendix E Theorem 4.1 indicates that, for gradient-based strong adversaries, the KI. divergence serves valid first-order surrogate for minimizing the adversarial
7

---

Published as a conference paper at ICLR 2026
(a) Hopper
(b) Walker2d
(c) Halfcheetah
(d) Ant
Figure 5: Natural and robust performance of SPO, ARPO, BARPO without guidance, and BARPO for four continuous control tasks in MuJoCo. BARPO (w/o g) consistently outperforms ARPO.
value. This surrogate is both appropriate and reliable from an optimization perspective. Motivated by this insight, we propose the following instantiation of our framework, termed Bilevel ARPO:
max Vaoov (0) (po), s.t. v² (s; 0) arg max DKL (To (-))()s))
(BARPO)
As illustrated Figures and 4 BARPO ef- fectively reshapes the landscape by elevating low-value but robust regions, thereby creating smoother paths toward better FOSPs and even global optima. This restructuring facilitates learning navigability while promoting updates along robust trajectories, ensuring simultane- ous improvements of optimality and robustness. In contrast, SPPO tends to follow vulnerable di- rections, frequently ascending fragile peaks. Meanwhile, ARPOOstrictly follows the most robust path (ariiilee,RRPPPL.....a..
(a) SPO
(b) ARPO
(c) BARPO
Figure 4: Illustrative Optimization Landscapes
but often gets stuck in valleys from overly strong adversaries, hindering escape from poor FOSPs.
We further delve into the theoretical understanding of BARPO in Appendices H.3 and H.4
5 EXPERIMENTS
In this section, conduct comprehensive comparison experiments and ablation studies to empiri- cally substantiate our theoretical insights and evaluate the practical effectiveness of BARPO.
5.1 IMPLEMENTATION DETAILS
Environments and Baselines. Following basclines (Zhang al. 2020c Oikarinen al. 1021 Liang et al.) 2022). we perform experiments on four challenging MuJoCo benchmarks (Todorov al. 2012) with continuous action spaces: Hopper, Walker2d, HalfCheetah, and Ant. We compare BARPO against several state-of-the-art robust training methods. SA-PPO (Zhang et all, 2020c) in- corporates KL-based regularization and solves the inner maximization problem using PGD (Madry 2018) and CROWN-IBP (Zhang et al. (020b), respectively. RADIAL-PPO OOikarinen et al.) 2021 incorporates adversarial regularization guided by robustness verification bounds derived from IBP (Gowal et al. 2019). WocaR-PPO (Liang et al. (2022) estimates worst-case values and also employs KL-based regularization. We use the official umplementations of these baselines train 17 agents per method under identical settings. Regularization coefficients are tuned appropriately for each method. To mitigate the effects of high variance in reinforcement learning training, we report the median performance across 17 runs to ensure reproducibility. More details are in Appendix [F
ct al.
Evaluations. We evaluate the robustness of agents on MuJoCo tasks using six adversarial at- tack methods: (1) Random Attack: adds uniform random noise to state observations; (2) Critic Attack (Pattanaik et al. 2018): perturbs states based on the action-value function; (3) MAD At- tack (Zhang et al.] 2020c): maximizes the discrepancy between policies in clean and perturbed states; (4) RS Attack (Zhang et al.] 2020c): first learns a robust action-value function, then con-
ducts critic-based attacks guided by it; SA-RL (Zhang et al. 202T): employs learned adversary
8

---

Published as a conference paper at ICLR 2026
Table 2: Average returns (+ std) over 50 episodes for baselines and BAR-PPO MuJoCo tasks. Results include natural return, returns under six attacks, the worst return and robustness under these attacks. Bold and underlined values indicate the top and second performances, respectively.
Environment
Method
Natural Return
Return under Attack
Worst-ease Robustness
Random
Critic
MAD
RS
SA-RL
PA-AD
Woest
PPO
3081 135
2923 100
2035 = 1815
1763 1:519
756 2:36
79 2:2
823 1:00
2:2
-0.974
Hopper
0.40%
RADIAL
WocaR
SA
3518 203
5254
3629
2835
3170 151
3637
556
3706 1:11
3657
3662 2.6
3150
2558 16
3045 NC
1507
1407 56
1426 755
1452
993
2124
1256 2:787
1096 1.576
1256 a
9923 117
-0.7 -0.718
-0.677
-0.634
BAR
3684
1340
-0.636
PPO
4662
4612 1:71
1:15
1:975
1062: :: 190
719. 102
336 750
336 250
-0.928
Walker2d
0.00
RADIAL
WocaR
SA
2531
4226
4875
105
2170
4347
4907
: 187
100
2063
4342
5029 4:74
2316
4833
4373
133
1171
3358
2775
12.9
1308
939
3355 à: 9113
2385
426
997 à 1153
1054
1353
9.121
997 à. 1153
1064
4.6
-0. -0.795
.0.752
-0.832
BAR
1732
1718 4:39
5:35
4750
4646 3:33
4285
1192
1192
-0.436
PPO
5048
536
4463
3281
511
1049 90
-213 166
.213
-1. 1.042
Halfchettah
RADIAL
SA
4780
4739
1:41
4983 à 1122
4642 75
50.35
4546
1345
2961
3759
1475
1963
2727
1327
1112
170
1522
1443 1, 100
1551 1138
1968 1294
1445 1882
1927 1112
-0.720
-0.698
WocaR
4723
1796
1816
4543
3302
2270
2:98
2270
-0.519
BAR
1837
2:0
1393
1393
-0.342
PPO
5381 1308
5329 2991
1696 =: 111.
1768 1:127
1097 t:5
-1298 1:318
-3107 +: 9771
-3107 + 1071
-1.577
Ant
RADIAL
SA
4158
$167 1:
4309
5217 4:419
716
5012
1628
5114 244
42015
4396 :: 1225
3742 1:6
4227
2364
2355
1261
4:
18:0
2355 11:33
2364 47
-0:46-2
-0.539
WocaR
4069
3911
3978
3689
3176
1868
1830
1830
-0.550
BAR
4979
4777
1:
162
2825 15V
2825 13V
-0.438
against the victim to perturb the state; (6) PA-AD (Sun et al. 2022) ) trains adversarial agent to identify& a vulnerable perturbation direction, followed by an FGSM attack along that direction.
BAR-PPO. We implement both ARPO and BARPO top of the PPO algorithm (Schulman et al.) 2017]. a widely adopted policy-gradiem method that serves as a standard baseline in adversarially robust DRL research. To improve convergence and efficiency, we integrate SPO dynamics into the bilevel framework using a regularization weight K, resulting in a practical instantiation of BARPO, referred to BAR-PPO. For solving the inner optimization problem, we employ SGLD (Gelfand & Mitter, 1991] to obtain the adversarial perturbation policy n/, In the outer optimization, we treat the inner solution as fixed and omit second-order terms to facilitate efficient updates of the primary policy. A complete description and further implementation details can be found in Appendix F In the following, BARPO refers to the combination of SPO-based guidance with the bilevel framework, BARPO without guidance uses only the bilevel framework. ARPO uses only the maximin framework, and ARPO with guidance improves the maximin training through combining the SPO guidance.
5.2 COMPARISON RESULTS
BARPO Consistently Improves over ARPO. Figure 5 compares the natural and robust perfor- mance of four methods: SPO, ARPO, BARPO without SPO guidance, and BARPO with SPO guid- ance. Detailed evaluation results are provided in Appendix G.. Notably, BARPO without SPO guidance consistently outperforms ARPO across all environments, improving both natural and bust returns. Even in environments like Walker2d and Halfcheetah, it achieves substantial gains in robustness. Specifically, BARPO without SPO guidance improves the natural return by approx- imately 75%. 104%, 156%, and 113% on Hopper, Walker2d, Halfcheetah, and Ant, respectively, while boosting robust return by about 4%. 173%. 209%, and 61% in the worst-case setting. Remark- ably, in the Hopper and Walker2d tasks, BARPO without SPO guidance achieves natural returns on par with state-of-the -fethe-rrt methods. As further illustrated in Figure 6 BARPO improves the training dynamics by guiding policy learning toward regions in the landscape that yield higher returns.
BAR-PPO Exhibits Superior Natural and Robust Performance. As shown in Table BAR-PPO demonstrates strong overall performance across all four MuJoCo environments, achieving both high natural returns and robust resilience to adversarial perturbations. In particular, BAR-PPO outper- forms vanilla PPO in terms of natural returns performance on Halfcheetah and Ant. In terms of worst-case robust returns, BAR- significantly surpasses all baselines, with improvements of 4%, 154%, 40%, and 20% over the second best, spectively. Notably, BAR-PPO achieves the highest robustness across Walker2d, Halfcheetah, and Ant, and is on par with the strongest robustness on Hopper, with negligible gap of only 0.3%.
Hopper and Walker2d, and achieves comparable

---

Published as a conference paper at ICLR 2026
Hopper
Walker2d
(c) Halfcheetah
(d) Ant
Figure 6: Natural training curves of ARPO and BARPO with and without SPO-guidame
Table 3: Comparisons between BARPO and ARPO with SPO-based guidance.
Eaviromment
Mithed
Natural Reture
Return under Attack
Worst-ciee Robustaes
Rundom
Critic
MAD
RS
S.A
BA-AD
Wasst
Hopper
ARPO (
3699 1
1652 m:
3693 2.
5132 W0
1130 2:30
1278 319
1756 415
11.00
-0.094
BARPO
3654
2 0.49
3692
3678 1
3487 V1
1340 4:47
1728 4:00
1908 412
1340 2:47
19%
-0.636
K.4%
Walker2d
ARPO (m g)
4206 3
4231 4:4
4253 4:1
4226 4:
3130 104
5565 008
2496 + 1871
2496 187)
-0.406
2 0:00
BARPO
4732
23 119
4718 9.
4727 5:14
4750 4.
4646 1:15
4285
1757
2699
c 130
2609
130
18. 1%
-0.436
[7.4%
Halfcheeiah
AREO (w g) BARPO
4997 91
4934 -:
5040 6:3
2570
6 1558
3605 4:4
1356
134
1086
758
1086
754
-0.783
4857
4 13325
4741 6:50
4803
6:34
4729 105
4265 1877
3894
1323
3181 à 19M
3181
1990
1956
-0.342
156
Ant
AREO 19 2) BARPO
5.90 13
5227
1:1
4768
1:10
4059 X21
2670
139
1185
2.35
2
1185
-0.773
Se 0.10L
5024
mn 6.8%
4979
4777
1
4843
139
4171 5t
3367 : W2
2825 159
2828
777
138%
-4.48
1-43%
5.3 ABLATION STUDIES
Effect of SPO-Guidance. As shown in Figures 5 and 6 incorporating 3PO-based guidance further enhances the natural performance of BARPO, particularly in more complex environments such as Halfcheetah and Ant. In certain cases, the enhancement in natural returns is accompanied by corre- sponding improvements in robust returns, as observed in the Hopper and Ant tasks. However, this incorporation often involves trade-off, where improvements in natural performance may lead to reduction in the worst-case robustness, as presented in the Walker2d and Halfcheetah environments.
Comparisons between BARPO and ARPO with SPO-Guidance. To further validate the effec- tiveness of BARPO, we compare it with ARPO enhanced by SPO-based guidance. As presented in Table 3 on the Hopper, Halfcheetah, and Ant tasks, BARPO exhibits a slight reduction natu- ral returns compared to ARPO with guidance, approximately 0.4%, 3.2%, and 6.8%, respectively. However, BARPO achieves significantly higher robust returns and markedly stronger robustness, with improvements of 19%, 193%. 138% in robust returns, and 8.4%, 56%, and 43% in robustness metries. On the Walker2d task, BARPO attains gains in both natural and robust returns, with a mi- nor compromise in robustness. These findings indicate that although integrating SPO guidance into ARPO can enhance performance to some extent, this direct combination does not effectively miti- gate the reshaping effect induced by the strongest adversary. Consequently, numerous sticky FOSPs with low returns persist, posing challenges for optimization. In contrast, BARPO substantially re- shapes the landscape, enabling the discovery of improved navigation that yields higher returns.
We also conduct additional ablations in Appendices G.6 and G.7 to identify the effects of bilevel structure, SPO guidance, and KL surrogate. These further validate the effectiveness of BARPO.
6 CONCLUSION
In this paper, we explore whether the theoretical alignment between optimality and robustness can be realized in practical policy optimization, and propose BARPO, novel bilevel framework that uni- fies standard and adversarially robust policy optimization. Through extensive analysis, we reveal a fundamental tradeoff between optimality and robustness in existing policy gradient methods, expos- ing a critical gap between theory and practice. Notably, our work demystifies the underlying cause of this tension from the perspective of optimization landscapes and value space geometry. Specifi- cally, the strongest adversaries in ARPO significantly reshape the learning landscapes by introducing numerous sticky and deceptive local optima. While this promotes robustness, it also hinders effec- tive policy improvement by making the landscape more rugged and difficult to navigate. BARPO refines the landscapes by modulating the adversary strength, thereby creating pathways toward su- perior policies and reducing the prevalence of poor sticky local optima. Surprisingly, this bilevel framework empirically showcases the potential for approaching the globally optimal robust policy, suggesting promising step toward bridging theoretical guarantees and practical performance.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All authors of this paper have carefully reviewed and adhered to the ICLR Code of Ethics,
REPRODUCIBILITY STATEMENT
All benchmarks, well the experimental setup, are based on derived from open-sourced work. We have given relevant references in the paper, and the details of our algorithm and experiments are provided in Section 5 and Appendices and G
We have provided detailed analysis and complete proofs for each theorem and proposition in the Appendices and
ACKNOWLEDGMENTS
Congying Han, Haoran Li, Jiayu Lv, and Tiande Guo acknowledge funding support from the Na- tional Key R&D Program of China (2021YFAION) YFA and the National Natural Science Founda- tion of China (Nos. 12431012, U23B2012).
REFERENCES
Alckh Agarwal, Sahand Negahban, and Martin J Wainwright. Fast global convergence of gradient methods for high-dimensional statistical recovery. The Annals of Statistics, 40(5):2452, 2012.
Alckh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept. UW Seattle, Seattle, WA, USA, Tech. Rep, 32:96, 2019.
Alckh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. International Conference Machine Learning, pp. 242-252. PMLR, 2019.
Heinz H Bauschke, Jérôme Bolte, and Marc Teboulle. descent lemma beyond lipschitz gradient continuity: first-order methods revisited and applications. Mathematics of Operations Research, 42(2):330-348, 2017.
Heinz Bauschke, Minh N Dao, and Scott B Lindstrom. Regularizing with bregman-moreau envelopes. SIAM Journal on Optimization, 28(4):3208 -222, 2018.
Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induc- tion attacks. In Machine Learning and Data Mining in Pattern Recognition: 13th International Conference, MLDM 2017, New York, NY, USA, July 15-20, 2017, Proceedings 13, pp. 262-275. Springer, 2017a.
Vahid Behzadan and Arslan Munir. Whatever does not kill deep reinforcement learning, makes stronger. arXiv preprint arXiv:1712.099: 2017b.
Roman Belaire, Arunesh Sinha, and Pradeep Varakantham. On minimizing adversarial counterfac- tual emoor in adversarial reinforcement learning. In The Thirleenth International Conference Learning Representations, 2025.
Shubham Bharti, Xuezhou Zhang, Adish Singla, and Jerry Zhu. Provable defense against backdoor policies in reinforcement learning. Advances in Neural Information Processing Systems, 35: 14704-14714, 2022.
Jérôme Bolte, Shoham Sabach, and Marc Teboulle. Proximal alternating lincarized minimization for nonconvex and nonsmooth problems. Mathematical Programming, 146(1):459-44 494, 2014.

---

Published as a conference paper at ICLR 2026
Jéröme Bolte, Shoham Sabach, Mare Teboulle, and Yakov Vaisbourd. First order methods beyond convexity and lipschitz gradient continuity with applications to quadratic inverse problems. SIAM Journal on Optimization, 28(3):2131-2152 2018.
Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo Chao Zhang, Songan Zhang, and Tuo Zhao. Robust multi-agent reinforcement learning via adversarial reg- ularization: Theoretical foundation and stable algorithms. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End- to-end autonomous driving: Challenges and frontiers. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Long Chen, Yuchen Li, Chao Huang, Bai Li, Yang Xing. Daxin Tian, Li Li, Zhongxu Hu, Xiaox- iang Na, Zixuan Li, et al. Milestones autonomous driving and intelligent vehicles: Survey of surveys. IEEE Transactions on Intelligent Vehicles, (2):1046-1056, 2022.
Robert Dadashi, Adrien Ali Taiga, Nicolas Le Roux, Dale Schuurmans, and Mare G Bellemare. The value function polytope in reinforcement learning. In International Conference Machine Learning, pp. 1486-1495. PMLR, 2019.
Kuangyu Ding, Jingyang Li, and Kim-Chuan Toh. Nonconvex stochastic bregman proximal gradient method with application to deep learning. Journal of Machine Learning Research, 26(39):1 2025.
Juncheng Dong, Hao-Lun Hsu, Qitong Gao, Vahid Tarokh, and Miroslav Pajic. Variational adversar- ial training towards policies with improved robustness. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS), 2025.
Dmitriy Drusvyatskiy and Adrian S Lewis. Error bounds, quadratic growth, and linear convergence of proximal methods. Mathematics of operations research, 43(3):919 948, 2018.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, pp- 1675- 1685. PMLR, 2019.
Yu Feng and Yuhai Tu. The inverse pariance-flatness relation in stochastic gradient descent is critical for finding flat minima. Proceedings of the National Academy of Sciences, 11809):20016617118 2021.
Mare Fischer, Matthew Mirman, Steven Stalder, and Martin Vechev. Online robustness training for deep reinforcement learning. arXiv preprint arXiv:1911.00887, 2019.
Tim Franzmeyer, Stephen Marcus McAleer, Joao F Henriques, Jakob Nicolaus Foerster, Philip Torr, Adel Bibi, and Christian Schroeder de Witt. Illusory attacks: Information-theone detectability matters in adversarial attacks. In The Twelfth International Conference Learning Representa- tions, 2024.
Saul B Gelfand and Sanjoy K Mitter. Recursive stochastic algorithms for global optimization in r'd. SIAM Journal on Control and Optimization, 2955):999-1018 1991.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations, 2020. URL https: /ttp:://pppeeevieeeeee wtt.teewwweeeeee.... net. fortm//frum? o HJggMFWW
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Sven Gowal, Krishnamurthy Dj Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann. and Pushmeet Kohli. Scalable verified training for provably robust image classification. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 4842-4851 2019.
12

---

Published as a conference paper at ICLR 2026
Zhaoyuan Gu, Zhenzhong Jia, and Howie Choset. Adversary a3e for robust reinforcement learning. arXiv preprint arXiv: 1912.0033, 2019.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in Ilms through reinforcement learning. Nature, 145:633-638 2025. doi: 10.10088441588--0550944442-----2
Junfeng Guo, Ang Li, Lixu Wang, and Cong Liu. Policycleanse: Backdoor detection and miti- gation for competitive reinforcement learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4699-4708, 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. Transactions Machine Learning Research, 2023. ISSN 1835-8856. URL httpttt://ppeeeeeeeeeeee
net, forfroouumme wtt.:rmmmdmcccca2225
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. In 5th International Conference on Learning Representations (ICLR) Workshop Proceedings, 2017.
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 40(4-5):698-72) 2021.
2021.
Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and countermeasures for adversarial attacks on deep rein- forcement learning. IEEE Transactions on Artificial Intelligence, 3(2):90-109, 2021.
Matthew Inkawhich, Yiran Chen, and Hai Li. Snooping attacks on deep reinforcement learning- In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Sys- tems, AAMAS '20, pp. 557-565, Richland, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 7781450375184
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dynamics. Advances in Neural Information Processing Systems, 33:8127-8133 2020.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the polyak-lojasiewier condition. In Joint European conference ma- chine learning and knowledge discovery in databases, pp. 795-811. Springer, 2016.
Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li. Trojdrl: evaluation of backdoor attacks on deep reinforcement learning. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2020.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 236))4900--4926, 2021.
Ezgi Korkmaz. Adversarial robust deep reinforcement learning requires redefining robustness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8369-8377, 2023.
Jernej Kos and Dawn Song. Delving into adversarial attacks deep policies. In International Conference on Learning Representations Workshop, 2017. URL httttt:t/ppeeeeeee/// ///pp//eeee/....//////// het//orrmmid=aJeebbe
13

---

Published as a conference paper at ICLR 2026
Aounon Kumar, Alexander Levine, and Soheil Feizi. Policy smoothing for provably robust inforcement learning. In International Conference on Learning Representations, 2022. URL
ltttps://ppenreiier net/ htwp://ewwwee..nttooooureeeeeee
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. Train- ing language models to self-correct via reinforcement learning. In The Thirteenth International Conference Learning Representations, 2025, URL htt.http////peeeeiiee... 1d=CjwERcAUTw,
Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert D Nowak. fully first-order method for stochastic bilevel optimization. In International Conference on Machine Learning, pp. 18083- 18113. PMLR, 2023.
Jacho Lee and Maxim Raginsky. Minimax statistical learning with wasserstein distances. Advances in Neural Information Processing Systems, 31, 2018.
Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. Convex and non-convex optimization under generalized smoothness. Advances in Neural Information Processing Systems, 36:40238-4000 40271, 2023.
Haoran Li, Zicheng Zhang, Wang Luo, Congying Han, Yudong Hu, Tiande Guo, and Shichen Liao. Towards optimal adversarial robust q-learning with Bellman infinity-error. In Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 29372. PMLR, 21-27 Jul 2024. URL https: s/proceedings mmr...mrr.... wtt.:..rese///222222 www.///2//1........ html
Haoran Li, Zicheng Zhang, Wang Luo, Congying Han, Jiayu Lv, Tiande Guo, and Yudong Hu. Towards optimal adversarial robust reinforcement learning with infinity measurement error. arXiv preprint arXiv:2502.16753 2025.
Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning, Advances in Neural Infor- mation Processing Systems, 35:22547-225611 2022.
Yongyuan Liang, Yanchao Sun, Ruijie Zheng, Xiangyu Liu, Benjamin Eysenbach, Tuomas Sand- holm, Furong Huang, and Stephen Marcus McAleer. Game-theoretic robust reinforcement learn- ing handles cooppee perturbations. In The Twelfth International Conference on Learn- ing Representations, 2024. URL
net/ www.ommiiddwwlll.. H
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In In- ternational Conference on Learning Representations, 2016. URL htttps:/
Yen-Chen Lin, Zhang- Wei Hong, Yuan- Hong Liao, Meng-Li Shih, Ming- Liu, and Min Sun. Tac- tics of adversarial attack on deep reinforcement learning agents. In International Joint Confer- ence on Artificial Intelligence, pp. 3756-3762, 2017. URL
249647
ijcai. 2017/525
Xiangyu Liu, Souradip Chakraborty, Yanchao Sun, and Furong Huang. Rethinking adversarial policies: A generalized attack formulation and provable defense in RL. In The Tweifth Interna- tional Conference on Learning Representations, 2024a. URL htttss://pperreieeee net/
Eorum? www.::ddlpbbbbccccmmm 1PMG.
Xiangyu Liu, Chenghao Deng, Yanchao Sun, Yongyuan Liang, and Furong Huang. Beyond worst- case attacks: Robust RL with adaptive defense via non-dominated policies. In The Twelfth In- ternational Conference on Learning Representations, 2024b. URL htttttts///peerreee
Zhihao Liu, Quan Liu, Wenjun Xu, Lihui Wang, and Zude Zhou. Robot learning towards smart Manufacturing. 77:102360
robotic manufacturing: A review. Robotics and 2022.
14

---

Published as a conference paper at ICLR 2026
Chris Lu, Timon Willi, Alistair Letcher, and Jakob Nicolaus Foerster. Adversarial cheap talk. In International Conference on Machine Learning, pp. 22917-22941. PMLR, 2023.
Zhaosong Lu and Sanyou Mei. First-order penalty methods for bilevel optimization. SIAM Journal on Optimization, 34(2):997-1199) 2024.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu, To- wards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations, 2018. URL httt:htts////tto httttpps//ppeereiiieee net/ (wworrmmiddriziiee22.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529-533 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp. 1928-1937. PMLR, 2016.
Takashi Mori, Liu Ziyin, Kangqiao Liu, and Masahito Ueda. Power-law escape rate of sgd. In International Conference on Machine Learning, pp. 15959-15975. PMLR, 2022.
Ion Necoara, Yu Nesterov, and Francois Glineur. Linear convergence of first order methods for non-strongly convex optimization. Mathematical programming, 175(1):69-077, 2019.
Buqing Nie, Jingtian Ji, Yangqing Fu, and Yue Gao. Improve robustness of reinforcement learning against observation perturbations via Loo lipschitz policy networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 14457-14465, 2024.
Arnab Nilim and Laurent Ghaoui. Robustness in markov decision problems with uncertain transition matrices. Advances in neural information processing systems, 16, 2003.
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui- Weng. Robust deep reinforcement learning through adversarial loss. Advances Neural Information Processing Systems, 34:26156-261 2021.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Ro- bust deep reinforcement learning with adversarial attacks. In International Conference on Au- tonomous Agents and Multiagent Systems, pp. 2040-2042, 2018. URL acm. org/
ci citat oon wfwmid==mmmmm..... 1d-3133033334
Lemrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International conference on machine learning, pp. 2817-2826. PMLR, 2017.
Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Policy gradient in lipschitz markov decision processes. Machine Learning. 100:255-288 283, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. International Con- ference on Learning Representations, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pp. 8707- 8718. PMLR. 2020.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pp- 387-395. PMLR, 2014.
15

---

Published as a conference paper at ICLR 2026
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifiable distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018. URL httpt:t:////m pttttps://ppenreviieie
net. forum? Hx-
Chung-En Sun, Sieun Gao, and Tsui-Wei Weng. Breaking the barrier: Enhanced utility and robust- ness in smoothed DRL. agents. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 46957-46987, PMLR, 21-27 Jul 2024. URL BtLLhhtp::///// hattps::/ppooceddingg wtt.iigssm.....ssss..mmm
press/essseee..ee. / v235/ sun2 4b. .tmml
Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. In International Conference on Learning Representations, 2022. httphtts: http:/ppprrrreeeeeeeeee/ee net fott//foum??? JM2KFbbJJJvV
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9-44, 1988.
Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martin-Martin, and Peter Stone. Deep reinforcement learning for robotics; A survey of real-world successes. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 39, pp.
2025.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Lun Wang, Zaynah Javed, Xian Wu, Wenbo Guo, Xinyu Xing, and Dawn Song. Backdoorl: Back- door attack against competitive reinforcement learning. In International Joint Conference on Ar- rificial Intelligence, pp. 6699-3775, 2021. URL https: //t.. wtg.:/..0000000..///3103 2021/509
Mingze Wang and Lei Wu. A theoretical analysis of noise geometry in stochastic gradient descent. arXiv preprint arXiv:2310.00692, 2023.
Yisen Wang. Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In Kamalika Chaudhuri and Ruslan Salakhut- dinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6586 6596.66555.0 PMLR, 09-15 Jun 2019. URL
mlr wwwppppwwssssweeeeeeepep www.wnngg..hnll....
Tsui- Wei Weng, Krishnamurthy Dj Dvijotham, Jonathan Uesato, Kai Xiao, Sven Gowal, Robert Stanforth, and Pushmeet Kohli. Toward evaluating robustness of deep reinforcement learning with continuous control. In International Conference on Learning Representations, 2019.
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathe-
matics of Operations Research, 38(1):153-183, 2013.
Stephan Wojtowytsch. Stochastic gradient descent with noise of machine learning type part ii: Continuous time analysis. Journal of Nonlinear Science, 34(1):16, 2024.
Fan Wu, Linyi Li, Zijian Huang, Yevgeniy Vorobeychik, Ding Zhao, and Bo Li. CROP: Certifying robust policies for reinforeement learning through functional smoothing. In International Confer- ence on Learning Representations, 2022a. URL
net/ wttp://uttteooummmimmem forum?id
HO HOjLHrl Zhmx!
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes as sgd. In International Conference on Machine Learning, pp. 10367-10376. PMLR, 2020.
Lei Wu, Mingze Wang, and Weijie Su. The alignment property of sgd noise and how it helps select flat minima: A stability analysis. Advances in Neural Information Processing Systems, 35:4680- 4693, 2022b.
16

---

Published as a conference paper at ICLR 2026
Huajian Xin, Z.Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu. Qiushi Du, Wenjun Gao, Haowei Zhang, Qihao Zhu, Dejian Yang, Zhibin Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search. The Thirteenth International net. f
Gou, Z.F. Wu, Fuli Luo, and Chong Ruan.
Conference Learning Representations, 2025. URL httthttts///ppneeeeeee id I4YAIwrsXa
Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. Advances Neural Information Processing Systems, 13:11511--15211 2020a.
Bohang Zhang, Du Jiang, Di He, and Liwei Wang. Rethinking lipschitz neural networks and certified robustness: A boolean function perspective. Advances in Neural Information Processing Systems, 35:19398-1944 2022.
Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsich. Towards stable and efficient training of verifiably robust neural networks. In International Conference on Learning Representations, 2020b. net/forru d=dkkkukkkk rFWB
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Ad- vances in Neural Information Processing Systems, 13:21024-210311 2020c.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning
on state observations with learned optimal adversary. In International Conference on Learning Representations, 2021. URL https: wttp:///ww/peeeeeeeeeeeeeeeeeeee net ottp/////////////rrer........e.. htt.:////epeeeeeeee.eeeeeeettaea /ftr:u22
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Rep- resentations, 2020d. / www.:nnettnndidiiieooooo.
net/ forumm BdBBiidXpBJBBBWe
Zhanpeng Zhou, Mingze Wang, Yuchen Mao, Bingrui Li, and Junchi Yan. Sharpness-aware mini- mization efficiently selects flatter minima late in training. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https:/ https////peeeeee..eomom
abnu
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In Ka- malika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Con- ference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp- 7654-7663. PMLR, 09-15 Jun 2019. URL httpt::///// fttt:::/pproceedine ngs. ml r.press/v97 thu19e.. htmll
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine learning, 109:467-492, 2020.
17

---

Published as a conference paper at ICLR 2026
A OUTLINE OF APPENDIX
We review related work on adversarial attacks and defenses for reinforcement learn- ing against perturbations on state observation.
Appendix BJ
Appendix C Detailed theoretical analysis of ARPO, including proofs for the adversary's 's policy
gradient (Theorem 3.1 in Appendix C.T convergence (Theorem 3.2) in Appendix C.2 and learning
dynamics in Appendix
Appendix D:
An intuitive analysis a toy ISA-MDP, corresponding to Propositions 3.1 and 3.2
Appendix The analysis and proof for the KL-based surrogate adversary (Theorem
Appendix F+ Additional implementation details, including pseudocode for ARPO (Appendix
F.1
and BARPO (Appendix H.2), hyperparameter settings (Appendix H.3) and compute resources
Appendix F.4).
Appendix G Additional experimental results, including detailed comparisons (Appendix G. statistical significance analysis (Appendix (.2). extended experiments on Humanoid (Appendix G.3). and ablation studies on the regularization coefficient к (Appendix G.4) and our proposed components Appendices/G..i and G.7).
Appendix H: Further discussions clarifying our contributions and convergence proof (Appendix the novelty of the bilevel framework (Appendix H.2). and a deeper theoretical examination of
H.1
BO (Appendices H.3 and
Appendices I and J: A discussion of limitations, future work, and a clarification on the use of large language models.
B RELATED WORK
B. ATTACKS ON STATE OBSERVATIONS OF DRL AGENTS
Adversarial Attacks on State Observations of DRL Agents. The susceptibility of DRL agents to adversarial attacks was initially exposed by Huang et al. (2017). who demonstrated that DRL policies could be significantly disrupted using the Fast Gradient Sign Method (FGSM) (Goodfel- low et al. 2013) in Atari environments. This seminal discovery laid the foundation for wave of subsequent studies on adversarial strategies and policy robustness in DRL. Building on this, Lin cet al. (211)) and Kos & Song (017)) proposed limited-step perturbation techniques aimed mis- leading DRL policies, revealing that even small, strategically applied modifications could impair agent behavior. In further advancement, Pattanaik et al. (2018) exploited the critic function along- side gradient descent to craft adversarial inputs that degrade policy performance more systemati- cally. [Behzadan & Munir (2017a) extended the threat model by introducing black-box attacks on DQN, confirming that adversarial examples could successfully transfer across different network ar- chitectures, thereby emphasizing their real-world applicability. Meanwhile, Inkawhich et al. (2020) demonstrated that even adversaries with access restricted to action and reward signals, without in- ternal model information, could still inflict considerable harm. In the domain of continuous control, Weng et al.] (2019) devised two-step adversarial framework leveraging learned dynamics models to generate more effective attacks. Furthermore, Zhang et al. (202T) and Sun et al.) (2022) advanced the field by training adversarial agents using reinforcement learning, giving rise to adaptive attack strategies known as SA-RL and PA-AD. Korkmaz (2023) investigated adversarial directions in the Arcade Learning Environment and discovered that state-of- the-art art robust agents (Zhang al.) 2020c Oikarinen et al. (202]) remain susceptible to perturbations aligned with policy-independent sensitivity axes. Most recently, [Ling et al. (2024) proposed temporally-couple attack strategy that exploits temporal correlations to further compromise the performance of robust policies.
Other Attacks on DRL Agents. Research by Kiourti et al. (2020): Wang et al. (2021); Bharti
ct al. (2022); Guo et al. (2023) delved deeper into backdoor attacks within reinforcement learning, revealing profound vulnerabilities that pose significant threats policy integrity. In novel contri- bution, Lu et al. (023) proposed an adversarial cheap talk framework, where an attacker is trained via meta-learning to manipulate agent behavior through indirect communication. Franzmeyer et al. (2024) introduced a dual ascent- bssent-based approach to train an end-to-end illusory attack capable of deceiving DRL agents effectively. In the context of multi-agent environments, Gleave et al. (2020)
18

---

Published as a conference paper at ICLR 2026
examined how adversarial policies manipulate or destabilize cooperative competitive dynam- ics among agents.
Collectively, these studies reveal the broad and persistent vulnerability of DRL agents adversarial manipulation across diverse threat models and environments. From observation perturbations and black-box attacks to communication- bbasd deception and multi-agent exploitation, adversaries have consistently demonstrated the ability to significantly degrade policy performance, highlighting the urgent need for continued advancements in defense mechanisms.
B.2 DEVELOPING ROBUST DRL AGENTS AGAINST PERTURBATIONS ON STATE OBSERVATION
Early Attempts to Develop Robust Agents. Early efforts by Kos & Song (2017): Behzadan & Munir (2017b) explored incorporating adversarial states into the replay buffer during training on Atari environments, although these approaches yielded only limited robustness improvements. To address this, |Ficcher et al. (2019) proposed decoupled architecture for DQN, separating it into Q- network and policy network, and robustly trained the policy component using generated adversarial states under provable robustness guarantees.
Smoothing Techniques. Shen et al. (2020) demonstrated that applying smoothness regularization can simultaneously improve both the natural performance and robustness of TRPO (Schulman et al., Mieea,WWeerr et al. (0022) employed Randomized Smoothing (RS) techniques to achieve certifiable robustness guarantees. More recently, [Sun et al. (2024) proposed an innovative smoothing approach designed to mitigate the common issue of robustness overestimation, thereby enhancing both natural and robust returns in tandem. Importantly, these smoothing methods are complementary to our work and can be seamlessly integrated into our proposed algorithms.
2015 and DDPG (Silver et al. 2014) algorithms. Building on this idea, Wu al. (2022a) and Kumar
Robust Multi-Agent DRL. In the context of multi-agent systems, He et al. (2023) investigated
state adversaries within Markov Games and developed robust multi-agent Q-learning and actor- critic algorithms achieve robust equilibrium solutions. Building on robustness regularization
techniques (Shen et al. 2020 Zhang et al. 2020c), Bukharin et al. (2024) extended these methods to multi-agent environments by considering sub-optimal Lipschitz continuous policies smooth settings. Liu et al. (2024a) proposed an adversarial training framework employing two timescales to ensure effective convergence towards robust policies. Another research direction involves alter- nating training between agents and learned adversaries (Zhang et al. [202]] Sun et al. (022), which Liang et al. (2024) further formalized within a game-themmreeee theoretic framework.
Beyond the Worst-Case Robustness. Furthermore, Liu et al. (2024b) introduced an adaptive de- fense mechanism during testing, leveraging set of non-dominated policies. The latest work by Belaire et al. (2025) considers the partial observability setting and balances the value optimization with robustness based on beliefs.
Adversarially Robust Training for DRL Agents. In this paper, we mainly focus on the adversari- ally robust training paradigm. Zhhng et al. (2020c) formally modeled atate-adversarial reinforcement learning as a state-adversarial Markov Decision Process and highlighted the potential absence of an Optimal Robust Policy. To tackle this, they introduced to balance robustness with nominal performance. Building on this, Oikarinen et al. (202T) employed certified robustness bounds construct the adversarial loss, which they integrated with standard training objectives. Liang et al. (2022) enhanced training efficiency by estimating worst-case value functions and combining them with classical Temporal Difference (TD) target (Sutton] 19888) or Generalized Advantage Estimation (GAE) (Schulman et al.) 2016). NNe et al. (2024) proposed DRL framework for discrete action spaces based on SortNet (Zhang et al. 2022), which ensures global Lipschitz continuity and eliminates the need for training additional adversarial agents. More recently, Li et al.) (0024 2025) developed the universal ISA-MDP framework, within which they formally proved the existence of the ORP and demonstrated its consistency with Bellman optimality error is a necessary condition for adversarial robustness. However, their theoretical analysis is limited to worst-case settings for RE-
beddstanneebbsed regularization technique
policies. Moreover, they established that an aufinity-masuuredd
19

---

Published as a conference paper at ICLR 2026
INFORCE and PPO. In contrast, our analysis builds upon practical policy gradient methods, making it broadly applicable to real-world policy optimization.
These studies underscore the crucial importance of developing robust DRL policies and highlight the ongoing challenges in enhancing adversarial robustness DRL agents.
B.3 ROBUST RL AGENTS AGAINST PERTURBATIONS ON TRANSITION DYNAMICS
Research robustness against transition uncertainty PPito al. 2017 Gu et al. 2019 Ka-
malaruban et al. 2020; Dong et al. 2025 is primarily formulated as Robust Markov Decision Pro-
cesses (RMDPs) Nilim & Ghaoui 2003 Iyengar 2005; Wiesemann al. 2013). This paradigm differs fundamentally from SA-MDP. In SA-MDP, the adversary corrupts the policy input, forcing the agent act on distorted observations. Consequently, robustness depends on the policy's stabil- ity with respect to state variations. In contrast, RMDPs involve perturbations to the environment's dynamics. The agent observes the true state, but the transition leads a worst-case outcome. Since the decision precedes the perturbation, state-space smoothness does not guarantee robustness against dynamic shifts.
C ADVERSARIALLY ROBUST POLICY OPTIMIZATION
In this section, we present a detailed theoretical analysis of ARPO. the subsection C.1 we show the analysis of policy gradient for the adversary. In the subsection C.2 we show the convergence analysis of ARPO. And in the subsection C.3 we explore the learning dynamics in the neighborhood of FOSPs in both ARPO and SPO.
C.I POLICY GRADIENT FOR ADVERSARY
Theorem C.1 (Policy Gradient for Adversary), Given a policy no. for all state E S, we have
V Viloos (8) E E
R(T) v log
t=
=ErnnoorsP
72t * a.) Vols
Big at matoorr (Qmecua V') log [ala'/ooss
Specifically, consider the direct parameterization representation for adversary va S - S + 8 + 02 E B(x). Then, for any state 8, E S, we have the state-wise policy gradient for the adversury
Vo, (
R(T) 0., log no (ar|s: 0z.)) T(s: 8:)
B
-ErnrgOM .
Σ 'QTrooo (st, 0e) Vo., log ( + 0m.)) I[st 8i)
Ess at ooudoore
[Qmoovs d'),''VVo,, log oo''' + +x) I(s' = =s)]]
where drov is the state-action visitation distribution under and II(.) is the indicator function. Remark C.1. Equation 2 indicates that for computing the strongest adversary, we can decompose into calculating the strongest adversary in every state after sampling.
Proof. The definition of (R
Thoove (8) = Err
0x)]s0 = = Errnsovo [ =
20

---

Published as a conference paper at ICLR 2026
The probability of attaining trajectory distribution Ho is
{su, do, 81, 01.... under ne and vo with initial state
Po,0(T) = (o(s0)=o(oollss(50)
II aislls-)..0-
[oo (au so) II 1(s (
For any state 8, compute the gradient of the adversarial value function with respect to
V Voove (8)
Va
((T))P,,(T|500 =
R(T)VoPo.o(Tlo)
R(F))o,,(T|ss =) log Po s)
(o,)))
V log( no o (ayls))
E R(T)
log( (4)(1)))))
E
R(T)
Σ Ve log (or
Let Tx = {s8, 00, 81, ..,...,......................... St, a.). Furthermore, we have that
v Voloo (8)
Br
log loggo(ac(u((st))
"r (ogg
1=0 1=0
E
Ta-1
i=0
E
E
Σ
(r((s az) + Exh(esnaa)
Vo Volggmo(a Ta-1
B
Ers
Vo T
1=t
B
"r(88, (:) Va logTo(arlussss))
i=!
E
B
1'r(si, ai) Vo Vo log To lo Ta
i=t
B
Σ rrr(s,, s Vo log TO (ololoo()))))
(QToous((s,,)) 0z) Vo Voloomo
21

---

Published as a conference paper at ICLR 2026
Algorithm Adversarially Robust Policy Optimization with d-Approximation Adversary
Input: training steps K in outer loop, outer loop step size lek inner loop PGD step Kp, inner loop PGD step size t/p, inner loop approximation accuracy 6, batch size
Output: adversarially robust policy no
1: Initialize and 0
2: for k = to K do
3: Initialize indicator vector / ||s PGD step number j
4: while I|| > 0 & < Kp
5: Sample trajectories 7 ~ Ty va P until the batch is full.
6:
Compute the unbiased estimation of the action-value function Quooro Q ( =
Eez 7 "rsr,, a.)), for all (St, oe) in the batch
7:
for in the batch do
8:
(a()) < vs(s) V/P sigu (Qroore (s, a) Vo log (a(aleu(s)) [(s)
9:
va (8) clip(uo (8), 6, +)
10:
I(s) = e (vn(s) - Bus I- Quoono a) Vo log Te (aleo(())) > 6)
11:
j +- j+1
12:
end for
13:
end while
Compute the unbiased estimation the gradient of the value function Vo VoTooo (po):
VoVngove (mw) := (--) Eimnee Queara (s,a) Vo log no (a (a)))(s))
Tu Voooe (nO)
16: end for
Furthermore, through the definition of the state-action visitation distribu- tion following the policy (e (-1a)) [(1 7) Dee 7 "I(s, 8 d')lso s). have that
o va starting from s: d (s', a')
1 Be (a) adddoro Tous (8, a Va log (al
(OV =
Specifically, for the representation for adversary v4 S + + 0. E B(s), we have
Vo., Fooue
R(T)
Vo., log (a(a)(2((5))
1=0
EE
R(T))
Vo,, log To + 0o.)
t=
Vo., log Te(ai + v..) Ilss = si)
R(T)
Σ
1=
Similarly, we have
2
Va Vioooo (s) ETTTYORS
x Qooooon (St, a, Va., lognola;1s + S,,) I(s; =;i)
1
De (f Tog'o + +x) I1s'' = s;)]
This completes the proof.
C.2 CONVERGENCE OF ADVERSARIALLY ROBUST POLICY OPTIMIZATION
We analyze the convergence properties of ARPO with 3-Approximation Adversary (Algorithm) I in this subsection.
22

---

Published as a conference paper at ICLR 2026
C.2.1 NOTATIONS
For sampling trajectory 7 (su, ag. ro....) 2g o v0, denote the unbiased estimation of the action-value function Quoove as
Quoove (Se, (e) =
I ae), V(si, ax) E
U2!
For simplicity of writing and reading of the latter, denote the sampled estimate as:
as
1
sg
(TTOeee (s, log ITe (alvo(s)),
whereas sg(-) is the stop-gradient operator, meaning that Quaori (s, a) is seen function accord- ing to (s, a) and is detached from the gradient operation for (0, 0) in the following context. This implies that
= 1
Vo
Quoove (8, a) Vo log no (a)va(s))
a; 1 Tuoove (s, a) V log ग (alv)(s)).
Furthermore, denote the unbiased estimation of the gradient of the value function VgV Vooo (mo) as
V Viono (Po)
Σ
V,
(s.a)Er
and denote the unbiased estimation of the gradient of the value function V
Vxoova (mo)
1
V Vnsovo (Mo)
Foo(s,a;8,0)
[T]
(s,a)Er
C.2.2 PREPARATIONS
We quantify the solution of inner optimization, that is, the adversary, by extending the First-Order Stationary Condition (FOSC) proposed by Wang et al. (201)).
Definition C.1 (6-Approximation Adversary). For a given policy wo and the sampled estimation
Quaovo (s, a). vo(s) satisfies the following condition:
(V)()) Sui
Quoovo (s,a) V log गe (alpo(s)) / S
max BvE
then, we called that the us(s) is 5-approximation adversary for the strongest adversary v² v"1 mp)-
Before our main convergence analysis, we provide a few assumptions utilized our analysis.
Assumption C.1 (Lipschitzz of Sampled Policy Gradient). The function ö(s, a; 0, 0) satisfies the gradient Lipschitz conditions as follows:
sup VVo((, az 0, - Voū(s,a;0', 0000) Lool10 0'112-
8.0,0
sup | (V, Voū(s, a; 0' 0)1| < Looll0 8 8'112, 6,0,0
sup Vapp\Ve((s,,;0,00 - Veū(s, 8 ||2 < Lool|u(s) 8' d'(ss)1 12, Vs, a.0
where Leg, Leo, Lag are positive constants.
Assumption C.I has been made in previous research adversarial robustness (inha et al. 2018 Wang et al. 2019). A line of studies on deep neural networks helps to justify this assumption Allen-Zhu et al. 2019
Du al. 2019 Zou al. 2020
23

---

Published as a conference paper at ICLR 2026
Assumption C.2 (Bounded Sampled Policy Gradient). The gradient of the function ö(s, 0, 0) with respect to 0 is uniformly bounded, i.e., 3 M, > 0, such that
sup Voo((s, a; 0) - Mç. $,0,0.0
Assumption C.2 can be verified by bounded rewards and the lower-bounded probability of the sam- pled action, which are mild conditions in practical algorithms.
Assumption C.3 (Locally Strongly Convex Adversary). For any state and action o, the function 0ss, a, 0, 8) is locally p-strongly convex in Bz(s) {vo(s) = s + d. ||0,11 co I c), i.e., for any , 82, we have that
1ー (s)) - (2)))
0(s, a; 01) 2
+ (V Vo((s a; 0, 02). (6) - 02(s))
2
Assumption CD3 has been studied in Sinha al. (018): Lee & Raginsky 2018 which can be verified by the relation between robust optimization and distributional robust optimization.
Assumption C,4 (Lipschitz Conditions of State-Action Visitation Distribution). The state- action visitation distribution under policy Tg o Vo de.o(s, a) = dn (s, a)
E )a -F--(- 1 1) )ooemaoevo [(1 -7) Ei=o 7'I(s= = 8, at = =a] satisfies the Lipschitz
conditions under the total variation distance as follows:
sup Iddeoo 00 - dg do/ooTTV S Laell0 0'112:
sup ||do, - do 1|TV S Taolli 0' 2/112-
Assumption C.4 has been derived by Pirotta et al. (2015) based on Lipschitz environments and pol- icy. Certain natural environments show smooth reward functions and transition dynamics, especially in continuous control tasks where the transition dynamics come from some physical laws (Bukharin
al. 2024 Li et al. 2025), such as MuJoCo environments, where we conduct numerical experi- ments. These environments are thus Lipschitz. The policy parameterized by neural networks can be seen as Lipschitz based on neural network analysis (Allen-Zhu al., 2019: Du et al. 2019 Zou
et al. 2020).
Assumption C.5 (Bounded Variance). The variance of the stochastic gradient is bounded fol-
lows:
E
V VT VoVaneu (mo) (Ho)
c²
Assumption C.5 is a common assumption in the analysis of stochastic optimization.
The proof framework of Theorem C.2 drawn on Wang et al. (2019): Pirotta al. (2015). We begin
by proving the following four technical lemmas.
Lemma C.1 (Lipschitz of Strongest Adversary). Suppose assumptions C.1 and C.3 hold. Then
have that the strongest adversary 0" (s;s 01) is Lee Lee -moothh i.e., given state for any 0₁ and 02 we have that
P
Leo
(s; - 8" (s; 02)112 <
100 02112-
/
Proof. For any state s and action a, under assumption C.3 we have that
0(8 az Oz 8¹ (0,)) 0( Oz 2" (0z) + (V,o(s, (00)) (s;
~ 10) - 02)11
+
2 0s, 02 1' (02)) 1 100 (s: 07
where the last inequality comes from (V a; O2 8" (0z)), 0 (1; (s;00))) 2 0. Similarly, have
ö(s, a; 02 0" (02))
> 0(s, O2 0" ('()))) (Voú(s, 02, 0 (01)), 0 - (s;
+ 1 10 0x;
2
24

---

Published as a conference paper at ICLR 2026
Combining the above two inequalities, we have
p pllo (s;
V Voū(s, 02 (01)), 0 (s; -s;;
Vo0(s,,;;0, 0 (0:)) Voš(s, 0" (61)), 0- - (s;
< |V 0" (01) V.e(s, 0r (01))ll (s; - 0" (s;02)112
$ Laell02 0 ll2 ||0" (s: 01) (3 02)12.
The second inequality comes from v-V a: 8,, 0" (01)). 0' 01) - (s; 02)) 2 O. The third inequality comes from the Sauchy-Schwardd inequality, and the last inequality comes from Assump- tion Therefore, we have that
Leo 1 01 1|2.
(1;0 - 8" (s;s; -
This proof is concluded.
Lemma C.2 (I (Lipschitz of Sampled Policy Gradient against Strongest Adversary). Suppose assump- tions C.I and C.3 hold. Then we have that the sampling estimation ö(s, a; 0, (0)) Le- i.e., for any 01 and O2, we have that
[Vo 01 (01)) Voδ(s, 02 0" (02))l2 - Lo 021l2
Lonkon where Le
Lee.
Proof. For any 0, and 0z. we have that
VV 01, 0r (01)) Vaū(s, 02, 0" (02))
< Vob(s, a; 01, (01)) - Voû(s, a; 0, 0" (02))]l
I[Vn(( 01, 0" (02)) Vod(s, a; 02 0" (02)) l
S Low |10* (1; (s; 02)1/2 + Loo 10 0211
Loo Lgg 100 02112-
The first inequality comes from the triangle inequality, the second inequality comes from Assump- tion C.I and the last inequality comes from Lemma C.I Therefore, the proof concluded.
Lemma C.3 (Smoothness of Adversarial Value Function against Strongest Adversary). Suppose assumptions C.I C.2 C.3 and C.4 hold. Then we have that Vnoow" (#s) (Ho) L-smooth, i.e., for any 0i and Oz, we have that
W ov (xoz) ( (mO) - Voggou" (Foo)) (VoV (Tea)))))))) 01 02) S 10100 02113.
VoV on (moz) (NO) VorTog (Fo2) (wo) 122 S L| - 02112,
where L Lookee + Loo + Ma (Laabee + Lao)
Proof. For any 0n and 0z we have
V Voraoon ou (see, ) (MO) VoVro (Foz)) (Ho)ll2
E (F (0)) Vo0(s,0;01, 0² (0:1)) (e (02) (o0 a; 02 0r (02)))
((s,a) (0(0,) Voū(s,a;01 0" (0:)) - Voi(s,a;01 0 (02)))
(1e0) Voū(s, 02 0- (B2))
25

---

Published as a conference paper at ICLR 2026
This inequality comes from the triangle inequality. For the first item, we have
Bleaa)dda [Vo 01, (01)) Voë(s, a: 02 0² (02))] S B(a,a)rdo, (00)) Vo 01 0r (01)) Voù(s, a; O2 0r (02) [12]
[ogo + Loo ||0: 02l12,
where the last inequality comes from Lemma
C.2 For the second item, we have that
((ea)) (0,) 20 (m)) Vob(s, az 02, 0 (02))
do, (esaa -o, v0
Voū(s, 02 0 (02))
+ B (100))
o 00 (82))
1 ||dg 0 do (22)) sup
V 0 02 (02))
5,
+ 00)0) - do (0,) (02) TV
sup
Vot(s, a; 02 (02))
8.a
S Lao Mo (01) - 0 (822 112 + Ma M - 02l12
Loo 1101 - 02112 + Lao ( 01 82ll2
S Ms
P
Ma (LaoLvo + Lao |10, 02112-
The penultimate equation comes from assumptions C.2 and C.4 Lemma C.T Thus, we have that
The last inequality comes from
V ou (MT))
VoV (102) (Ho)))
Loo Lee Me
LaooLo + Lao )) |101 02112-
Therefore, the proof of this lemma is concluded.
Lemma C.4 (Bounded Error of Approximate Stochastic Gradient). Suppose assumptions C. and C.3 hold. Then we can control the error of the approximate stochastic gradient in the ARPO with
d-upproximation adversary in definition C.I (Algorithm]] T]. Specifically, we have that
8
VoVnoors - Vo VoVngow (r (po))))) s Loo V k
Proof. We have that
VoVnoo (mo) (oo
(Vo6(s,a;0,0) - 0,
(a.0)ET
(o,ū(s, Vol(s, 0; 0- (0))l
(R.O)T
Loo [|((s) 0" (s;
(s.a)ET
The first inequality comes from the triangle inequality, and the last inequality comes from Assump- tion C.I Under Assumption we have that
o(s a: 0, 0) 2 6(s,a;0, 0 (8) + (V V.. (0)) 0(s) (s; 0)) + # [1((s) - 0* (s: 0)
2
6(8, a; 0, 8 (0)) 2 0(.. (, 0(s) + 11(0)) 0² (s: O))
26

---

Published as a conference paper at ICLR 2026
Combining the above two inequalities, we have that
p ml]((s) - 0 0)1 < (Vnū(s,a;0,)) a,, 0¹ (s;
< 8 - (Vob(s,a;0, 86.
os::))
The second inequality comes from that ve a d-approximate maximizer of i(s, a; 0, f), i...
(V, a; 0, 0), ((s) (s; 0)) S 6.
The last inequality comes from the optimality of 0", i.e.
(Vaü(s, 0z 0, 0² (0)), (s) - ''' (s) 2 0.
Therefore, we have that
8 1
V Viooo - V VoV S Loo
This completes the proof of this lemma.
C.2.3 MAIN CONVERGENCE RESULTS
Based on these assumptions, we derive the main convergence property of ARPO. Theorem C.2 (Convergence of ARPO).
Denote
max ming VT (pO)
ming V (Ho). Under assumptions C.I
C.1.2.
C.3
C.4 and
C.5
set the step size 7) V =2
Then, for the ARPO with 5-approximation adversary in Definition we have
C.1
(Algorithm and K 2 g²
AL
K-1 ΣE
VoVe ou (Fo (nD))
AL K
2T220
S
40
+
where L Log Las + Loo + Ma LLaaboa + Lao)
Proof Theorem For convenience, we denote the objective function V(0) = V= (TT) (po),
the accuracy stochastic gradient G(Ø) = V V (To) (po). and the approximation stochastic gradient C(0) = Vooro (vo). According to the gradient Lipschitz of V shown in
Lemma C.3 we have that
V(0++1)
L |0k+1 0x112
2 V(Ok) + (VoV - 0x)
2
V(Ok)) + TH (VVV (O...)))((() Lnf
[IG(0w)11
((x)) + V (0x)11 - Lnn
|G(0W)l12 + (VVV voV(()))
2
= =V(0k) + TM (1 - 2 [[
Litk
Ton
+16 (1 - Lmne (VOV((L), G(PA) - VoV(BA) 2
[|G(BL) (oV(8))
=v(O + Th (1- 11- Link [VVoV + Th (1- LMT) (VoV(Ok), G(O.))
+ Th (1- - LmIk) (VoV G(OA) - voV(O+))
2 C(00))
V VoV(0.)
27

---

Published as a conference paper at ICLR 2026
By Young's inequality, we have
- G(Ow)) 5 =
(IVov + -G(0)) G(W)1B)
[G((0.) G(O.) +(Ox) 33 G(O,) (1000//000((000---00033300000-00
Therefore, we have that
V(0++1)
2 V(Ok) + 176 (1- Litk)
2 (0:)l1
Tl (1 - LMT)) (1VAV(00)]
+ [[G(0)) G(OW)
2
+ 1]k (1 - LMn) (O.V,
VoV().)
LIN (IIG(0L) - G(O+) - [[((0)) - VoV
= -V(O.) + Th
V (0w))l - The
2 (1 Lmx) (((s)) - G(0x))1
2
+ m7 (1- - Lmm) (Vo (Ok). G(O+) - Ve (0x)) - LG(,) - V (0x))
Taking the conditional expectation on both sides of the above inequality, we have that
E ((2))) V 10x+11) 0s]
s Th [1VvV |/ 8x] + ThK
E [|G(BL) - G(OW) 0x) + m (1 - Lmn B (V)V VoV(O)) G(Ox)) 6k]
(1+ (1+)
+ L (0.) (o,())) 0x]
= 1Kk E [ 0x] + 7ee
(1 +1+ LMRR E [IG((k) - G(Ow)l 0x]
2
2
+ L [1+(())) (Ov())) 0s]
12900
1/k E [1V (0x) 1| 0x] 116 (1 LTn)
P
+ L (0w) V.V (0x) 0s]
L200
1h E [IV (0x) 0.] (1 LTpa)
Lnño0
The first equation comes from the unbiased estimation G(O), i... B[G(0)] = (oV inequality comes from Lemma C.4, and the last inequality comes from Assumption C.3 telescope sum of the above inequality, we have that
The second
Taking the
K-1 Tk
1200
K-1
7 VVVV <E(((K)) v(O))
Σ
(1
LTK)
Σ
2
we have that Line 1. Furthermore, we have that T.. Because 77 V TLK and K >
AL
K [ S 40
AL
2
+
K
4
K=
Therefore, the proof of this theorem concluded.
C.3 LEARNING DYNAMICS NEAR FIRST-ORDER STATIONARY POLICIES
We explore the properties of FOSPs in ARPO and SPO by analyzing the learning dynamics in their neighborhoods. Our framework of analysis is drawn on the linear stability analysis of stochastic gradient descent (Wu et al. 20226) and sharpness-aware minimization (Zhou et al. 2025).
28

---

Published as a conference paper at ICLR 2026
C.3.1 LEARNING DYNAMICS NEAR FOSPs IN ARPO
We first consider the learning dynamies near FOSPs in ARPO. Let (6", ,0',," (8*) be a first-order stationary
poliey-adversary of V (oo). To simplify analysis, we consider (0", +))) satisfies the following second-order optimality conditions:
VoV = 0, V VoV = 0, Voo you (16) x 0, V V (AD) > 0.
(4)
Note that it makes no sense on the gradient when the objective adds a constant. Therefore, to explore the local properties around (0", ),"), denote L(0, 9) == Vxg- (μo) V= (po). In the neighborhoods of (0", )²), we considering the local quadratic approximation of [(0, () as:
[(0, 0) - (0- 0²) Holo", ,0) - 0*) + 2
(0 07) T "Ho(O,, ,0) 8*),
(5)
Co(o,0)
Co(8.0)
where Ho(0", 0²) == V [(8", 0²) and Ha( 0*) == V ( -0)) are the Hessians with respect
and 1, respectively. Then, we have the following optimality conditions from (4.
VeL(0", 0*) = o, VoC(0", 0*) = 0, Vooc(0' 0*) > 0, 22 ~ー) - O.
(6)
Let & denote the stochastic variable of sampling and let the sampled estimation f(,;0,0)
=
u(s a;8, 0, d) defined in Denote the sampled policy gradient estimation
1 B
0), voLę(0. vof(s; VoLg(0,0) B
L Vof(s(0,0)
RE
Consider the following single-loop adversarially robust policy optimization iteration:
Uk+1 0x + eV eV 0x),
(Single-Loop ARPO Iterator)
0k+1 = 0g 7/V Ve 0x+1).
Policy Gradient Noise. Denote the 1-batch gradient noise (0. Vef(s; 0) VaL(O, 8) and (0, 0) = aaf 0) VaL(O, 0). Let Eo 0) = E (E, (E)..(0,))e..d (0, 0) and E0, 0) = E (0, (0, T] as the 1-batch gradient noise covariance. Then, have that
Eo = E Vof(s (ol - V.C V= (0) E. [Vi(ss; Vo. ] [=L(0,
(7)
(8)
Furthermore, denote the sampled full-batch gradient noise as (0, = VeLg(0, 8) VoL(0, 9) = Evee (0, () and < (0, () : VoLz(0,0) Vol(0, = 1 E1Ee Ev. (0,0) Then, for the full-batch gradient noise, we have that
[ (0.00) (0.00)
3 (0, (0, 1 Eo((,
(9)
B
E [ (0, 0) T)
- [E,, 0) 0 = (0.
(10)
Based on these notations, we make coercive assumption about gradient noise based on Hes- sians. Similar insights have been theoretically and empirically validated in a variety of studies about
stochastic gradient descent in supervised learning settings (7hu al.) 2019 Feng Tu 2021 Wu
et al. 2020 20226 Mori et al. 2022 Wang & Wu 2023 Wojtowwtsch 2024 Zhou et al. 2025
Assumption C.6 (Coercive Policy Gradient Noise). There exists > O, such that for all 0 and 0 the following coercive conditions hold:
(6, 9) Tr
Eo(O. V) Ho(O. 2-2kol| I1.
[(0 0) (g 0)) 2 2ng Ho,(, 0) Th
[C0,, 0)
29

---

Published as a conference paper at ICLR 2026
Local Stability of ARPO. Similar to the linear stability of stochastic gradient descent (Wu al. 2022b) and sharpness- aware minimization (Zhou et al. 2025). we define the local stability of ARPO. Note that in the practical optimization process, only local stable optima can be selected.
Definition C.2 (Local Stability). The first-order stationary policy-adversary (0* 00) is said to be locally stable if there exists a constant C > such that for the Single- Loop ARPO Iterator] the local quadratic model 55) [(0, () satisfies R. [[ 0x)] S CR CBICC(0 fo)l, Vk 2 0.
Based these, characterize the Hessian properties of FOSPs in ARPO.
Theorem C.3 (Flatness Bound of FOSPs in ARPO). Let , be a firstorder stationary policy- adversary in ARPO and satisfy the second-order optimality condition (6) Denote Amin(-) and Amax(( as the minimum and maximum eigenvalues, and let d be the state space dimension. (0",0) is locally stable, L(0g, Bo) > 0, 2 2/Xmin (Ho(0", 10000"0"0 (), S 2/ x ())))
B 2 de' de Keg ( Ho ))²
and Assumption C.6 holds, then for single- ARPO, we have
that
(1H6(0% 0'
Kon? )
B
2xoll 2nollHo 02)1 . (1-22o11HHa
B
(11)
B
KGT?
where B is the batch size, is the coercive coefficient, e is the attack budget, and rj is the step size. Remark C.2. L(0,, (0) > O means that V* owe (po) > Viag aoen (Ho), which holds in practical training from the scratch.
Proof. Because L(0n, On) > O, by recursion, prove that L(0x-1, 02) ) 0 and L(Ok; 0n) 0, wk 2 1. Then, we have that
C(0k+1. DR+1)
1
2 (0x+1 0-) Ho (0.(000))(0m0 000((0x+1 - 0²) - (0k+1 10x+1-0") T 1100000")000 -0"
11
Without loss of generality, we can let 0 0- Denote H1 Hi (0" 9r) ) H2 = Ho 0*).
According to the definition of the sampled full-batch gradient noise, we have that E (0, 0) VoLe(0, d) Vo f), and (8, 0) (, 0) V ((,
Then, on one hand, we have that
E /
= E [0s v/V VoLo 0 0k+1)) (1 HH, - n (o 2 0k-1))]
E [0s- VoL(Oe 0x+1) T (0x, 0x+1)) 111 (0x 17 Ox+1) 1 (0x, DR+1))]
E [0 VoL(ke 0k+1)) (b VoVo(( 0x+1))]
+ a [5. (0x, 2z+1) H.EH (0k, Ok+1)]
11²
= E [[0. VLL(n. Ok+1)) H1 (02 v/VoL(O 0x 1))+
E
(r (Eo(Os, 0z+1) Hi)]
B
2nj²
KO H 11 [L(Ok, 0x+1)]
2 E [00 VoL(O, 12+1)) TH (0, VoVo(( 0x+1))]
B
2n KO H1 1122 [[(0s, Bx+1)]
E 10% (T TH 1/H1
277² Kel |1
2 E 10, H1 +
E [C( 0k+1)]
B
2E (Ok, 0x.) 2n7250
H1
11?
E [[(k,, 0x+1)]. B
The third equation comes from E [[ (0, 0] inequality comes from Assumption C.6
0. The fourth equation comes from 9 The first 2
Because of 2 Am,j(H,)" have that H1 2 21. This
30

---

Published as a conference paper at ICLR 2026
comes that H1 2nH 2 H = H1, which leads to the last inequality. Now, we need to estimate the lower bound [Co(0k. 1)x+1]].
E [ 0k+1)l
E 10% H10g +0/+1 H20x+1
= E [o 0x)l + II.
On the other hand, we have that
E [
E (10x + +CV CVVo 02)) H2(0, + +VVo 0n))
E [Ox + 0k) + (0,,
(0x. 0x)) H2(0; + +V 1k) + < (0x, 0x))]
E [(0 + <VoL(() 0k)) H2(0; v. 0.))]
2E
[ (0k. 1&)) H2 (0x, 0k)]
2
E [00x + <VL 0k)) (2 + c..C( 0x))]
Tr [Eg (0x. H2)]
E0(0:
B
2e
Kg
H2
12 E [(2,, 0k)
2 E [(. + v 2(0n, 0k)) H2 + vo 0w))]
+
B
20 122
E (1 + (1 012
E [C(Ok, 0x)1
B
262 Kio 112211
2 E [0] H28x]
[C[Ok; 9x)]
B
2c K. |12111
= 2E (0x. 0x)l
[C(0,. 9x]]
B
The third equation comes from E
S
(0, 0)] 0. The fourth equation comes from 10 The first
inequality comes from Assumption comes that (1 222 + ell2) H2 20113 22 2 112, which leads to the last inequality. Therefore, we have that
C.6 Because of S A=ma we have that H2 - 21. This
E [[tk, 0 Ok+1)] 2 (1-
2 ekoll [(,,, 0x)
(12)
B
Because of B 2 d de2 (222)2, we have that
B
|| S d. (- -22)
2kg"
Exe Halli 0.
which indicates that 1
Furthermore, we have that
12200 11l
(
2nD H2ll
E [[C(0x+1, 0k+1)) 1
[[ 0x)]-
(13)
B
Hence, (0",0)) ) locally stable for stochastic ARPO, so the following inequality must hold:
(1+ 1
11
2xx (1-2ma) H2
1
(14)
B
B
In conclusion, we have that for local optimizer (0*,0*) of Viooos (1o), it holds that:
B.
(1-
2 2kool
B
K
(15)
B
K
This completes the proof.
n
31

---

Published as a conference paper at ICLR 2026
observe that for FOSP in ARPO, if the adversary-side curvature is bounded as V3 V [[o)]l I B (xx then the policy-side curvature ARPO matches that of SPO (as later shown in Theorem C.4 i.e., || V V ove- (Ho))l (/(mn²²) This implies that ARPO achieves sufficiently strong robustness, i.e., flat adversarial curvature, it can preserve gen- eralization comparable SPO. Moreover, greater robustness may further enhance generalization. However, if robustness is insufficient, generalization may be significantly degraded.
From Theorem
C.3.2 LEARNING DYNAMICS NEAR FOSPS SPO
Similarly, for an FOSP 0" in SPO, we establish the learning dynamics analysis. Let 0" be first- order stationary policy-adversary of yT (uo). To simplify analysis, we consider 0* satisfies the following second-order optimality conditions:
VaV=o (un) = O, V Va (HO) - 0.
(16)
Note that it makes no sense on the gradient when the objective adds constant. Therefore, to explore the local properties around 8", denote [(0) V (Ho) V (Ho). In the neighborhoods of 0",
we considering the local quadratic approximation of [(0) as:
[()) = 1 - 0*) Ho (0" 0*),
(17)
where H²)
(00*) is the Hessians with respect to 9 respectively. Then, we have the following optimality conditions from (16).
VoL(O") = O, C(0 > O.
(18)
For a sampling trajectory T = (so, ao, ro..... ~ Po denote the unbiased estimation of the action-value function QTh as
(T) = Σ (rse""r(s,,a)) av), V(s az) E
1'2
For simplicity of writing and reading of the latter, denote the sampled estimate as:
1 1-7 sg (s.) llog (als),
0(s, 0)
(19)
whereas sg(-) is the -ooddira operator, meaning that (s, a) is seen function according to (s, a) and is detached from the gradient operation for the following context.
Let & denote the stochastic variable of sampling and let the sampled estimation [(s; 0) =
(8, az ()) as defined in Denote the sampled policy gradient estimation as
1 VoLe(0)
Σ
Vof(s;))
B
Consider the following SPO iteration:
0k+1 - (
(SPO Iterator)
Policy Gradient Noise SPO. Denote the 1-batch gradient noise as (0) Vaf(s,;
VeL(O). Let (0) = E Co. (O) as the 1-batch gradient noise covariance. Then, we have that
Eo E Ves(s, ()))) VoL(O)V,C(0)]
(20)
Furthermore, denote the sampled full-batch gradient noise {)) - B (0))), Then, for the full-batch gradient noise, we have that
voC(O)
(0)C) (0) 1 (0)] -0
(21)
Based on these notations, we make a coercive assumption about gradient noise based on Hes- sians. Similar insights have been theoretically and empirically validated in a variety of studies about
stochastic gradient descent in supervised learning settings (Zhu et al. 2019 Feng & Tu 202T Wu
et al. 2020 2022b Mori al. 2022 Wang & Wul 2023
Wojtowytsch 2024 2024 Zhou al.
2025
32

---

Published as a conference paper at ICLR 2026
Assumption C.7 (Coercive Policy Gradient Noise in SPO). There exists K >> 0. such that for all 8, the following coercive conditions hold:
Σo(0) [C(0)] Ho(0)
Tr
2kg (0)
Local Stability of SPO. Similar to the linear stability of stochastic gradient descent (Wu et al. 2022b) and sharpness-aware minimization (Zhou et al. (025), we define the local stability of SPO. Note that in the practical optimization process, only local stable optima can be selected.
Definition C.3 (Local Stability of SPO). The first-order stationary policy 0* is said to be locally stable if there exists a constant C > 0 such that for the SPO Iterator the local quadratie model 77 [(6) satisfies [[[[0+)] S (E)) vk 2 O.
Based on these, characterize the Hessian properties of FOSPs in SPO.
Theorem C.4 (Flatness Bound of FOSPs in SPO). Let 4 be a first-order stationary policy in SPO that satisfies the second-order optimality condition. If 0* is locally stable, L((o) > 0, and Assump- tion C.7 holds, then for SPO, we have that
B KmA
Ho( )1E S
(22)
Proof. Because ((0)) > O, by recursion, we can prove that C(0x) 0, Vk: 1. Then, we have that
1 (0++1 - 00) 10x00)(0x++ 0*).
[(0x+1) = 2
Without loss of generality, of the sampled full- full-batch gradient noise, we have that </ (0) VoLę(0) VoL(O).
let 0* O. Denote H1 H0()"). According to the definition
Then, we have that
[0 k+1 [00)00mm
E (0x () H1 -Voo((()))
E
[O0 v)Vo(()) - -n5 (0x)) HI - v/Vo(())) - (EB((())) (0x))]
E
[Ox v)((())) H1 (Voo((e)) 7 [5/ (0x)) &
112
E [O) v)C(O)) H4 VoL()())
E
Tr (Eo (0()
B
E [( 17 (o)( H1 []Lk(())
+
2m
Kg
HH
11:
E [[(ba)
B
E [0% (1 nH 1 (1 n01 + 277² Ko
H1 11?
IE [C()w]]
B
2n No 11 [[((0))
B
The third equation comes from E inequality comes from Assumption which leads to the last inequality.
SA (0.0) 0. The fourth equation comes from 21
The first
C.7 Because of H1 ž= 0, we have that H, Hn m 2 0,
Furthermore, we have that
2 |
E [[((+++1) 2
E [C(+))
(23)
B
Hence, 0* is locally stable for stochastic SPO, so the following inequality must hold:
7"² H 112 B
1
(24)
In conclusion, we have that for a local optimizer 8* of a (po). it holds that:
B
||11 <
(25)
This completes the proof.
n
33

---

Published as a conference paper at ICLR 2026
D ANALYSIS ON INTUITIVE ISA-MDPS
In this section, we illustrate and analyze detailed two-state and two-action ISA-MDPs to provide intuitive explanations from the numerical properties and their value space geometry. These examples are drawn on Dadashi et al. (209)).
Lemma D.1 (Line Theorem DDdaahi et al. 2019)). Let be state and policy. Then there exist two s-deterministic policies in Syss denoted Te, Tas such that for all x' E eY5usp'
fu(Te) 3 fo(T') 3 fo(mu),
where fu(T)) V", and (s) describe the set of policies that agree with all states except
Furthermore, the image of fo restricted A is line segment, and the following three sets are equivalent:
(iff (Y3())))
(f) (amee + - (1- a [0, 1],
(a)(ofo(TT)) + (1- (1-a) a [].
Based on the above Lemma, we derive the following Proposition. Proposition D.1. There exists an ISA-MDP that satisfies the following properties:
(1) Its robust policy space has a cut point, i.e., there exists a policy a such that TRobust/ / {T) is disconnected, where RRobust { E II (s) V-oo" (x) (s), Vs).
(2) The FOSPs in ARPO are more than those in SPO. Moreover, let TT be FOSP under SPO, and AA be a FOSP under ARPO. Then, n A is robust policy with V (μO) - (wo) < (V* (Uo))
V (Ho)), where (μ²) = min, V (1o) is the worst value in ISA-MDP.
Proof. Consider two-state, two-action ISA-MDP M (S, A. P. Y, with S {si, $2).
A {a1, ) and reward and transition structure specified as
a();,az) = F((i 1)IAI + - 1]. P(sk|si. az) P(( A) + j- 1][}),
where
- 0..,). p= -[0.0000033] [0.99, 0.01], [0.2, 0.8) [0.99.0.01]]
Set the discount factor 7 0.9, and let B(si) B(s²) {si, $2). Parametrize policies as θ = ((a 3) 0 < < 1,0 8 < 1}. with (aip(ai|s)) = a, (a. 82) B
The robust policy parameter space is defined as
Robuutt {(a, 3) E E Jou VFa.3.
We derive that:
Aa,3
VFa.2
(26)
Pa.g"
Ba,9
( =
(27)
Pa,8
(8))
0.24885.3 - 0.21635
0.0261 A
(28)
da
Pa,g
Pa.
0.24885x + 0.0711
0.0711
(29)
03
Pa.8
Pa.."
T.. (52)
0.248853
0.18135
0.0261B,
(30)
da
Pa.a
Pa.a² 2
T.. (82)
0.24885a + 0.0711
0.0711
Ba.
(31)
03
Pa.B
P. Pa.9"
34

---

Published as a conference paper at ICLR 2026
where
Pa. | y
= (0.991 - 0.711 + 0.109) + (0.261a + (0.09) 0000-00000 0.891),
A 0.1305a + (0.35a + 0.1) - 0.991) 0.0045,
g = 0.1305a + (0.35a + 0.1)(0.7113 - 0.891) + 0.0545.
Direct calculation from 26 and 27 shows that the policy ø1, achieves the highest value, i.., T1,1 is globally optimal. Moreover, the tollowing ordering holds:
V (0.16, 1.89) = To ~ (-0.81, T
2 V x (-9.55, 0.33) = m p s (-2.47,-1.71) -1.11)
(32)
To further analyze value relations, define
((B,s) :- V (8) TT (8),
E 82)
Explicit calculation shows there exists a threshold 0.777 such that
[((3s) 0, if > B, f(3,s)= 0, if B, [(3,s) 0, if < B,
(33)
and this sign structure holds identically for both $1 and 82.
Applying the Line Theorem D.I for any a1 02 < 1 and for each fixed
VToi (s) < ( if >
VTog.0 (s) Vagg. (s), if B B,
(34)
1 V9l2. (s), if B <
again for both 81 and 82-
Now, for value improvement in the B direction, define
g(a, = V (s) V=..0 (s).
Explicit computation confirms that g(a,s) 2 0 for any 0 s S and both 81,82. Thus, for any S Bi < 82 S and fixed a. V (8) < (a))
(35)
Let us now determine the structure of the robust policy region. By definition, for any (a, E θ To 1² (Na,e) arg ell,, VT, where Ila,3 {Ta Fo,0, TB T8,9).
If 8, then 11,,a ) so (a, B) is always robust. If 3 < min 3}: Using 34 for 3 < gives V (s) < V and V (s) 2 V (s). Together with (35) and B < 3, V (s) S F (s). Therefore, V (s) is the minimal value in I, g for both states, robustness holds.
For other regions (a, (3 $ {a (B S min(a, }}}, by similarly using (34 and (35). one can verify that the minimal value in aa, is always attained at different policy (specifically, at 13 Ta.as or T3,9 the respective regions). Thus, the robust region is exactly the union
+ Robust ((a, B) = 3) (la, 3) min(a, ())-
Notably, the point (3, 3) serves as a cut point: removing it disconnects θ Robustt This completes the proof of property (1).
For property (2), we show that, unlike SPO, the ARPO objective can admit more FOSP. Indeed, we focus on
a B 2 0. 92 0,
max V Fa (8) a
subject to:
3 B 2 0, 1 2 0.
35

---

Published as a conference paper at ICLR 2026
We analyze the Karush-Kuih Kuhn-Tucker (KKT) conditions at (a. (0.0) to verify FOSP. The
Lagrangian is defined as:
[8,, o. B, VTa A1(a 9) + A2 Aa 33) + 24 o.).
The KKT conditions at (a, () (0,0) are:
Stationarity:
0V (s)
(0.0) + 21 d4 = 0,
do
0V (s)
(0.0) A1 + A2 - A3 0.
03
Primal feasibility: (0,0) satisfies all inequality constraints.
Dual feasibility: D 2 0, j = 1,2,3,4,
Complementary slackness:
(1 - 3) = 0, A2 0, As - 3) => = 0, A1 - a)) = 0.
Since at (a, = (0, 0). the gradients
(V) (0.0) and 8V 05 (0.0) both exist, and by the structure of the problem the corresponding dual variables can be chosen to satisfy the equations
Da
above, all KKT conditions are satisfied. Therefore, (0, a FOSP for ARPO, even though it not a FOSP for SPO.
Let iTS = TT,,, n = To,o, μf the uniform distribution on this example, x = #1,0- By 32
we know that
VTs (po) x 1.03, (po) = -0.65, V (HO) 2.09.
Then, we have that
ve (Ho) (po) = 3.12, (=A(H)) - (-0)) = 1.44 < (V (1O) ) v-(w))).
This completes the proof.
E ANALYSIS ON SURROGATE ADVERSARY
In this section, we provide the analysis and proof about the surrogate adversary.
Theorem E.1 (Surrogate Adversary). For any policy Te. state E S, and K let the adversary & be computed via Ktstep gradient descent on the adversarial value function, i.e., Oa.k = daN-11 -s 1 V Vasens (s)la,=0,,p)) --1. <- S K. Assume step sizes πs S 1/(x, 5) for some 8 > 0,
and if G( TT v) (TI( (+))) ( < 1. Then, have
(*) Voooee (s) Amax (F' ( ((Ss,T,,1 O ((()))
28
where Fs, is the Fisher information matrix of log no us (als) with respect to 02 0, defined as Fag = F(0,8) = Eawsrol-ss) (Vo, log (a)s + 0 0=)1 (Va, log To(als + 0=)lo,o)) As maX,+0,EB(s) Amax (V Viaova (s)le=o,) and Xumaxe (.) is the largest eigenvalue of matrix.
Proof. We begin by examining the local behavior of the KL-divergence between the policy at the original and perturbed state:
G(s: no; va) DKL (=0 0.)).
Let us expand G(s; ne, vg) for small 0,:
Σ ne (a log To(als) no (a(as + 0.)
G(s;
(-)o [log (ogg)(als)
log to 0.)]
36

---

Published as a conference paper at ICLR 2026
Now, expand log (a + 1),) at v, 0 using Taylor expansion:
(a|s) ()]ss)+ + Vo log mogma(al ( (a) o v log + log ma (a(aa+++0)) = log ne (
Plugging this into the difference, log (o(a)s) log To(als + U.), and taking the expectation over ~ (-)s:
Ea (-1s) V, log (o(a - D v3, log (ogTo(ls))o.=00) + +0110.113).
The first term equals zero because
(- [Vo, log To(a)))la, V., Σ To(a|s 0.) 0,
since the sum of probabilities is always 1.
The second term can also be separated using Fisher information matrix properties:
Fs.0 == Eawno (-1s) Va, log vv(als)Vo, a, log [
and
E Ban (18) V3 log [[(as))]] = F...--
Therefore, combining the above,
G(s, uo) = 00 Fs,0 0(119.113.
-
From the expansion above, since Fe is positive definite (or at least positive semi-definite) we have:
1
Amin (Fa 1 - - F,,0D, S 5 >max (Fe.e) 110.112.
Neglecting the lower bound (since we only need an upper bound for [1))1 isolate |00
G(s: Te, vo) + v0) 101.3 Ama
Here, the ( term comes from inverting the Taylor expansion when 0, small. We now analyze the decrease in the value function resulting from K steps of gradient descent:
For each step.
P.,k = Us k/a 1 Vo, V (8) (,,
Let V&2 = VToonon (s). Expanding V& around 0 da using Taylor's 's theorem:
V/= = V + Vo, Vmoooo (s) [s) (Da.k 12,k-1)
+ 2 (00. - De,b-1) T V V (s)1o.=0..-- (D..k - 1e.k-1) + O(11s,. -02.-1113).
Summing over &o = to and noting telescoping sum of Vx-1 - Vn:
Vee (s) - v (s)
g0o0o n- (s) xoOKs (s))
0
11.t 01.
0, ViOONS (s)1o,=0.e_ 0ck-1)
K
Σ
a.1k 1a.k-1) T (=ooo (80000())10.20,,1111 2a,k-1)
2
+0
| 0a.
37

---

Published as a conference paper at ICLR 2026
The gradient update yields:
d. = = Vo, V (5)10,=0,x-
For small enough steps, and since the step size k-a,-1 I 1/(A, + 6) ensures sufficient decrease (similar the classic descent lemma in optimization), we can bound the value difference as:
v (s) Vaocnon 1 (s) a0, - +
Σ
110 a 02,K- 112
By Jensen's 's inequality,
K
K
2
Σ
- - 0a.
Σ
D - 12,k-1)
K
= K
Now substitute the earlier quadratic relationship between |0.x|l' !1? and G(s; no. vo ):
6 |1.. k O(10,, ( VTOOVO,,K (s) 2
25
Amax (Fs,0) G(8; Vo..x) + O(G(s; mo, vo,,x) ) 3//2).
This completes the proof.
F ADDITIONAL ALGORITHM DETAILS
F.1 ADVERSARIALLY ROBUST PROXIMAL POLICY OPTIMIZATION (AR-PPO)
We apply the ARPO paradigm on top of the PPO algorithm and further incorporate the SPO-based guidance, yielding Adversarily Robust Proximal Policy Optimization (AR-PPO). The overall algo- rithm is presented Algorithm
F.2 BILEVEL ADVERSARIALLY ROBUST PROXIMAL POLICY OPTIMIZATION (BAR-PPO)
We apply the BARPO paradigm on top of the PPO algorithm and further incorporate the SPO-based guidance, yielding the Bilevel Adversarially Robust Proximal Policy Optimization (BAR-PPO). The overall algorithm is presented in Algorithm! g
F.3 ADDITIONAL IMPLEMENTATION DETAILS
We run 2048 simulation steps per iteration and use a simple MLP network for all agents. SA-PPO, WocaR- and BAR-PPO PPO are trained for 2 million steps (976 iterations) Hopper, Walker2d, and Halfcheetah, and 10 million steps (4882 iterations) on Ant for convergence. RADIAL-PPO is trained for 4 million steps (2000 iterations) on Hopper, Walker2d, and Halfcheetah following the official implementation and 10 million steps (4882 iterations) on Ant. The attack budget e is lin- early increased from 0 to the target value during the first 732 iterations on Hopper, Walker2d, and Halfcheetah, and 3662 iterations on Ant, before continuing with the target value for the remaining iterations. This scheduler is aligned with Zhang al. (2020c): Liang al. (2022) without extra tun- ing. The regularization weight is chosen from {0.I. 0.3, 1.0) for BAR- PPO. We run 10 iterations with step size c/ 10 for both methods and set the temperature parameter 3 1 10 for SGLD.
We implement all PPO agents with the same fully connected (MLP) structure as Zhang et al. (220c);; Oikarinen al. (002). In the Ant environment, we choose the best regularization parameter k in (0.01,0.03 0.1, 0.3, 1.0} for SA-PPO, R RADIAL-PPO, and WocaR-PPO to achieve better robust- ness. For fair and comparable agent selection, we conduct multiple experiments for each setup, repeating each 17 times to account for the inherent performance variability in RL. After training, we attack all agents using random, critic, MAD, and RS attacks. Then, we select the median agent by considering the natural and robust returns using the SA-RL and PA-AD attacks to further robustness evaluation, because these attacks involve quite high computational costs.
final agent. This chosen agent is then attacked
38

---

Published as a conference paper at ICLR 2026
F.4 EXPERIMENTS COMPUTE RESOURCES
All MuloCo experiments were conducted on machine with an AMD EPYC 7742 CPU and no GPU acceleration.
BARPO's training takes approximately 2 hours for Hopper, Walker2d, and HalfCheetah, and 9 hours for the more complex Ant environment. This represents roughly a 2x increase in wall-clock time compared to standard, non-robust PPO training. Crucially, this overhead is on par with other leading robust baselines (Zhang et al., 2020c; Oikarinen et al. 2021 Liang et al. 2022). ) which require simi- lar inner-loop computations to find the adversary. The choice of a CPU-only setup follows baselines, as the wall-clock time in these MuloCo tasks overwhelmingly dominated by the environment sim- ulation overhead, ensuring a fair comparison of the algorithmic sample complexity.
Furthermore, we confirm that BARPO introduces no significant additional memory overhead com- pared standard PPO training.
Algorithm Adversarially Robust Proximal Policy Optimization (AR-PPO).
Input: Number of iterations T, a schedule 6ę for the perturbation radius c, robustness weighting si
1: Initialize policy network Ta (als) and value network Vay (s).
2: for 1 to I do
3: Run πe, in the environment to collect a set of trajectories D = { containing [D] episodes, each тę trajectory containing [Te] samples, тą == {( 0k, Tk,s, 8k,++1)) E [[rx]]-
4: Compute reward-to-go Ry.; for each step é in every episode using the trajectories and dis- count factor 7.
5: Update value network Vo, (s) by regression on the mean-square error:
Ful (sw:)) Re.s)
Oy -- arg min Ov
Σ TRED
Σ TK
6: Estimate advantage As,' i for each step in every episode l using generalized advantage esti- mation (GAE) and the current value function Vov (s).
7: Solve the inner optimization using gradient descent (PGD):
8: For all k i, run PGD to solve (the objective can be solved in batch):
(8(
81, arg min
9 a, (on.a)
Ae.
To. (at. (8k,s)
where g(x, v) = min (a y, clip (x, 1 n,, + 17) y). Compute the approximation of the outer optimization:
9:
Lar(
Σ Σ TLDD 1=0
ITal
(-1
(
ne TT((n,1(s..) Ar.i ))
Σn (T
To, (ak.)
Update the policy network by minimizing the vanilla PPO objective and the AR objective (the minimization is solved using ADAM):
(a(an))s Aes) + k. (ar (ax,
4.t1 < arg min
Σ
Σ 9
Σg |Ty]
FLED
11: end for
39

---

Published as a conference paper at ICLR 2026
Algorithm 3 Bilevel Adversarially Robust Proximal Policy Optimization (BAR-PPO).
Input: Number of iterations T, schedule E; for the perturbation radius e, robustness weighting R.
Initialize policy network To (a(s) and value network Vo, (s).
2: for = 1 to T do
3: Run тe, the environment to collect a set of trajectories = {Ta} containing [D] episodes,
each ти is trajectory containing [Tx] samples, == {[SE,i, Tk,i, 88,5+1)), [[xx]]-
4: Compute reward-to-go Ry for each step in every episode using the trajectories and dis-
count factor 7,
5: Update value network Vol (s) by regression on the mean-square error:
Tue V (sw.i) Ra.s)
Dy arg min 0v
Σn Th
TEED
==
6: Estimate advantage Ag, for each step in every episode i using generalized advantage esti- mation (GAB) and the current value function Vi,
7: Solve the inner optimization using stochastic gradient Langevin dynamics (SGLD):
8: For all k i, run SGLD solve (the objective can be solved in batch);
$1, arg max KL s, S. (sh.e)
(-o((s))))
9:
Compute the approximation of the outer optimization:
Iaa
1 - H (TT(-18M.)) - To(an,(sss)) Asd))
Lbor (0) = Σx TRDD T=
To (ax
where g(r, y) min far y, clip (ær, 1,, 1 + 1)) y) 10: Update the policy network by minimizing the vanilla PPO objective and the BAR objective (the minimization is solved using ADAM):
In
To(aa.i(sk.)) An.e + n. Char
0.t1 & arg min
Σ Th
9
TT, (ak,)lsk.))
0
TRED
==
11: end for
G ADDITIONAL EXPERIMENT RESULTS
G.1 COMPARISON OF SPO, ARPO, BARPO WITHOUT GUIDANCE, AND BARPO
In Table 4 compare the natural and robust performance of SPO, ARPO, BARPO without guid- ance, and BARPO for continuous control tasks in MuJoCo. We also plot Figure 7 the version of Figure without the standard deviation. We can that BARPO, without guidance, consistently and significantly outperforms ARPO in natural and robust returns with respect to the mean performance. Moreover, BARPO further improves the overall performance by utilizing the SPO guidance.
G.2 STATISTICAL SIGNIFICANCE OF COMPARISON RESULTS
We discuss the statistical significance of comparison results between BARPO and baselines from two key considerations: (1) the comparison of mean performance, and (2) comprehensive analysis incorporating both the mean and the standard deviation.
Superiority in Mean Performance. From the perspective of mean returns. BARPO demonstrates a clear and significant advantage. To quantify this, we have calculated the average improvement of BARPO's mean returns each baseline across all four environments, both with and with-
40

---

Published as a conference paper at ICLR 2026
(a) Hopper
(b) Walker2d
(c) Halfcheetah
(d) Ant
Natural and robust performance of SPO, ARPO, BARPO without guidance, and BARPO for four continuous control tasks. BARPO (w/o g) consistently outperforms ARPO.
Figure 7:
Table 4: Natural and robust performance SPO. ARPO, BARPO without SPO-guidance, and BARPO for continuous control tasks in MuJoCo. Bold and underlined values indicate the top and second-best performances, respectively.
Environment
Paradigus
Natural Return
Random
Criti
MAD
Return under Attack
RS
SA-RI.
PA-AD
Wort
Werst-case
Robustness
SPU
3081
2925
2035 003
1763 1:510
750
79 0.2
823
79
-0.974
Hopper
ARPO
2101
2058
5155 176
2325 2:44
1001 1:00
1032
1799 4:47
1001
-0,524
3875
1195
1000
40,718
BARPO
1728
SPO
4507
.0.928
2alker2d
ARPO
2063 2.45
1815 1:
2360 :: N21
1944
1360-11 135
1270 922
1460 19
-0.394
0.00
BARPO (6/0
4287 11
4289
4251
10
4223 1: 36
4201 1:14
301 cm
3473 405
3473
.8.190
BARPO
4732 50
4718
4727
4750
4646
4285
2699
1192
2699
-0.436
SPO
5048 1N
4463 1:40
3281
918
1049
-213
-69
-213
-1.042
Halfchettah
ARPO
1365
1359
1402
1230
1079
1216
1079
-0.236
BARPO
3612
3654
5036
3365
3337
.0,078
1:50
3181
SPO
1.
4696 +: 1915
1768 1:15
914
1971
-3107
-1,577
Ant 0.10
ARPO
1709 204
2026 15
1916 131
1839 200
1661 155
1648
686
1675
233
1648
-0.034
BARPO (W/0
3647 à 132
3590
3802 3M
3597 à X1
3420 (7)
2760)
688
2649
541
2659 501
-0.220
BARPO
5024 117
4979
4777
4843
120
4171
3367
2825
757
2825
-0.438
out attacks. The results, summarized in Table 5 show that BARPO consistently and substantially outperforms the baselines average.
Table 5: The average improvement of BARPO's mean of returns over baselines.
Compared Method
Natural Return
Random
Critic
MAD
RS
SA-RL
PA-AD
Worst
SA
+4.17%
-2.19%
+8.20%
+28.29%
RADIAL
+29 -29.40%
+37.89%
+34.20%
+48.35%
+77.60%
+294.27%
+24.88%
+156.68%
WocaR
+7.21%
+92.61%
SPO
+2.56%
+6.94%
+82.99%%
+197.59%
+350.34'%
+1238.33%
+1708,3895
+1000.944%
ARPO
+128.23%
+149.80%
+168.34%
+167.73%
+80.299
Advantage Considering Standard Deviation. Beyond the mean, robust comparison must ac- count for performance variance. Our results show that BARPO's 's advantage holds even when con- sidering the standard deviation. (1) In the majority of cases, the upper performance bound of the baselines (i.e., mean + standard deviation) still lower than the mean performance BARPO alone. (2) More strikingly, in several scenarios, the upper bound of baseline's 's performance (mean stan- dard deviation) is lower than the lower performance bound of BARPO (mean standard deviation).
These points strongly indicate that the performance distributions are well-separated, and BARPO's superiority is not an artifact of sampling variance. This demonstrates a statistically significant edge over existing methods.
G.3 EXTENDED EXPERIMENTS ON HUMANOID TASK
Humanoid task in the MuJoCo environment is a 3D bipedal robot designed to simulate human and is widely considered the most difficult MuloCo locomotion task, featuring:
A 348-dimmnsi observation space (31.6x Hopper, 20.5x Walker2d/HHalfCheetah 3.3x Ant).
41

---

Published as a conference paper at ICLR 2026
A 17-dimensional action space ( (5.7x Hopper, 2.8x Walker2d/HailiCheetaah 2.1x Ant).
For this challenging environment, we implemented a comprehensive comparison of methods: SPO, ARPO, BARPO (w/o g). ARPO (w/ g). BARPO, and the robust bascline SA-PPO. Each method's training was repeated 17 times. We evaluated both the natural performance and the robust perfor- mance under four different attack types, and report the results of the cediann-errormmin agent. For the training of ARPO (w/ g), BARPO, and SA-PPO, the hyperparameter к. was selected via a grid search over the {0.1, 0.3, 1.0).
Our results, shown in Tables and demonstrate that BARPO continues to achieve a strong natural and robust performance in this significantly more complex environment. This further validates the effectiveness of our approach.
Table 6: Performance of SPO, ARPO, and BARPO without guidance in the Humanoid task.
Paradigm
Natural Return
Random
Critic
MAD
RS
SPO
$227 :: 963
5320 1 730
4232 :: 1545
1059 +: 563
910 1: 595
ARPO
553 1
1 5871 1066
BARPO (w/o g)
6177 6177 18
6204 :: 13
6180 + 18
Table 7: Performance of SA-PPO, ARPO with guidance, and BARPO in the Humanoid task.
Paradigm SA-PPO
Natural Return
Random
Critic
MAD
RS
6315
6313 16
6385 10
6321 = 15
5889 92
ARPO (w/ g) BARPO
6035 38
6041
5978 559
5940 + 93
5637 + 622
6615
6612 32
6698
6623 + 40
6031 + 827
G.4 ABLATIONS ON REGULARIZATION COFFFICIENT
The regularization coefficient plays key role in robust training. Following baselines (Zhang et al. 2020c Oikarinen et al., 2021: Liang et al.) (022), we performed a limited grid search for BARPO in {0.1, 0.3, 1.0]. We emphasize that adversarial RL typically requires moderate tuning, and method introduces no additional burdens than standard methods the field. We also add ablations to reflect the performance of BARPO across different i choices in Table g
Table Natural and worst-case robust performance of BARPO across various K.
Env
Natural Return
Worst Robust Return
1.0
3636
1083
Hopper
3684
1340
0.1
2151
959
1.0
4401
3254
Walker2d
4732
0.1
4672
1615
1.0
3843
3433
Halfcheetah
4837
0.1
5133
749
1.0
5024
2825
Ant
5557
1413
0.1
5596
439
G.5 ABLATIONS ON INNER STEPS AND STEP SIZES
We further compare the BARPO's performance with different inner steps (10, 15, 20) and vary-
ing step sizes (scaling from 1x to 3x the base ratio of e/steps) in Tables 9 and 10 Across these variations, BARPO consistently maintained its performance, indicating it not sensitive to specific parameter tuning.
42

---

Published as a conference paper at ICLR 2026
Table 9: Natural and worst-case robust performance of BARPO across various inner steps.
Env
Return
10-steps
15-steps
20-steps
Natural
3677
3742
3652
Hopper
Worst
1240
1049
1194
Natural
4401
4733
4519
Walker2d
Worst
4140
3536
3913
Natural
4623
4408
4455
Halfcheetah
Worst
3453
3845
3972
Natural
5024
4917
4852
Ant
Worst
4171
4333
4303
Table 10: Natural and worst-case robust performance of BARPO across various inner step sizes.
Env
Return
e/steps
2e/steps
3r/steps
Natural
3677
3573
3673
Hopper
Worst
1240
1242
1250
Natural
4401
4424
4219
Walker2d
Worst
4140
4284
3159
Natural
4623
4307
4549
Halfcheetah
Worst
3453
3754
3988
Natural
5024
4772
4853
Ant
Worst 4171
4124
4226
G.6 FURTHER ABLATIONS ON EFFECTS OF THE BILEVEL STRUCTURE AND SPO
As shown in "Effect of SPO Guidance" of Section 5.3 (Figures and 6). we perform ablations comparing BARPO with and without SPO guidance, showing that SPO improves natural return but reduces robustness. Additionally, introducing bilevel optimization on top SPO substantially improves robust return with only minor degradation in natural return.
detailed in the second part of this section. Table shows that
To further disentangle the effects of the bilevel structure and SPO, we now include additional com- parisons (ARPO, ARPO w/ g, BARPO w/o g) and report the relative improvements of ARPO w/ and BARPO w/o compared to ARPO in Table I1
Table 11: Relative improvements of ARPO w/ g and BARPO w/o g compared to ARPO
Env
Paradigm
Natural Return
Random
Critic
MAD
RS
SA-RL
PA-AD
Worst
ARPO [W g)
+76.0%
+1..1%
Hopper
BARPO (w/o g
+74.9%
+76.6%
+13.7%
+60.7%
+19.4%
+0.5%
-41.1%
+3.6%
Walker2d
ARPO (w g)
+100.8%
+133.1%
+81.5%
+117.4%
+130,8%
+180.7%
+70.9%
+96.5%
BARPO (wdo g
+104.6%
+136.3%
+79.3%
+117.2%
+208.9%
+208.1%
+137.9%
+173,5%
Halfcheetah
ARPO (W g)
+253.9%
+261.9%
+270.9%
+83.3%
+193.1%
+25.7%
-10.7%
+0.6%
BARPO (wdo g)
+156.3%
+165.0%
+168.8%
+157.9%
+195.6%
+211.9%
+174.4%
+209.3%
ARPO [W g)
+215.3%
+15.00
+141.3%
+120.79
+60.79
25.1%%
+4.3%
22 1%
Ant
BARPO g)
+113.4%
+77.2%
+92.3%
+95.6%
+106.4%
+68.0%
+58.7%
+61.3%
G.7 COMBINATION EFFECT OF BILEVEL FRAMEWORK AND KL SURROGATE
We have found that BARPO improves the optimization landscape. We further analyze whether the improvement primarily arises from the bilevel optimization structure itself from the KL surrogate objective. We clarify that the improved optimization landscape in BARPO arises from the combi- nation effect of both the bilevel optimization structure and the KI. surrogate objective. This means that neither component alone is sufficient to achieve the observed improvements.
First, a bilevel structure by itself does not guarantee a smooth or well-behaved landscape. As noted in Section 4.1 both SPO and ARPO can be interpreted as special cases of bilevel optimization. However, they suffer from undesirable first-order stationary policies. The empirical improvements of BARPO over SPO and ARPO thus highlight the importance of well-chosen surrogate objective. Our Theorem 4.1 provides theoretical support for using the KL divergence as a meaningful and tractable surrogate. In contrast, poorly chosen surrogates without such properties may result in
43

---

Published as a conference paper at ICLR 2026
worse behavior. In particular, the reformulation 36 offers promising direction for identifying optimal surrogates theoretically.
Second, the KL surrogate alone is insufficient without the bilevel structure. To disentangle the tributions of each component, we conducted am ablation study comparing BARPO to an augmented version of SPO that uses the KI. surrogate directly within minimax formulation. The results (see Table 12) show that BARPO consistently achieves a better trade-off between natural and robust re- turns. Specifically, in Hopper and HalfCheetah environments, BARPO outperforms the Minimax baseline on both metrics. In Walker2d and Ant, BARPO achieves slightly lower natural returns but significantly higher robust performance, suggesting a better cobustess-ooptmalle balance overall.
Table 12: Comparison of KI. surrogate with minimax and bilevel structure.
Fav
Stracture
Natural Return
Worst Robust Return
Hopper
Bilevel
3684
1340
Walker2d
Minimax
4875
997
Bilevel
4732
2669
Hallcheetah
Minimax
Bilevel
47501
3181
1441
Ant
Minimax
Bilevel
5367
2355
H ADDITIONAL DISCUSSIONS AND CLARIFICATIONS
H.1 DISCUSSION ON ASSUMPTIONS IN THEORETICAL ANALYSIS OF CONVERGENCE
We offer the following clarifications and discussions on the assumptions the theoretical analysis of convergence for ARPO.
Clarification for core contributions. Our main contribution is not the convergence proof itself, but the identification of a fundamental tension between optimality and robustness in policy gradient methods, and further developing principled solution to mitigate this tradcoff. Our theoretical analysis reveals the root cause of this tradeoff, the landscape distortion induced by the strongest adversaries, and motivates the design of bilevel framework to alleviate it. The convergence proof merely supports the conclusion that ARPO converges to FOSPs.
Justification of assumptions. We acknowledge that assumptions like Lipschitz smoothness and strong convexity may not strictly hold for deep neural networks. However, they are widely adopted in prior works on adversarial robustness (Sinha et al. 2018 Wang et al. 2019). Lipschitz continuity of sampled policy gradients has been explored and supported in various works (Allen-Zhu et al.)
2019 Du et al., 2019 Zou et al.) 2020) that study smoothness properties of deep neural networks. The assumption of locally strongly convex adversaries also common in robust optimization liter- ature (Sinha et al. 2018 Lee & Raginsky 2018) and can be justified through its connection with distributionally robust optimization.
Potential relaxations. These conditions can be further relaxed. Specifically, the Lipschitz gradi- ent assumption can be relaxed to (Lo, L1) -mmoothhssssss (Zhang al. 2020d a). generalized smooth-
ness (Li et al. 2023), or smooth adaptivity (Bauschke et al. 2017 Bolte et al.)
2018 Ding et al.
2025). Strong convexity
be weakened to weak strong convexity (Necoara et al. 2019). restricted
secant inequality (Agarwal et al.) 20012), PL condition (Karimi et al. 20066). KL condition (Bolte)
al. 2014), error bounds, and quadratic growth conditions (Drusvyatskiy & Lewis 2018
H.2
CLARIFICATION ON NOVELTY OF THE BILEVEL FRAMEWORK
We clarify that our core contributions extend significantly beyond the proposal of using a bilevel framework. We view the bilevel framework not the end goal, but significant tool to address fundamental challenges in adversarial robust policy optimization that we, to our knowledge, are the first identify and analyze. Our key contributions cohesively threefold:
44

---

Published as a conference paper at ICLR 2026
New Insight into Inherent Vulnerability. We reveal that even under theoretically aligned models, standard policy optimization is inherently vulnerable. We pinpoint the underlying cause: its ten- dency to converge fragile first-order stationary policies. This vulnerability is not due to model mismatch but a fundamental issue in the optimization process, which is a perspective overlooked in the prior literature.
Tension. We uncover and analyze a fun- damental tension between optimality and adversarial robustness in practical policy gradient methods. We further attribute this trade-off to an intrinsic reshaping effect induced by the strongest adversary on the optimization landscape, an insight not explicitly identified in prior works. This analysis explains why naively optimizing for robustness can degrade performance.
Revealing and Analyzing a New
Principled Solution to Mitigate the Trade-off. Motivated by the above findings, we propose the bilevel optimization framework. More importantly, we propose practical, theoretically grounded instantiation of this framework designed specifically to mitigate this trade-off and find policies that are both high-performing and robust.
While bilevel optimization as a concept is not new, its application analyze and solve this spe- cific, newly identified problem in policy-based RL is, to the best of our knowledge, entirely novel. We believe the main novelty lies in the complete narrative: identifying the problem, analyz- ing its root cause, and providing theoretically prrincppll solution.
To the best of our knowledge, we are the first to use bilevel framework to analyze the adversarial robustness of policy-based reinforcement learning.
H.3 QUANTIFY THE GAP BETWEEN BARPO AND ARPO
We quantify the connection between our KL-based bilevel formulation (BARPO) and the original minimax formulation (ARPO) from two perspectives.
Theoretical alignment. We emphasize that under ISA-MDP, BARPO and ARPO share the same global optima in theory. Theorem 4.1 further shows that BARPO's inner optimization is approxi- mately equivalent that of ARPO.
Optimization proximity. Following Kwon et al. 2023 Lu & Mei 2024). for sufficiently large a, BARPO can be reformulated:
min I. (0, v,, O) Egnpa | (s)
0.00
+ +aBsyja [KL (TT(s) TO o v (s)) KL (T (s))))
We quantify the deviation from the original objective using the bound:
L (s:0, V (s)l S V V (s) maans (s))
+ al KL (TT(s) I| o (s)) - KL (TT(s)]) o vo(s)) |.
The first term is bounded by CT oo no us) through Theorem 5 in Zhang al. 22020c The second term can be bounded by C Pinsker's and Pinsker's inequalities. Specifically, when second term is also O ( (v /L o | a ug) by Taylor expansion. Thus, the deviation between BARPO and ARPO can be controlled by the square root the maximum KL divergence between Tg 2 and na o Da.
To
TOOV
[T vo no Me) by reverse and Ty uy are not too far apart, the
These connections indicate that BARPO, while structurally different, retains strong theoretical connection to the ARPO.
H.4 FURTHER THEORETICAL UNDERSTANDING OF BARPO
We further delve into a deeper theoretical understanding how BARPO reshapes the optimization landscape from three complementary perspectives:
45

---

Published as a conference paper at ICLR 2026
Adversarial Value Improvement through Landscape Lifting, Let v denote the solution to BARPO's inner problem and v denote the strongest adversary defined in ARPO. By definition, y=))"(>) 2 ( indicating that BARPO tends to elevate low-value regions in ARPO's ' land- scape. Furthermore, Theorem 4.1 shows that maximizing the KL surrogate aligns with minimizing adversarial value: when the KL surrogate is high, the adversarial value is correspondingly low. This provides a theoretical link between BARPO's surrogate objective and ARPO's robustness.
Smoothing via Regularized Maximin Reformulation. Following prior work (Kwon et al., 2023 Lu & Mei 2024), when is sufficiently large, BARPO be reformulated equivalent max- imin problem:
[ViRov" max (0, 1/, 00 [ (s) (mo(s)) ( (s) KL (e) vo(s)))) 0,00
(36)
Compared to ARPO's objective, this formulation introduces regularization term that has smooth- ing effect on the outer landscape. Structurally, this resembles the use Bregman divergences in defining the Moreau envelope (Bauschke al. 2018), which is known to smooth the nonsmooth optimization problems.
Robustness under Date-driven Value Approximation. In practice, ARPO optimizes lower bound L(T, v) S V due to practical data-driven estimation for V i..., the practical objective becomes max= min, [[=, v), which is longer equivalent the ideal adversarial objective. In contrast, BARPO's inner optimization minimizes an upper bound on the adversarial value, while the outer loop maximizes lower bound. This formulation more faithful to the original robust motivation and offers better resilience under value function approximation, helping to improve per- formance practice.
We believe these perspectives offer promising foundations toward more complete theoretical un- derstanding of BARPÓ and plan to explore them in greater depth in future work.
LIMITATIONS AND FUTURE WORK
Despite the advancements introduced in this work, several challenges remain open for future re- search. First, we solve BARPO using straightforward approach that ignores the second-order term. This solution process can be further improved via an advanced optimization method on bilevel problem, such as transforming the bilevel optimization into penalty-based minimax formu- lation and then solving it via fully first-order gradient methods. Additionally, we derive a surrogate for the strongest adversaries under specific conditions, which approximates but does not coincide with the strongest adversary. Nonetheless, BARPO, based on this surrogate, achieves strong robust- even significantly outperforming ARPO, which relies the strongest adversary, in tasks like Walker2d and HalfCheetah. While these empirical results are somewhat aligned with theoretical consistency between optimality and robustness, there remains larger theoretical space further explore. Finally, our analysis and results can be naturally extended to more general settings where strict theoretical consistency may not hold, offering valuable insights for related fields.
USE OF LARGE LANGUAGE MODELS
The core method development in this research does not involve large language models (LLMs) as any important, original, or non-standard components. We utilized the LLM to assist only in the writing process of this manuscript. All content, ideas, and arguments were conceived and written by the human authors. The LLM was then used to provide suggestions for improving grammar, phrasing, and readability. The authors reviewed these suggestions and retained full editorial control, taking sole responsibility for the final version of the text.
46


Reviews:
[{"review_id": "iEdPQftcvW", "reviewer": "Reviewer_bepT", "summary": "This paper introduced the Bilevel ARPO (BARPO) that balances the optimality-robustness trade-off, by adjusting the adversary strength to promote traversable optimization paths which smoothed the optimization landscape.", "questions": "* The experiments mainly focused on continuous control tasks. How does BARPO generalizes to tasks with large discrete action space? Could the authors comment on potential challenges (e.g. combinatorial action spaces, function approximation issues)?\n* Is there a way to tune BARPO (e.g., via a schedule on adversary strength) to balance return vs robustness, for different application needs?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper was well written and easy to follow. Assumptions and motivations were clearly justified.\n* The insights on how the 'valleys' were formed and how to 'bridge' them were novel and interesting.\n* Theoretical analyses were sound.\n* In experiments BARPO was evaluated against extensive attack/robustness conditions.", "weaknesses": "* BARPO employs a KL-based surrogate for the inner minimization to approximate the strongest adversary. \n  * While the paper shows that minimizing this surrogate aligns with minimizing the true adversarial value, the reviewer wonders whether the authors have additional theoretical or empirical insights regarding the convergence rate under this surrogate formulation. In particular, are there quantifiable bounds or guarantees on the degree of robustness potentially lost due to the surrogate approximation?\n* From table 8 it showed that the performance could be fairly sensitive to $\\kappa$. \n  * Any findings around how it can be tuned would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ncPFMMhpto", "reviewer": "Reviewer_wh98", "summary": "This work investigates a conflict between theoretical and practical consistency between reinforcement learning (RL) policies' optimality and adversarial robustness. A prior work proposes the concept of ISA-MDP, where the optimal robust policy (ORP) is the same as the nominal optimal policy under the Bellman optimality. However, in practice, such consistency can rarely be achieved. This works analyzes why the theoretical alignment cannot be practically realized, trying to close the gap. In particular, the authors analyze the optimization dynamic of standard policy optimization (SPO) and adversarially robust policy optimization (ARPO). They first prove that both SPO and ARPO converges to stationary policies rather than global optima. Then, they observe that SPO is vulnerable and ARPO is robust yet yielding significantly lower nominal returns, showing a critical tradeoff between robustness and optimality. To address this challenge, the authors analyze the optimization landscape and value geometry induced by SPO and ARPO. Motivated by their findings, a bilevel optimization framework is proposed to achieve a good balance, achieving robustness without sacrifice of nominal returns.", "questions": "- What is $\\theta_0$ in Theorem 3.2?\n- I think in the original ISA-MDP paper[1], the authors' implemented algorithms (CAR-PPO-SGLD/PGD) do not show significant natural/nominal performance drop without attacks, sometimes even outperforming SPO methods (PPO in [1]). This observation is consistent with empirical robust RL works such as [2]. Why the performance of ARPO drops this badly in Table 1? \n- Could you explain what Figure 3 is trying the say?\n- Why using a lower bound in Theorem 4.1 as the surrogate? If the goal is to minimize the gap between the nominal and under-attack returns, then shouldn't we use an upper bound? \n- Where do Figure 4 comes from? Are the values in the figure true objective values (of SPO, ARPO and BARPO) for some RL problems, or are the values only for illustration purposes? \n- Could you please explain in detail the connection between different parts of Section 4.2? Specifically, why \"KL divergence serves as a valid first-order surrogate for minimizing the adversarial value\" can lead to better policy learning? And why \"This surrogate is both appropriate and reliable from an optimization perspective\"? Lastly, why the lower bound in Theorem 4.1 leads to the proposed bi-level optimization problem? The reviewer fails to see the connections. \n- While the proposed bi-level optimization problem is intuitive enough (restricting the perturbed policy to be close to the original policy), why this has the effect of changing land scape? In particular, how can it \"reshapes the landscape by elevating low-value but robust regions\"? I believe the authors state this claim without any justification.   \n\n\n[1] Towards Optimal Adversarial Robust Reinforcement Learning with Infinity Measurement Error\n[2] Robust Adversarial Reinforcement Learning", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- To the best of reviewer's knowledge, the convergence analysis for the max-min optimization of ARPO is new. \n- The geometry illustration (Figure 2) in Section 3.2 is informative and helpful.\n- The experiments are comprehensive.\n- The proposed bi-level optimization problem is intuitive (although the motivation and path reaching there are highly confusing).", "weaknesses": "- There is no interpretation or explanation for Proposition 3.1. $V^-$ is undefined. I am not sure why this result is useful. It only states that the gap between the nominal return and the return under attack can be much larger for ARPO than SPO polices. In fact, the whole subsection 3.1.2 feels disconnected from the rest of the work. \n- I expect the author to draw insights from their theoretical analysis in 3.1.1 but they only provide empirical evidence. In addition, the empirical evidence seems to be conflicting with existing works and thus cannot provide strong evidence for the claimed tension between optimality and robustness. \n- While Figure 2 of Section 3.2 is informative, the rest of it is confusing. In particular, $\\Pi_{Robust}$ in proposition 3.2 is undefined. Moreover, the claims of proposition 3.2 is vague. The authors needs to explain the vague statement \"ARPO tends to yield more isolated FOSPs than SPO\" while Proposition 3.2 only proves that there exist an ISA-MDP with a cut point. \n- Figure 3 is very confusing.\n- After Section 3, I don't see a rigorously presented tension between optimality and robustness, contradicting the claim in the introduction that \"We uncover a new form of optimality-robustness tension arising in policy optimization\". I would recommend the authors to consider enhancing the presentation there and make the components of this work more connected. \n- There is a same problem with Theorem 4.1. It appears disconnected from the proposed algorithm. The authors claim that \"we need to specify a suitable inner objective G that both promotes policy learning and maintains strong robustness\" and prove that KL divergence is an effective surrogate. But it is unclear why such surrogate leads to better policy learning and maintains robustness. \n- While Figure 4 is helpful, it is disconnected from the theorem.\n- Overall, the presentation of work needs considerable improvements, preventing the reviewer from evaluating the soundness of the proposed algorithm. While I see the technical contribution of this work, its lack in presentation coherence and smoothness severely undermines the quality of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "99Itc8Ibfr", "reviewer": "Reviewer_LLxi", "summary": "The paper is mathematically sound and shows that ARPO can converge to first-order stationary policies (FOSPs) rather than global optima. It also empirically explains FOSP for both SPO and ARPO. To address these issues, the authors introduce BARPO, a bi-level variant that reshapes the optimization landscape and yields improved robustness in practice.", "questions": "- Is the BARPO also robust to various attacks? Are the $\\varepsilon$ budgets predefined per task, and would changing $\\varepsilon$ alter the reported metrics and the ranking between methods?  \n\n- How sensitive are results to the number of inner steps, the step size, and the inner optimizer used to solve the adversary? \n\n- Table 3 shows BARPO with SPO guidance outperforming ARPO with guidance. Quantitatively, how does BARPO with guidance compare against the strongest baselines in Table 2 under the same attack settings? \n\n- Why use SPO guidance in BAR-PPO? What would happen if you instead added an ARPO-style guidance term?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The theoretical development is clear, and the paper is well written. \n\n+ The paper explains the FOSPs in SPO and ARPO and motivates a unified and clear way.\n\n+ On MuJoCo benchmarks, BAR-PPO shows strong natural and robust performance and adding SPO guidance improves clean returns.", "weaknesses": "--", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e3JaV9i5GO", "reviewer": "Reviewer_RuhM", "summary": "This paper investigates the longstanding tradeoff between optimality and adversarial robustness in deep reinforcement learning (DRL). The authors find that, in practice, standard policy optimization (SPO) and adversarially robust policy optimization (ARPO) still exhibit a tension: SPO yields high returns but is fragile to perturbations, while ARPO improves robustness but suffers large performance drops. To address this, the authors propose BARPO (Bilevel Adversarially Robust Policy Optimization), a bilevel framework that modulates adversary strength to preserve robustness while improving landscape smoothness and navigability. BARPO generalizes SPO and ARPO as limiting cases. Theoretical analyses show convergence to FOSPs and introduce a KL-based surrogate for the adversary. Experiments on MuJoCo continuous-control tasks demonstrate that BARPO outperforms ARPO and strong baselines in both natural and robust returns.", "questions": "- Could BARPO and all the analysis between optimality and robustness extend to other adversarial settings instead of limiting to state perturbation?\n- Although BARPO seems promising from the empirical results compared with all the baselines, those baselines may not be the state-of-the-art, where the latest one was published in 2022. Compare with more recent work may make this work more convincing. For example, [1] is also mentioned in the related work while no direct comparison in the experiments.\n\n#### [1]  Liang et al., \"Game-theoretic robust reinforcement learning handles temporally-coupled perturbations.\" ICLR 2024", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a central and underexplored issue with the trade-off between robustness and optimality, offering a novel and intuitive geometric explanation for the empirical gap between theory and practice.\n- Strong theoretical support and comprehensive empirical results.\n- Figures illustrating optimization landscapes (e.g., SPO vs. ARPO vs. BARPO) are intuitive and effectively convey the main ideas.", "weaknesses": "- The bilevel formulation and KL-based inner optimization introduce extra hyper-parameters  and computational cost. An explicit runtime and sensitivity analysis would strengthen the empirical case.\n- Robust RL includes several major paradigms (e.g., distributionally robust MDPs, risk-sensitive RL, adversarial policy training). Could the authors explicitly clarify how BARPO relates to these categories? In particular, how does it extend or differ conceptually and algorithmically from prior adversarial training methods such as [1] and its latest extensions like [2]? A concise comparative discussion (e.g., in the related work) would make the contribution and novelty more evident.\n\n#### [1] Pineto et al., \"Robust Adversarial Reinforcement Learning.\" ICML 2017\n#### [2] Dong et al., \"Variational Adversarial Training Towards Policies with Improved Robustness.\" AISTATS 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)