Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation
Authors: Kosuke Nakanishi
Akihiro Kubo
Yuji Yasui
Shin Ishii
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Deep Learning
Adversarial Learning
Robustness
Observation Robustness
Off-Policy
Actor-Critic
f-Divergence
Primary area: reinforcement_learning
TL;DR: We propose a novel approach that eliminates additional interactions with the environment in the observation robust alternative framework.

Abstract: Recently, robust reinforcement learning (RL) methods designed to handle adversarial input observations have received significant attention, 
motivated by RL's inherent vulnerabilities. 
While existing approaches have demonstrated reasonable success, 
addressing worst-case scenarios over long time horizons 
requires both minimizing the agent's cumulative rewards for adversaries and training agents to counteract them through alternating learning. 
However, this process introduces mutual dependencies between the agent and the adversary, 
making interactions with the environment inefficient and hindering the development of off-policy methods.
In this work, we propose a novel off-policy method that eliminates the need for additional environmental interactions by reformulating adversarial learning as a soft-constrained optimization problem. 
Our approach is theoretically supported by the symmetric property of policy evaluation between the agent and the adversary.
The implementation is available at https://github.com/nakanakakosuke/VALT_SAC.

Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation
Kosuke Nakanishi 12
Akihiro Kubo 344
Yuji Yasui? Shin Ishii 134
Abstract
promising framework for such applications, demonstrating performance that surpasses human capabilities by acquiring high-dimensional representational power through function approximation (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019; Wurman et al., 2022). However, the concerns are raised that only for the small input perturbations, such as adversarial attacks, significant performance degradation occur, not only in supervised learning tasks (Szegedy et al., 2014; Goodfellow et al., 2014; Kurakin et al., 2016; Paper- not et al., 2016), but also in decision making and control tasks (Huang et al., 2017: Lin et al., 2017).
Recently, robust reinforcement learning (RL) methods designed to handle adversarial input ob- servations have received significant attention, mo- tivated by RL's inherent vulnerabilities. While existing approaches have demonstrated reason- able success, addressing worst-case scenarios over long time horizons requires both minimiz- ing the agent's cumulative rewards for adversaries and training agents to counteract them through alternating learning. However, this process in- troduces mutual dependencies between the agent and the adversary, making interactions with the environment inefficient and hindering the devel- opment of off- offl-olliy methods. In this work, we propose novel off-policy method that elimi- nates the need for additional environmental in- teractions by reformulating adversarial learning as a soft-constrained optimization problem. Our approach is theoretically supported by the sym- metric property of policy evaluation between the agent and the adversary. The implementa- tion is available at https:/ /github. com/ nakanakakosuke /VATT SAC.
Robust decision- under uncertainty has long been studied by formulating worst-case scenarios to ensure stable performance in adverse conditions. Classical approaches in- clude game-theoretic models such as the two-player Marko- vian game (Shapley, 1953), and the Robust Markov Deci- sion Process (Robust-MDP) framework (Nilim & EI Ghaoui, 2005; Iyengar, 2005; Tamar et al., 2014), which seeks poli- cies that perform well under the most pessimistic transition dynamics. However, these methods are typically restricted to tabular or linear settings and do not scale well to com- plex environments (Tessler al., 2019). More recently, Pinto et al. (2017) proposed an adversarial RI. framework in which an agent is jointly trained with an adversary to approximate a max-min Nash equilibrium (Nash, 1951), yielding policies that are empirically robust, though lacking theoretical guarantees.
1. Introduction
In recent years, advancements in computational technology, coupled with the practical successes of deep neural networks (DNNs) (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016), have fueled expectations for auto- mated decision-making and control in increasingly complex environments (Kober et al., 2013; Levine et al., 2016; Ki- ran al., 2021). Deep reinforcement learning (DRL)
These early studies primarily addressed robustness by ex- plicitly modeling uncertainty in environment dynamics or introducing adversarial agents to simulate worst-case behav- iors. In contrast, recent work has highlighted the need to distinguish between different sources of uncertainty and to formulate robustness at the level of specific failure modes. In this work, we focus on observation uncertainty, which has emerged as critical yet underexplored challenge in DRL.
DDepartment of Information Science, Kyoto University, Ky- oto, Japan HHnda R&D Co., Ltd., Tokyo, Japan ATR Neural Information Processing Laboratories, Kyoto, Japan "International Research Center for Neurointelligence, The University of Tokyo, Tokyo, Japan. Correspondence to: Kosuke Nakanishi <ko- suke nakanishi @ hondad.>> Shin Ishii cishii@i.koooou.ac...
Recent research (Zhang et al., 2021; Sun et al., 2022; Liang et al., 2022) identifies two main types of vulnerabilities for observation robustness in DRL. The first arises from the function approximation properties of DNNs, often mani- festing as sensitivity to small perturbations due to lack of smoothness in the learned policy. The second vulnerability
Proceedings of the 4rnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
stems from the dynamics of the environment and is consid- ered within the framework of MDPs.
there has been a surge of research activity in the field of supervised learning, particularly focusing various adver- sarial attacks and corresponding defense methods (Kurakin et al., 2016; Papernot et al., 2016; 2017: Carlini & Wagner, 2017; Ilyas et al., 2018). In the context of DRI,, Huang et al. (2017) demonstrated that similar challenges could arise from small perturbations, such as the Fast Gradient Sign Method (FGSM). This study led to the early proposal of various attacks observations and corresponding robust RL methods (Kos & Song, 2017; Behzadan & Munir, 2017; Mandlekar et al., 2017; Pattanaik et al., 2018).
To illustrate this, consider a scenario where an agent must cross deep valley using one of two bridges. One bridge is short but narrow, while the other is longer but wider. With clear vision, the agent may prefer the shorter path; how- ever, under noisy or partially observed conditions-sus heavy fog-the the wider, safer bridge may be the more reli- able choice. Such decision decision-making requires incorporating long-term rewards considerations into robustness design, rather than focusing solely on local smoothness or output consistency as is common in supervised learning settings.
as
Recently, a line of research has focused on maintaining the consistency (smoothness) of the agent's s policy to acquire robustness against observation perturbations (Zhang et al., 2020b; Shen et al., 2020; Oikarinen et al., 2021; Sun et al., 2024). Zhang et al. (2020b) first defined the adversarial ob- servation problem as State Adversarial-MDDR (SA-MIPss) They proved and demonstrated that the loss in performance due to adversarial perturbations could be bounded by the consistency of the policy. However, in this study, they did not show practical methods to create the adversary that could estimate the long-term rewards assumed in the SA- MDPs. Due this gap, the proposed robust methods were not robust enough against stronger attacks.
To address the observation vulnerability stemming from the MDPs, Zhang et al. (2021) propose an approach where the (vietim) agent and the corresponding (optimal) adversary are trained alternately, resulting in robust agent. They refer to this framework as Alternating Training with Learned Adversaries (ATLA).
Although this alternating approach is theoretically robust and has shown practical success, it has several limitations and unresolved challenges.
First, the mutual dependency between the victim agent and the adversary effectively doubles the required sample size for training, resulting in inefficiencies and increased com- putational cost. Moreover, to the best of our knowledge, no existing off-policy actor-critic method can address such interactions within the standard MDP framework.
To address this issue, Zhang et al. (2021) proposed that the optimal (worst-case) adversary for the policy could be learned as a DRL agent. The policy, learned alternatively with such an adversary, can become robust against strong attacks (ATLA). Building on the ATLA framework, Sun et al. (2022) suggested dividing the adversary into two parts: searching for the mislead direction in the action space for the policy and calculating the actual perturbations in the state space through numerical approximation (PA-AD). This makes the adversary capable of handling high-dimensions state problems (such as Atari), which are difficult to ad- dress in the ATLA framework. These frameworks provide practical methods for on-policy algorithms (PPO, A2C), but applications for off-policy algorithms were not shown. This is because the adversarial attacker learns from trajectories generated by the (current) fixed agent, but the agent policy is constantly updated during training.
In this study, propose novel framework that eliminates the need for additional interactions with the environment and is compatible with off-policy actor-critic algorithms. Our contributions are summarized as follows: (1) We theo- retically prove that it is possible to construct an alternative adversarial framework for observation robustness without re- quiring an explicit RL process for the adversary. (2) We pro- pose two practical algorithms that demonstrate both sample efficiency and robustness, while simultaneously providing insights into underexplored regions in off-policy RL.
Our research advances the fields of RL and off-policy frame- works by addressing key limitations and introducing a prac- tical approach to enhance performance and generalization.
Liang et al. (2022) introduced an additional worst-case action-value function to improve the policy's 's robustness by utilizing mixture with the original value function, re- ferred to as Worst-case-aware Robus/ RI. (WocaR---R. This approach computes the worst-case action-value through convex relaxation and heuristic gradient iterations, thereby omitting additional environment steps, unlike ATLA-basch methods (ATLA, PA-ATLA). WocaR-RL demonstrated ef- fectiveness in high-dimensional discrete action domains using off off-policy algorithms (e.g., DQN). However, applica- tions for off off-policy algorithms in continuous action domains
2. Related Work
Adversarial A and Defense on State Observations. Building on a seminal work by Goodfellow et al. (2014),
'This assumes that neither environment model (true or ap- proximated) nor sufficient prior data is available. Such models or data entail challenges like model errors, data collection, and OOD issues. These issues are actively studied in the contexts of model-based and offline RL, but are considered out of scope in this work in order to maintain focus.
2

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Environment
Environment
Nothing More!
Interact Erv.
Interact Env.
Interact Env.
Ynteract Edv.
gather
F(s'ls,
gather
gather
F(s'|s, @)
(s, myyzx
(s,3, -T, K
(s, G, r, T.yoo
Agent Training
Adversary Training
Agent Training
Adversary Training
using
using
using
Virtually Done via:
nT(ālS), (((as)).Vle<$51
Tyxx v((s)
π () (Slss
update
update
update
(1) Symmetric Property
Policy Evaluation
Policy Evaluation
Policy Evaluation
argmax, fof Vi argmax, post
Soft Worss Casse Bellman
V Vftx
V V
V
T T V
ysoft + T yooft V
Policy Improvement
Policy Improvement
Policy Improvement
(2) Soft Optimization
argmaXRen Vfix
y = vrgmaX Vi ftx
π Tooft argmax
V
yoofe
V
ssoft -Analyticul Solution
-Numerical Method
(a) Alternative training framework (ATLA)
(b) Our proposal framework (VALT)
Figure 1. Overview of two learning frameworks for creating a robust agent by countering a corresponding adversary. Figure la illustrates the conventional approach, which altemaaes between training the agent via RL and training the adversary via another RL. This method requires twice as many environmental interactions due to the mutual dependency between the agent and the adversary. In contrast, our proposed approach, shown in Figure 1b, climinates this inefficiency while still accounting for the corresponding adversary.
were not shown and remain unknown.
against reward poisoning during training has also been inves- tigated (Ma al., 2019; Rakhsha et al., 2020), though such attacks typically considered as training-time data pertur- bations rather than test-time vulnerabilities, since the reward function itself does not directly affect policy performance at evaluation time (Schott et al., 2024).
As separate line of research, there are studies that address the ratio and temporal strategies of attacks as multi-agent problem (Lin et al., 2017; Gleave et al., 2020; Sun et al., 2020; Liu et al., 2024; Franzmeyer et al., 2024). However, these studies focus on the objectives of attack efficiency or the stcalthiness of attacks. This motivation differs from our approach, which limits the attacker based on constraints from an assumed distribution.
In connection with off oof-policy RL, recent works in offline RL have proposed methods that incorporate adversarial dynam- ies avoid duitr--dist (OOD) actions via max-min formulation (Rigter et al., 2022), as well as techniques that handle perturbations in dynamics or inputs under offline settings (Panaganti et al., 2022; Tang et al., 2024).
Adversarial Attacks on Other Components. Robustness against dynamics uncertainty has been extensively studied, starting from the classical Robust MDP formulation (Nilim & El Ghaoui, 2005; Iyengar, 2005; Tamar al., 2014), which considers set of uncertain transition models. Sub- sequent works have extended this line of research by incor- porating additive perturbations to the dynamics (Morimoto & Doya, 2005; Pinto et al., 2017; Reddi et al., 2024; Dong et al., 2025), handling dynamics parameters (episodic vari- ations) (Peng al., 2018; Vinitsky al.. 2020; Tanabe al., 2022), or modeling time-varying dynamics based contextual information (Zhang et al., 2023).
Although many of these studies are related in spirit, they are based on different problem settings and assumptions. Therefore, we do not conduct direct empirical comparisons in this work.
Additional related work and details are provided in Ap- pendix A.
3. Preliminaries and Background
3.1. State Adversarial Markov Decesion Process
Other studies focus on perturbations in the agent's action output or its stochastic components, often assuming noise- like corruptions during execution (Tessler et al., 2019; Klima et al., 2019; Lee al., 2020; Tan et al., 2020). Robustness
We assume that the agent receives perturbed inputs from the environment, as proposed in SA-MDP (Zhang et al., 2020b). which deviates from standard MDP. The SA-MDP can
3

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
be characterized by the parameters (8, A,,, where: S denotes the state space, S represents the obser- vation space available to the agent, Á is the action space, F S.S x Á + (SS) defines the environment's transition probabilities, with (S)) denoting the set of probability dis- tributions over the state space S. r is the reward function, y E (0.0, 1.0) is the discount factor, and So is the set of initial states. In the observation-robur RL framework, we assume the agent observation is perturbed by some environ- ment model (adversary), v s) E N S - (SS), then the agent policy, T 8) E IT: S > ((A), acts based on the perturbed observations, Here, N is the set of all possible perturbation models, and II is the set of all possible policies.
for the agent with a fixed adversary, and Vas (s) represents the expected value for the adversary with fixed agent. Due to the two optimization schemes, (3) and (4), previ- ous works (Zhang et al.. 2021; Sun et al., 2022) have been limited to on-policy methods (e.g., TRPO, PPO, A2C), and use twice the number of training steps. We summarize the procedures for this framework in Fig. la.
4. Methodology
To address the issue of sample inefficiency, we propose a new framework that does not require additional interactions with the environment while achieving optimization compa- rable to alternative training, as depicted Fig. 1b. The core strategy consists of two key steps:
The goal of robust RI. is to derive a policy that max- imizes the cumulative discounted reward along tra- jectory, even under perturbed observations: J[, æ| R... 1=== 7 ('r(s,,,,)) - To mitigate the high vari- ance associated with trajectoryddependent estimation, most research focuses on estimating the expected action- value and state-value functions, defined respectively as ( a) = Ee, Ei (r((s,,á) 880 = 8, āg = a] and/or (s) Bu,,.F EE 7'((st, @e) 50 = s], in-
(1) Symmetric Property: Updating the agent through policy evaluation with a symmetric property, which enables deriving the optimal adversary's 's value function directly from the agent's 's perspective. without requiring an explicit training process for the adversary.
(2) Soft Optimization: Applying soft optimization scheme with utilizing the agent's value estimation to obtain the (soft) optimal adversary.
stead of directly maximizing J.
To provide meaningful scenarios, we assume that the per- turbation is constrained within an - -nom ball of radius e around the true state as the most related research (Pat- tanaik et al., 2018; Zhang et al., 2020b; 2021; Sun et al., 2022; Oikarinen et al., 2021; Liang al., 2022). It can be represented a restricted perturbation subset.
These procedures are naturally realized by introducing the soft-constrained adversary, which enables efficient optimiza- tion under the proposed framework. We refer to this frame- work as Virtual ALtemative Training (VALT), as lacks an explicit RI. process for the adversary.
B. = (
Ve E::: (= 8) = if 13 =//0 >e, } (1)
In this study, we adopt the Soft Actor-Critic (SAC) (Haarnoja et al., 201 20;bb;) framework as the base off-policy actor-critic method, but the discussion remains applicable even without the entropy term H.
otherwise (3 {s) 2 0,
3.2. Max-Min Alternative Training for Robust RL
To learn a robust agent policy that performs effectively under worst-case scenarios induced by adversary. the solution satisfies the following Nash-equilibrium condition:
4.1. Soft Constrained Adversary
The optimization problem for the adversary in Eq. (3) and Eq. (4) is a hard-constrained problem due to the constraint defined in Eq. (1). To address this, we relax the problem by introducing a general f f-divergence term, modifying the total objective function as follows:
D Tyy arg max arg min J[, m]
nell
vEB.
(2)
arg min arg max J/v m].
vEB.
Zhang et al. (2021) adopt an alternating training approach to approximate this equilibrium. In this method, the agent and adversary are alternately optimized while holding the other fixed. The process alternates between the following objectives:
Definition 4.1 (Soft Constrained Optimal Adversary).
Asoft
arg m (MVx ml); PEN
(5)
J|, n] A Jlvs TT] + aent H( OV) v) + aattk Dy(u || p)-
A arg max Eorpp.TIe B
Ten [V [s)] ~ -J [M, Tox]
(3)
(6)
T/ arg max E Ea-pener Hell
VT (s) J [Pixx m]
(4)
Here, Dy(v p) represents the f-divergence with respect to the prior distribution p(8 s), and aank is the corresponding regularization coefficient. The term H(T o v) represents the Shannon entropy of the joint distribution x o v, with Cent as
where Pu,x represents the discounted state visitation proba- bility under v and n, VM, (s) represents the expected value
4

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
its coefficient. In this introduction, we make the following two assumptions:
This theorem demonstrates that the adversary's value func- tion can be avoided by leveraging the agent's action-value function, as the fixed point satisfies V =V Vanaa
Assumption 4.2 (Sufficient Support Assumption of the Prior). For all s, 8 E S, if ( 8) > 0, then p( s) > 0. Assumption 4.3 (Properties of the f Function). The func- tion f used in the f-divergence term. / (:) ). is assumed to be continuously differentiable and convex with respect to
youa-
To exploit this property, we ensure that Tem satisfies the contraction property by solving Eq, (5) from the agent's per- spective. In the next subsection, we propose two approaches to achieve this.
Assumption 4.2 is easily satisfied in this work by setting p as the uniform distribution such that dom(p) = dom (B.). For Assumption 4.3, we select appropriate functions for f, as described in a later subsection.
4.3. Soft Optimization Approach
We aim to obtain the (soft) optimal adversary in Eq. (5) by utilizing the agent's value function:
Daaoft
arg max- J[, x] = arg max Egsup |- Vynan (s)]
4.2. Symmetric Policy Evaluation
PEN
MEN
To approximate JLv, x using the value function for adver- sary learning, we first consider the formulation of the RL
This approach climinates the need for direct adversary policy improvement, which is conventionally expressed as:
process. In the following, we describe a soft worst-case (op- timal) policy evaluation for both the agent and the adversary, assuming fixed agent policy.
V * arg max Egsup PEN
[V (s)
Proposition 4.4 (Soft Worst Policy Evaluation for Agent).
Approach 1: Analytical Solution- Modeling. In this first approach, we utilize the case where an analytical solution can be derived.
(T) āc) 2 r((st āz) + yEx
min VyTte (ot=1) PEN
(7)
(st) V (s)) - E. EE (Q 6z]]]
+ Cent H(T v) Qatth (y 11 p),
Lemma 4.7 (KI-divergence Analytical Solution). If we set f(x) = x log(z) in Eq. (7), the analytical solution of Eq. (5) is given by:
Proposition 4.5 (Soft Optimal Policy Evaluation for Adv.).
(T+ (5), 5e) c(st, 3t) + BBFor
max
Vi
(8++1)
p( p(5|))
8) exp ( (-VT(s,8)/aatt)) (9)
WEN
(5 s)
Z
Vi" (st) E. (s,,(s, 5:]] Qent -((T o v) Dy (1///// p),
where V (s, 5) =Ex 1Q (C((s)a) â)] + Qent Qen)H(TT) and the partition function that normalizes the distribution.
(8)
where c(s, 8) Ex(-r(s,ā) a]] is reward function for the adversary, Tem is Bellman Operator for the agent, and Th""h is a Bellman Operator for the adversary, both in the soft worst adversary case as in Eq. (5). Due to a symmetric structure of the two propositions, we call the important theorem as:
Moreover, regarding the policy evaluation for the agent:
Theorem 4.8 (Contraction Property in the KL Case). In the case of Eq. (9), the Bellman operator you possesses the "ycontracttion property.
The full derivation of the Lemma 4.7 and the Theorem 4.8 is provided in Appendix B.2.
Theorem 4.6 (Symmetry of y-Contraction Properties). For fixed agent policy л, if there exists bounded func- tion Q77 (s, ā) such that (s, 5) IxI-Q' ā)], and if Ti is a "ycontraccion operator, then THx also a -ccontraction operator. Moreover, Tymo and THT" share fixed points with opposite signs, Vy = max, V
For the practical implementation, similar to the policy im- provement in SAC (Haarnoja al., 201 2018,;)), we prepare variational model, Vmodel. then update this model so as to be similar distribution 1/ ssott
L(Vonda) ExcpR DKL Mandaal || pexp (-V* htp7/aasr /aattk)
V -min, pag
VT yrng
Z
Proof Overview. Substitute Q% = EA[ (x,
x 1pp [Eumnse [craeck log (m s) + V const.]]
(10)
and c(s,8) 8) = E. E- a)] into Eq. (8). By multiplying both sides of Eq. (8) by 1, and flipping max into min,
We refer to this strategy as Virtual Alternative Training with the Wortt method (VALT-SOFT). See Appendix C.2 for more details about the implementation.
Eq. (8) can be transformed into a form similar to Eq. (7). See Appendix B.I for more details.
LI
5

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Approach 2: Approximation-Basee Numerical Solution. Inspired by the work (Belousov & Peters, 2017; 2019), we can transform Eq. (5) into non-minimization form as:
Proposition 4.11 (Policy Improvement with Fixed Adver- sary).
Tarw = arg min DKI (TT uge || wiis o vga); m111
Lemma 4.9. (Dual Form of Soft Constrained Adversary)
exp (Qun Toll (8, (km))) V.
(13)
us 200 (3 s) p(5 ()(M))))
1 1/ (-rr)--------- (-V=->
--V
x($
S.L.
Rold
7
Cuttk
(11)
To confirm the improvement achieved by this operation, we present the following theorem:
where (f*)) represents differential function of the conju- gated function for f. X means Lagrangian variable, Ig is an unit vector for the observation space, and i. is comple- mental slackness variable.
Theorem 4.12 (Policy Improvement Theorem with a Fixed Adversary). Civen a fixed adversary v and a policy x. define a new policy Tnes such that for all states 8 as Eq. (13). assuming that Q7 and Ened exp((T7 (s, a)) are bounded for all states 8, it follows that
This result shows that the solution distribution of Eq. (5) shaped based on the prior p. and that f, Gattk, and V determine how the probability mass is distributed over the space S. Here, we focus a-divergent cases, which are a subclass of f-divergences, allowing us to control the ten- dency of the distribution's 's shape. When a < 1, the dis- tribution concentrates strongly on the worst-case scenarios while remaining flat elsewhere. We can approximate this distribution by using the mode probability, denoted Kioorss and the flat probability for others as follows:
Que (8, 22)20000 (s,
(14)
for all actions a and states s.
Proof. See Appendix B.4.
Considerations for Behavior Policy. To calculate Eq. (13), we use data from replay buffer R as: Tnew arg min Ep [Dka]- For collecting It we use a uniform distribution p as a proxy due to the high computational cost of PGD in VALT-EPS. In the HalfCheetah environment, we rely entirely on states with adversarial noise, whereas for the high-dimensional input task, Ant, training VALT-SOFT with entirely adversarial state inputs leads to corruption due to over-pessimism. We found that 50:50 mix of π(. s) and o v(- 8) generally works well. A key challenge in off- off-policy robust learning remains: the mismatch between the data distribution PR and the distribution induced by the current behavior policy. Future work involves addressing this mismatch by refining adversarial intensity and improv- ing methods for handling off-policy data more effectively, while accounting for the progress of learning, Appendix E.3 analyzes how the adversarial ratio in the behavior policy affects performance.
Riweest +
1- wooon
if arg min V (s,8",
caoft (8]s) 1
~
[8.1
S'EB
Gocml [
otherwise,
(12)
where |S.] represents the measure of the observation state space within the e-bounded domain. Also this case satisfies the required condition for the Theorem 4.6, otherwise
Theorem 4.10 (Contraction Property in the Epsilon Case). In the case of Eq. (12), the Bellman operator T possesses the 7-contraction property.
A derivation of the Lemma 4.9 and a proof of the Theo- rem 4.10 are provided in Appendix B.3. In the practical use case, Eq. (12) can be approximated by combining the uni- form distribution with a numerical gradient approach, simi- lar to the Projected Gradient Decent (PGD) attack (Madry et al., 2018: 8: Pattanaik et al., 2018; Zhang et al., 2020b). We refer this strategy as Virtual Alternative Training with the Epsilon-Wood Worst method (VALT-EPS). See Appendix C.1 for more details about the implementation.
5. Experiments
In this section, we set up experiments to confirm our algo- rithm achieve good sample efficiency and robustness es- pecially against the adversaries. We use four OpenAl Gym MuJoCo environments (Brockman et al., 2016; Todorov et al., 2012): Hopper, HalfCheetah, Walker2d, and Ant, as utilized in most prior works (Zhang et al., 2020b; 2021; Oikarinen et al., 2021; Liang et al., 2022).
4.4. Policy Improvement with Fixed Adversary
Based the preceding discussion, we demonstrate that the soft (worst-case) adversary can be determined without requiring additional interactions or extra value estimations. In the final phase, we fix the adversary and enhance the agent's 's policy using a modified version of the maximum- entropy policy improvement approach.
5.1. Baselines and Implementations.
We select SAC (Haarnja et al., 2018a;b) as our base off- policy method. Since existing studies have not reported
6

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
HalfCheetah
Hopper
Walker2d
Ant
12000
8000
6000
Avrage Episodic Return
7000
10000
5000
6000
8000
4000
5000
6000
4000
3000
1500
3000
Rohu-st-PPO
ATLA-PPO
1000
2000
2000
WPCaRRPPP
2000
500
1000
1000
Robust-SAC
SACPPO
VALT-SOFT
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.0
0.5
1.0
1.5
2.0
(a) Learning curves for all methods.
12000
8000
Avrage Episodic Return
1 10000
3000
5000
6000
8000
6000
4000
3000
2000
1000
2000
2000
1000
RobusT SAC VALT-SOFT
0
0.00
0.25
0.50
0.75
1.00
0.0
0.5
1.0
1.5
3
1e6
1e6
Env. Steps
(b) I Learning curves focusing only on sample-efficient SAC variants.
Figure 2. Learning curves for four MuJoCo control tasks evaluated under no-attack settings. Subfigure 2a shows results for all algorithms, while Subfigure 2b focuses on sample-efficient off-policy methods. Solid lines indicate average evaluation scores, and shaded areas represent standard deviations across different random seeds. For simplicity, we abbreviate +reg., +wreg. and "PA-", and omit "-SAC" unless explicitly stated. Note that these ligures illustrate training performance and sample efficiency under clean conditions, while robustness against attacks is evaluated separately in Table 1.
the application of the latest off-policy Actor-Crite meth- ods?, particularly SAC, we incorporated these methods into our SAC implementation and tuned the appropriately. Additionally, for comparison, we included a representative robustness-enhancin, method based on the on-policy Proximal Policy Optimization (PPO) algo- rithm (Schulman al., 2017) our experiments.
For fairness, the LSTM-based PPO when available in the provided code.
Comparison with Off-Policy (SAC) Settings. We pre- pare four baseline algorithms for off-policy settings: (1) the original SAC (SAC); (2) Robust-SAC, a SAC version of the robust regularizer proposed in Zhang et al. (2020b); and (3) SAC-PPO. an off-policy max min alternative training method similar ATI.A-PPO, but using SAC the agent and PPO the adversary.
Comparison with On-Policy (PPO) Settings. We eval-
uate five representative methods: (1) Original PPO (PPO) (Schulman et al., 2017); (2) Robust-PPO, which incorporates a robust regularization term into PPO, as pro- posed by Zhang et al. (2020b; 2021): (3) ATLA-PPO, an ad- versarial training approach based on Eq. (3), introduced by Zhang al. (2021): (4) PA-ATLA-PPO variant of ATLA- PPO designed operate in lower-dimensional space than the observation space (Sun et al.. 2022); (5) WocaR-PPO, which integrates the worst-case-awarc action-valie function method proposed by Liang et al. (2022).
SAC-PPO is novel method we introduced specifically for this comparison, offering theoretically sound alternative training for off-policy algorithms, although it suffers from sample inefficiency. the Ant environment, SAC-PPO combined with the PA-ATLA technique (Sun al., 2022) to address the large observation space; refer to this variant as PA-SAC-PPO.
We also attempted to adapt the WocaR technique to the off- policy setting (WocaR-SAC), but it failed to perform consis- tently across the four benchmark tasks. The corresponding experiments and analysis are described in the paragraph
Whiile DDPG is implemented in Zhang et al. (2020b), it ap- pears that DDPG does not perform adequately the four MuloCo benchmarks.

---

Off-Pollcc Actor-Critic for Observation Robustness: Virtual Alternative Training
Table 1. Average episodic rewards (+ standard deviation) for the median seed models of our proposed methods (VALT-EPS, VALT-SOFT) and other baselines across four MuJoCo tasks. For learning barning-based attacks, attack parameters are consistently optimized, and the minimum scores are reported. All evaluations are conducted over twenty episodes with different random seeds. The highest scores within each on-policy or off-policy category are highlighted in bold, and among the most sample-efficient methods, the highest-scoring ones are shaded in gray...
As robustness metric, WC summarize the lowest score observed across all attacks in the rightmost column.
Envtronment
Method
Natural Reward
Unifurmn
MAD
(minV)
PGD
PGD
SA-RL (SAC)
PA-AD (SNC)
SA-RL (PP0)
PA-AD (PPO)
Weest ottucks
PPO Robus:
59172:136
5674
3521-248
1447
4448=134
65:55
61136
21=77
125-231
50132146
55301:10 141
4596:1125
4792163
3313-1139
5115:122
34081903
253981092
37365625
2599
ATLA-1E0 -ng.
5062::06
5168:113
4813:110
4820:9979
4520-136
6132:967
4048
7504-720
37954573
7504
PA-ATT-A-PPO ereg.
5012159
suse dim
RFO Fiveeg
2M
4353419
41981:455
324561256
41271-747
3875=106
1995
1566:1081 1081
264361224
1566&1006
adtinndim
c- 0.15
Rebust
1995
63501168
6111
6127454
5857
94344-171
25011:311
4215-247
3183-375
202644177
17575-419
34312393
5197-533
19714:144
33672-468
2036:4417
-51--207
SAC2P0 +ing.
7487:147
7154:14
6030 1084
1823--871
5836-138
8706-1122 12
28526835
349-776
3008+100
1823-871
VALT-EPS-SAC 4mgg
61276112
5900154
94
4647:189
5814=107
1316:199
53963113
5707 100
4647:150
MALF-SUFTSSAAC +ing
IM
598
5798155
54363.0
4639169
4713=K3
PPO
5M
30151579
3122-743
24632742
356R
1360-227
2628
2683 1050
571
L4604510
571:1466
Robou--PPO
SM
9617421
1670 56
1056-088
1599-117
3814 403
M86
3278 677
1528-
118 1908
1938-959
AILA PPO srop.
34548333
3581152
444514414
3453464
455
200211311
1354
1666-665
208014477
Huyper
PA-ATLA-FPO
2502:019
3696:819
1301-301
1739
3052=1162
1000-114
1000:1144
Wooak-99U +wang
969--9
M633-245
3686115
1175:410
2477-444
278667004
1015-541
1909
actiondii::
SAC RAbua-SAC
3100=1
8213:6
535715
1979
3165411
2079
0,073
3857
823418
2047:20
520022
10
3114126
295214
5015E8
204T16
14
17
1209-1
3286112
1
1196-3
VALT-EPS-SCC rigg
339722
5264211
3358-2
3367
335529
3264:6
3313
VALTSOFTSSAC kng
6.5M
5228:2
3200 1302:10
13
2016:3
346283
303413
300219
3059-7
3031
2006:3
Bolur.PPO
PPO
5M
SM
4
43314
7921
45
39471887
41625:1
2140:120
409727
540 1200
2035500
624 1365
281
196
10
590711955
20962
28118898
ATT A-PIO
IIM
W77- 105
4001-66
18836 544
5425 1060
2559
2406
1683
S416-941
2380-1773
MN-ATTA--P1O amg
10M
3570 1150
114-1185
1021-1117
4156 156
1341-915 910
253411807
1801
1625
1791
1765-995
900
1854--000
1791
dim:
Wocak P8O miveg
457
434
4050-1006
4391-831
5990-1000
317
adion-dinz
SAC
L.M
54661-47
9601 1006
42884-1280
2000-2510
21434-4552
330- 967
40155
12524-1778
357-567
0.05
Rabust SAC
5560121
5824241
54525862
5553136
5645
5648367
5190
SAC-PTO
3498
4044:1423
106511901
235321156
3266-1144
1912-159
176
568
LMSS
1656
1343-178
VALT-EPS-SC
5916190
506-146
5467--2671
5447--2621
VALESOFTSAC
LSM
5501224
5009137
5579
532719255
5000771
524918
1981
5455:119
5468
2 498121101
PIO
10M
5628 440
5886
2006
1813-11
147
1955-1888
5884735
114-177 170
511-30 30
143-79
-004 -004+130 139
I0M
5543-89
5483:
116
223
4240-10066
555
3391:142
83
3196 R3
AILA-PO +rg..
2IM
5433-997
S374E 100
51953 101
3855:11553
5147
105-5105
2400
Aut cont-dim 111
PA-ATLA--PO avos.
28M
53502:104
S318 122
5200
503422903
6059
405011044
3231:341
2300
27352145
2700
Wozak-F000 exan
10M1
5501+183
5490 174
4018-1014
5256 140
4811-949
76474
187-195
1526-125
886-167
-697+199 195
oction dim:
0.15
Robuss-SAC
SAC
3M
3M
655
1548
196
4847+2878
271774
1687+-125
23:53
937--899
-1061370
1213
31214
7047:7700
6811016
1055-1155
-25--322
1851-22085
-T10002
1185341221
1016
PA SAC PTO
6531:11730
1508
5080:11484
177545663
2200::1710 1794
416952144
354:4488
3754=691
400742110
3541-468
VALT-EPS&SAC 1ieg.
0352.7748
5129-22907
50194:1915
1176
2406
4710
4964:189) 1891
1735-3 1466
4585:30022
1487-1177
VALTSOFT:SAC
67191166
528752005
2757
3058:11221
3808
515251653
"WocaR-SAC Settings" in Appendix D.2.
ActionDiff (MAD) (Zhang et al., 2020b), PGD (Madry et al.,
2018; Pattanaik et al., 2018; Zhang et al., 2020b), and Ro- bustSarsa (RS) (Zhang et al., 2020b). For PGD, we use distinct notations to specify the target: PGD (minQ) for SAC and PGD (min V) for PPO.
Notation for Regularization Settings. Apart from the original PPO and SAC, robust regularization techniques are applicable and are used in most of the proposed methods. We denote methods that use the robust regularizer from Zhang et al. (2020b) with the suffix (+reg.), ), and those that utilize the robust regularizer from Liang et al. (2022) with the suffix (+wreg.).
Learning-Based Optimal Adversary Attacks. We fur- ther incorporate more advanced attackers, including SA-RL (PPO) (Zhang et al., 2021) and PA-AD (PPO) (Sun et al., 2022). These attackers are also evaluated with the SAC algorithm, where they are denoted as SA-RL (SAC) and PA-AD (SAC), respectively,
Additional details on the implementations and hyperparam- eters are provided in Appendix D.1 and Appendix D.2.
5.2. Attacker Settings
Those details of the implementations and hyper-parameters are provided in Appendix D.3.
We adopt the same attack scales commonly used in previous studies (Zhang et al., 2020b; 2021; Oikarinen et al., 2021; Sun et al., 2022; Liang et al., 2022): e 0.15, 0.075, 0.05, 0.15 for HalfCheetah, Hopper, Walker2d, and Ant, respectively.
5.3. Result
Training Sample Efficiency. Fig. 2a presents the train- ing curves for our proposed methods (VALT-EPS, VALT- SOFT) and other baselines. The x-axis represents the num- ber of environment interaction steps, while the y-axis shows the average episodic return during unperturbed evaluation.
Heuristic Attacks. For robustness evaluation, we employ several heuristic attack methods: Random (Uniform), Max-
8

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Most PPO-based algorithms require at least 5M steps for HalfCheetah, Hopper, and Walker2d, and 10M steps for Ant. Alternative training-based methods (ATLA-PPO, PA- ATLA-PPO) need upproximately twice as many steps as other PPO-based approaches due to their iterative training processes. Although WocaR-PPO is highly efficient due to the absence of adversary training. it still requires 2M, 2M, 2M, and 10M steps for HalfCheetah. Hopper, Walker2d, and Ant, respectively.
First, SAC-based methods demonstrate superior sample ef- ficiency. Under clean (unperturbed) environments, they achieve higher episodic rewards with fewer environment interactions than PPO-based methods, primarily due to the data efficiency of off-policy learning.
Second, SAC-based variants are more sensitive to the PGD (minQ) heuristic attack. This effect especially pronounced for methods based on alternating training, such as SAC- PPO, VALT-EPS, and VALT-SOFT. This sensitivity arises because SAC maintains an explicit action-value function that estimates long-term returns under the agent's policy, When these Q-functions are adversarially degraded (e.g.. by minimizing the predicted value), performance drops significanll- approximating that of an optimal adver- sary. We further analyze this phenomenon in Appendix E.2 by ablating the adversarial effect during policy evaluation.
In contrast, Fig. 2b focuses on sample-efficient off-policy methods, excluding SAC-PPO. SAC-based algorithms demonstrate superior efficiency, achieving training within 1M, 0.5M, 1.5M, and 3M steps for HalfCheetah, Hopper, Walker2d, and Ant, respectively. This corresponds to a 3x- 10x improvement over standard PPO-based methods and 1 1.5x-4x improvement over WocaR-PPO, the most efficient on-policy method known to us. Furthermore, VALT-EPS and VALT-SOFT achieve comparable or even more stable training across different seeds, particularly for Ant.
Third, SAC-based variants vanilla SAC and Robust-SAC--een to exhibit strong robustness on tasks such as Hopper and Walker2d. Eysenbach & Levine (2022) showed that the maximum entropy framework naturally induces robustness, with theoretical justification for re- silience against dynamics and reward variations. While our study focuses observation robustness, note that observation perturbations can be interpreted as modifying the agent's 's perceived dynamics, which can be formalized as J = (8 8, ā), where the true dynamics remain un- changed, but the agent cannot distinguish between perturbed and unperturbed states due to limited observability.
Robustness Evaluation. Table summarizes the evalu- ation results of the four benchmarks across all algorithms, with the environment interaction steps provided in the third column from the left. To facilitate an easy comparison of robustness among the methods, the worst attacked scores are presented in the rightmost column. Across all four bench- marks, our proposed methods, VALT-EPS and VALT-SOFT, exhibit exceptional robustness, particularly in the more chal- lenging tasks of HalfCheetah and Ant.
The task dependency of robustness and the suitability of specific methods across tasks could be further systematized theoretically. This represents an important direction for future research, particularly in practical applications.
SAC-PPO, which can be seen as a natural formulation of the alternative framework, demonstrates more robust per- formance compared to the original SAC. However, its ro- bustness remains moderate when compared to Robust-SAC. Interestingly, our proposed methods (VALT-EPS and VALT- SOFT) outperform not only the simple regularization-basee method Coob but also SAC-PPO, despite the lat- ter's potential to naturally implement the alternative formu- lation.
6. Conclusion and Future Work
We propose a novel approach that optimizes the correspond- ing adversary without requiring additional interactions with the environment, making it suitable for off-policy settings. Our algorithm is grounded in solid theoretical foundation, and evaluation results demonstrate both excellent sample efficiency and consistent success.
We hypothesize that although the natural alternative ap- proach aims to reach Nash equilibrium, its practical im- plementation is hindered by the difficulty of finding a sharp saddle point, which requires carefully balancing the learn- ing dynamics of both the agent and the adversary (Reddi et al., 2024). In contrast, our objective function, defined in Eq. (6), not only addresses the sample inefficiency of exist- ing methods, but also reformulates the problem as Quantal Response Equilibrium (QRE) (McKelvey& & Palfrey, 1995; Reddi et al., 2024), leading to a more stable learning process for discovering robust equilibrium solutions.
For future work, we highlight three directions. First, the algorithm can be improved by integrating recent advances in functional smoothness to enhance robustness. Second, the framework may be extended to other domains, such as domain randomization. Third, a promising direction is to explore adaptive correction methods that leverage past replay data and adapt to current environments, enabling more efficient learning in settings like offline RL, curriculum learning, and online domain adaptation.
SAC-based variants exhibit three notable characteristics that distinguish them from PPO-based variants.
9

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Acknowledgements
Boularias, A., Kober, J., and Peters, J. Relative entropy inverse reinforcement learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 182-189. JMLR Workshop and Conference Proceedings, 2011.
We sincerely appreciate the cooperation of the conference committee and the constructive comments and suggestions from the reviewers, which have significantly improved the quality of this paper. We also acknowledge the open- source software and libraries used in this research, includ- ing PyTorch, OpenAl Gym, MuJoCo, B auto LiRPA, as well as foundational work on robust rein- forcement learning.
Brockman, G., Cheung. V., Pettersson, L.. Schneider, J.,
Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint arXiv: Sttii::6000.114000 01540, 2016.
Carlini, N. and Wagner, D. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 3-14, 2017.
We would like thank Satoshi Yamamori and Sotetsu Koya- mada for their valuable advice on this research. We also extend our sincere appreciation all members of the Com- puter Science Domain at Honda R&D Co., Ltd., especially Ken linuma, Akira Kanahara, Kenji Goto, Umiaki Matsub- ara, Ryoji Wakayama, Keisuke Oka, and Koki Aizawa, for their cooperation and support.
Chow, Y. and Ghavamzadeh, M. Algorithms for cvar op- timization in mdps. Advances in Neural Information Processing Systems (NeurIPS), 27, 2014.
This research was supported by the authors' affiliation, Honda R&D Co., Ltd., and Kyoto University, Japan. It was partially funded by JSPS KAKENHI (No. 22H04998) and NEDO (No. JPNP20006) and made use of the super- computer system provided by the Institution for Information Management and Communication (IIMC) of Kyoto Univer- sity.
Chow, Y., Tamar, A., Mannor, S., and Pavone, M. Risk- sensitive and robust decision-making: a cvar optimization approach. Advances in Neural Information Processing Systems (NeurIPS). 28, 2015.
Chow. Y., Ghavamzadeh, M., Janson, L.,, and Pavone, M. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research (JMLR), 18(167):1-51, 2018.
Impact Statement
Christodoulou, Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910.07200) 2019.
Our research primarily focuses proposing robust RL methods, which we believe will significantly advance the practical applications of RI. However, important to note that robustmess methods, including those proposed in our methods, require additional computational resources compared to the original RL algorithms. Consequently, utilizing these methods in scenarios where robustness is not critical requirement may result in increased computational costs. Furthermore, as with RL in general, there exists potential risk of misuse in applications involving automation and optimization. Nonetheless, we believe that the positive utility of assisting humanity in creating a better society far outweighs these concerns.
Dong, J., Hsu, H.-L.,, Gao, Q., Tarokh, V., and Pajic, M. Variational adversarial training towards policies with im- proved robustness. Proceedings of the 28th Interna- tional Conference on Artificial Intelligence and Statistics (AISTATS), 2025.
Eysenbach, B. and Levine, S. Maximum entropy rl (prov- ably) solves some robust rl problems. In 10th Interna- tional Conference on Learning Representations (ICLR), 2022.
Filar, J. A., Kallenberg, L. C., and Lee, H.-M. Variance- penalized markov decision processes. Mathematics of Operations Research, 14(1):147-161, 1989.
References
Altman, E. Constrained Markov Decision Processes, vol- ume 7. CRC Press, 1999.
Filar, J. A., Krass, D., and Ross, K. W. Percentile per- formance criteria for limiting average markov decision processes. IEEE Transactions on Automatic Control, 40 (1):2-10, 1995.
Behzadan, and Munir, A. Whatever does not kill deep reinforcement learning, makes it stronger. arXiv preprint arXiv: 2017.
Fox, R., Pakman, A., and Tishby, N. Taming the noise in reinforcement learning via soft updates. In UAI, 2016.
Belousov, B. and Peters, f-divergence constrained policy improvement. arXiv preprint arXiv:1801.00056, 2017.
Franzmeyer, T., McAleer, S. M., Henriques, J. F., Foerster, N., Torr, P., Bibi, A., and de Witt, C. S. Illusory attacks: cetemmattoo-thhooree detectability matters in adversarial
Belousov, B. and Peters, J. Entropic regularization of markov decision processes. Entropy, 21(7):674, 2019.
10

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
attacks. In International Conference on Learning Repre- sentations (ICLR), 2024.
Ilyas, A., Engstrom, L.., Athalye, A., and Lin, J. Black-box adversarial attacks with limited queries and information. In Proceedings of the International Conference on Ma- chine Learning (ICML), pp. 2137-2146. PMLR, 2018.
Fujimoto, S., Hoof. H., and Meger, D. Addressing function approximation error in actor-critic methods. In Inter- national Conference on Machine Learning (ICML), pp- 1587-1596. PMLR, 2018.
Tyengar, G. N. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
Gleave, A., Dennis, M., Wild, C., Kant, N., Levine, S., and Russell, S. Adversarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2020.
Kamalaruban, P., Huang. Y.-T., Hsich, Y.-P.. Rolland, P., Shi, C.,, and Cevher, V. Robust reinforcement learning via adversarial training with langevin dynamics. Advances in Neural Information Processing Systems (NeurIPS), 33: 8127-8138 2020.
Goodfellow, I., Shlens, J., and Szegedy. C. Explaining and harnessing adversarial examples. JSME International Journal, Series Solid Mechanics, Strength of Materials, 33(4))468-473, 12 2014.
Kim, G.-H., Seo, S., Lee, J., Jeon, W., Hwang, H., Yang, H., and Kim, K.-E. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In Interna- tional Conference on Learning Representations (ICLR), 2021.
Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T., and Kohli, P. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv: 1ww..1111...111111 2018
Kim, G.-H., Lee, J., Jang, Y.. Yang. H., and Kim, K.-E. Lob- sdice: Offiin learning from observation via stationary distribution correction estimation. Advances in Neural In- formation Processing Systems (NeurIPS), 35:6252-8264, 2022.
Ha, S., Xu, P., Tan, Z., Levine, S., and Tan, J. Learning to walk in the real world with minimal human effort. In Proceedings of the Conference on Robot Learning (CoRL), volume 155 of Proceedings of Machine Learning Research, pp. 11 110-1120. PMLR, 2021.
Kingma. D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Rein- forcement learning with deep energy-basee policies. In International Conference on Machine Learning (ICML., pp. 1352-1361 PMLR, 2017.
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Al Sal- lab, A. A., Yogamani, S., and Pérez, P. Deep reinforce- ment learning for autonomous driving: A survey, IEEE Transactions on Intelligent Transportation Systems, 23 (6):4909-4926, 2021.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off- off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Con- ference on Machine Learning (ICML), pp. 1861-1870. PMLR, 2018a.
Klima, R., Bloembergen, D., Kaisers, M., and Tuyls, K. Robust temporal difference learning for critical domains. Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 350-358, 2019.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.008 2018b.
Kober, J., Bagnell, J. A., and Peters, J. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, (211)::238-127,, 2013.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Kos, J. and Song, D. Delving into adversarial attacks on deep policies. arXiv preprint arXiv:1705 064500 2017.
Kostrikov, I., Nachum, O., and Tompson, J. Imitation leam- ing via off poolcy distribution matching. In International Conference on Learning Representations (ICLR), 2020.
Howard, R. A. and Matheson, J. Risk-sensitive markov decision processes. Management Science, 18(7):356-36, 1972.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), volume 25, pp. 1097-1105, 2012.
Huang, S., Papernot, N., Goodfellow, I., Duan, Y., and Abbeel, P. Adversarial attacks on neural network policies. arXiv preprint arXiv: orXiv: 7002 2017.

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Kurakin, A., Goodfellow, I., and Bengio. S. Adversar- ial examples in the physical world. arXiv preprint arXiv:I607.002 2016.
Mandlekar, A., Zhu, Y., Garg, A., Fei-Fei, L.., and Savarese, S. Adversarially robust policy learning: Active construc- tion of physically-plausible perturbations. In IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pp- 3932-3939. IERE, 2017.
Lee, X. Y., Ghadai, S., Tan, K. L., Hegde, C., and Sarkar, S. Spatiotemporally constrained action space attacks deep reinforcement learning agents. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 34, pp. 4577-4584, 2020.
McKelvey, R. D. and Palfrey, R. Quantal response equi- libria for normal form games. Games and Economic Behavior, 10(1):6-38, 1995.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to- end training of deep visuomotor policies. The Journal of Machine Learning Research (JMLR), 17(1):1334-1373, 2016.
Mnih, V., Kavukeuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529-533, 2015.
Liang, Y., Sun, Y., Zheng, R., and Huang, F. Efficient ad- versarial training without attacking: Worst-case-awarr robust reinforcement learning. Advances in Neural Infor- mation Processing Systems (NeurIPS), 35:22577-22561, 2022.
Moos, J., Hansel, K., Abdulsamad, H., Stark, S., Clever, D., and Peters, J. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction, 4(1):276-3155 2022.
Lillicrap. T. P.. Hunt. J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous con- trol with deep reinforcement learning. International Conference on Learning Representations (ICLR), 2016.
Morimoto, J. and Doya, K. Robust reinforcement learning. Neural Computation, 17(2):335 -39,, 2005.
Nachum, O. and Dai, B. Reinforcement learning via fenchel- rockafellar duality. arXiv preprint arXiv:2001.00 01866, 2020.
Lin, Y.-C., Hong, Z.-W., Liao, Y.-H., Shih, M.-L., Liu, M.- Y., and Sun, M. Tactics of adversarial attack on deep reinforcement learning agents. In Proceedings of the 26th International Joint Conference on Artificial Intelligncce (IJCAI), pp. 3766-3762, 2017.
Nash, J. Non-cooperative games. Annals of Mathematics, 54(2):286-295, 1951.
Nilim, A. and El Ghaoui, L. Robust control of markov decision processes with uncertain transition matrices. Op- erations Research, 53(5):780-798, 2005.
Liu, X., Chakraborty, S., Sun, Y., and Huang, F. Rethinking adversarial policies: A generalized attack formulation and provable defense in rl. In International Conference on Learning Representations (ICLR), 2024.
Oikarinen, T., Zhang, W., Megretski, A., Daniel, L., and Weng, T.-W. Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Pro- cessing Systems (NeurIPS),
Liu, Z., Guo, Z., Cen, Z., Zhang, H., Tan, J., Li, B., and Zhao, D. On the robustness of safe reinforcement learn- ing under observational perturbations. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
2021.
Osogami, T. Robustness and risk-sensitivity in markov decision processes. In Advances in Neural Information Processing Systems (NeurIPS), volume 25, 2012.
Ma, Y., Zhang, X., Sun, W., and Zhu, J. Policy poisoning in batch reinforcement learning and control. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M. Robust reinforcement learning using offline data. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Sys- lems (NeurIPS), 2022.
Ma, Y., Shen, A., Jayaraman, D., and Bastani, O. Versa- tile offline imitation from observations and examples via regularized state-occupancy matching. In International Conference on Machine Learning (ICML), pp. 14639- 14663. PMLR, 2022.
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, B.-Z., and Swami, A. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on Security and Privacy (EuroS&P), pp. 372-387. IEEE, 2016.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations (ICLR), 2018
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, B, B.-Z., and Swami, A. Practical black-box attacks against
12

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
machine learning. In Proceedings of the 2017 ACM on Asia Conference Computer and Communications Se- curity (ASIA CCS), pp. 506-519,2011
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In Proceedings of the 32nd International Conference Machine Learning (ICML), volume 37, pp. 1889-1897, PMLR. 2015.
Pattanaik, A., Tang. Z., Liu, S., Bommannan, G., and Chowdhary, G. Robust deep reinforcement learning with adversarial attacks. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Sys- tems (AAMAS), 3, 2018.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms.
arXiv preprint arXiv: arXiv: 2017.
Shapley, L. S. Stochastic games. Proceedings of the Na- tional Academy of Sciences, 39(10)1095-1100, 1953.
Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3803-3810 3810. IEEE, 2018.
Shen, Q., Li, Y., Jiang, H.,, Wang, Z.,, and Zhao, T. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning (ICML), pp. 8707-8718. PMLR, 2020.
Peters, J., Mulling, K., and Altun, Y. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 24, pp. 1607-1612, 2010.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-559, 2017.
Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. Ro- bust adversarial reinforcement learning. International Conference on Machine Learning (ICML), 70:2817-2826, 2017.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Interna- tional Conference on Learning Representations (ICLR), 2015.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines:: Reliable rein- forcement learning implementations. Journal of Machine Learning Research (JMLR). 22(268):1-8, 2021.
Sun, C.-E., Gao, S., and Weng, T.-W. Breaking the bar- rier: Enhanced utility and robustness smoothed DRL agents. In Forty-first International Conference on Ma- chine Learning (ICML), 2024.
Rakhsha, A., Radanovic, G., Devidze, R., Zhu, X., and Singla, A. Policy teaching via environment poisoning: Training-tin adversarial attacks against reinforcement learning. In International Conference Machine Learn- ing (ICML), pp. 7974-7984. PMLR, 2020.
Sun, J., Zhang, T., Xie, X., Ma, L., Zheng, Y., Chen, K., and Liu, Stealthy and efficient adversarial attacks against deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 34, pp. 5883 3883-5891, 2020.
Reddi, A., Tölle, M., Peters, J., Chalvatzaki, G., and D' D'Eramo, C. Robust adversarial reinforcement learn- ing via bounded rationality curricula. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Sun, Y., Zheng, R., Liang, Y., and Huang, F. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl. In The Tenth International Conference on Learning Representations (ICLR), 2022.
Rigter, M., Lacerda, B., and Hawes, N. RAMBO-RL; Ro- bust adversarial model-based offline reinforcement learn ing. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2022.
Sutton, R. S. and Barto, A. G. Introduction to reinforcement learning. MIT Press, 1998.
Szegedy, C., Zaremba, W., Sutskever, Bruna, J., Erhan, D., Goodfellow, I,, and Fergus, R. Intriguing proper- ties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Priori- tized experience replay. In International Conference on Learning Representations (ICLR), 2016.
Tamar, A., Mannor, S., and Xu, Scaling up robust mdps using function approximation. In Proceedings of the 31st International Conference on Machine Learning (ICML), pp. 181-189, PMLR, 22-24 Jun 2014.
Schott, I.., Delas, J., Hajri, H., Gherbi, E., Yaich, R., Boulahia- Coupahia-Cuupees, N., Cuppens, F., and Lamprier, S. Robust deep reinforcement learning through adversar- ial attacks and training: A survey. arXiv preprint arXiv:2403.0m 00420, 2024.
Tan, K. L., Esfandiari, Y., Lee, X. Y., Sarkar, S., et al. Robustifying reinforcement learning agents via action
13

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
space adversarial training. In 2020 American control conference (ACC), pp. 3959-3964, IEEE, 2020.
Ying, C., Zhou, X., Su, H., Yan, D., Chen, N., and Zhu, J. Towards safe reinforcement learning via constraining conditional value-at-risk. In Proceedings of the Interna- tional Joint Conference on Artificial Intelligence (IJCAI), pp. 3673-3680, 2022.
Tanabe, T., Sato, R., Fukuchi, K., Sakuma, J., and Aki- moto, Y. Max-min off-policy actor-critic method focus- ing worst-case robustness model misspecification. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Sys- tems (NeurIPS), 2022.
Zhang, H., Chen, H., Xiao, C., Gowal, S., Stanforth, R., Li, B., Boning, D., and Hsich, C.-J. Towards stable and efficient training of verifiably robust neural networks. In International Conference on Learning Representations (ICLR), 2020a.
Tang, X., Marques, A., Kamalaruban, P., and Bogunovic, I. Adversarially robust decision transformer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
Zhang, H.. Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. Robust deep reinforcement learn- ing against adversarial perturbations on state observa- tions. Advances Neural Information Processing Sys- tems (NeurIPS), 13:21124-211077 2020b.
Tessler, C., Efroni, Y., and Mannor, S. Action robust rein- forcement learning and applications in continuous control. In Proceedings of the 36th International Conference on Machine Learning (ICML), pp. 6215-6224, 6224, PMLR, 2019.
Zhang, H.,, Chen, H., Boning, D. S., and Hsieh, C.-J. Ro- bust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations (ICLR), 2021.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model- based control. In 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pp. 5026-5033. IEEE, 2012.
Zhang, J., Zheng, Y., Zhang, C., Zhao, L.,, Song, L., Zhou, Y., and Bian, J. Robust situational reinforcement learning in face of context disturbances. In International Confer- ence on Machine Learning (ICML), pp. 41973-41989 PMLR, 2023.
Vinitsky, E., Du, Y., Parvate, K., Jang, K., Abbeel, P., and Bayen, A. Robust reinforcement learning using adversar- ial populations. arXiv preprint arXiv:2008.0 01825, 2020.
Vinyals, O., Babuschkin, I., Czameckii W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in stareraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350-3544 2019.
Wurman, P. R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T. J., Capobianco. R., Devlic, A., Eckert, F., Fuchs, F., et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):223-225 228, 2022.
Xu, H., Jiang, L., Li, J., Yang, Z., Wang, Z., Chan, V. W. K., and Zhan, X. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference Leurning Representations (ICLR), 2023.
Xu, K., Shi, 7.,, Zhang, H., Wang. Y., Chang, K.-W., Huang, M.. Kailkhura, B., Lin, X., and Hsich, C.-J. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Yang, Q. Simão, T. D., Tindemans, S. H., and Spaan, M. T. Wesae: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAA/ Con- ference on Artificial Intelligence (AAAI), volume 35, pp. 10639-10646, 2021.
14

---

Off-Police Actor-Critic for Observation Robustness: Virtual Alternative Training
APPENDIX
We describe additional related works, provide more detailed theoretical derivations, offer additional evaluation results, and deeper discussions on the future research direction to help the reader gain a deeper understanding of our work. In the following, we temporarily set aside strict notation and represent expressions like "JuEA " as "LaeA" ." when discussing continuous action space.
A. Additional Related Work
A.1. f-Divergence Constrained Methods.
The use of optimization methods constrained by f-divergence has spanned various contexts from historical applications to recent advancements. Historically, these methods were utilized for relative entropy maximization in inverse RL (Boularias et al., 1). In offline RL and imitation learning (IL), they primarily serve to constrain divergence from the probability density distribution of state-action pairs within the dataset (Xu et al., 2023; Kostrikov et al., 2020; Nachum & Dai, 2020; Kim al., 2021; 2022; Ma et al., 2022). Moreover, methods that limit update intervals by distribution constraints (old policy, prior) during policy updates (Peters et al., 2010; Fox et al., 2016; Schulman et al., 2015; 2017; Belousov & Peters, 2017; 2019) are closely related to our proposed soft-constrained adversary approach.
A.2. Additional Details on Robust RL for Adversarial Dynamics
We additionally discuss robustness in dynamics, particularly focusing on adversarial training due to its diversity and importance.
Broadly, robust RL can be categorized along two axes: (I) the target of uncertainty or perturbation (e.g., observation, dynamics, action, reward), which determines the appropriate modeling framework and optimization strategy; and (2) the treatment of adversariality, which varies depending on theoretical assumptions.
For the second axis, some approaches seek formal guarantees via Nash equilibria or Stackelberg games with fixed update orderings, while others inchuin-ncl our adopt alternating training as a practical approximation. Beyond hard max- min formulations, recent studies have also explored relaxed, risk-sensitive objectives such as entropic risk and value-at-risk. bridging connections to risk-aware MDPs and RLs (Howard & Matheson, 1972; Filar et al., 1989; 1995; Osogami, 2012; Chow & Ghavamzadeh, 2014; Chow et al., 2015; 2018; Ying et al., 2022).
This area has become an active focus of research, with recent surveys aiming to systematically organize and compare diverse formulations and assumptions in robust RL (Moos et al., 2022; Schott et al., 2024).
Adversarial Disturbances To consider the pessimistic dynamics scenario for the complex model-free settings beyond the tabular or linear cases (Nilim & El Ghaoui, 2005; Iyengar, 2005), external forces are introduced to the dynamics model, modified from F(s' | s, a) to Fis' 8, aj, a2). Here aj is the agent action and d2 is the adversarial action, which is assumed to be chosen by an adversary which decides the action from the same observation s.
In the early stages, Morimoto & Doya (2005) proposed robust RI. method that accounts for input disturbances and modeling errors by introducing external adversary and applying robust control method based on / theory (RRL). Pinto et al. (2017) extended this approach by incorporating DRL approximators, specifically TRPO (Schulman et al., 2015), and proposed Robust Adversarial RL (RARL), which enhances robustness against dynamic perturbations from the training environment and external forces. Their approach consists of two RL. components: an on-policy agent RL and an on-policy adversarial RL, resembling the alternating training on state observations (ATLA) (Zhang et al., 2021). Subsequent studies reported that RARL sometimes converges suboptimal solutions and addressed this issue using gradient-based techniques (Kamalaruban et al., 2020). Reddi et al. (2024) proposed novel method, Quantal Adversarial RL (QARL), which mitigates this suboptimality by relaxing the hardmin-hardmax formulation through entropy regularization. Although its formulation and approach resemble VALT-SOFT-SAC. a significant difference remains: QARL still relies on alternating training between the agent and the adversary because it assumes a two-player zero-sum Markov game with two action inputs for the dynamics, F(s') 8, a a2). More recently, Dong et al. (2025) proposed a method that considers an ensemble of adversarial disturbers (typically around 10) instead of a single corst-case adversary, and updates the distribution using divergence-consstne constrained optimization or percentile- criteria.
15

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Interestingly, these methods improve not only robustness adversarial disturbances but also generalization to dynamic variations, such changes in friction or mass.
Dynamies Parameter As another form of dynamics modification, closely related to the previous paragraph, many studies consider modeling the environment as F(s' , a, w), where w represents uncertain physical parameters such as mass or friction coefficients. These works assume the ability to control such parameters in simulation or collect diverse training data across different settings. By explicitly manipulating these parameters, the agent learn be robust to dynamics uncertainty.
One simple approach is Dynamics Randomization (Peng et al., 2018), which maximizes expected performance over a distribution of environment parameters. To avoid local minima, Vinitsky et al. (2020) propose using multiple concurrent adversaries to introduce diverse perturbations. Another line work (Tanabe et al., 2022) incorporates the effect of uncertain parameters into the Q-function, and applies gradient-based optimization to find perturbations that degrade the value function.
Compared to the external-force-basee methods described earlier, these parameter-based methods typically assume that the environment parameters (i.e., w) remain fixed within each episode. This assumption helps avoid overly conservative policies, as the agent can adapt across episodes without facing highly variable dynamics within single rollout.
A.3. Constrained MDP State Observation.
There are similar problem settings that address observation robustness, considering them within the framework of safe Constrained Markov Decision Processes (CMDP) (Altman, 1999). In this setting, the agent is required to maximize the accumulated reward while simultaneously ensuring that separately defined cost measure remains below certain threshold.
Yang et al. (2021) proposed an off-policy method, Worst-Case Soft Actor- Critic (WCSAC), which efficiently solves the problem maintaining the accumulated cost constraint below a predefined limit. Their approach utilizes the Lagrangian- constrained SAC framework (Ha et al., 2021) to enforce cost constraints effectively.
Furthermore, Liu et al. (2023) conducted additional analyses and introduced two novel types of adversarial attacks: one that maximizes the cost and another that maximizes the reward while violating the safety constraints. Notably, the latter attack is particularly stealthy and highly effective in compromising agent safety without significantly altering the reward signal.
While these studies are crucial contributions to the field, our problem setting specifically considers an adversary that directly hinders the maximization of the accumulated reward sum rather than enforcing cost constraints. Thus, we do not include them as direct points of comparison our study.
B. Details of Derivation and Proof
B.1. Symmetric Property of Policy Evaluation
Proof of Theorem 4.6. Recall that Eq. (7) satisfies the 7-contraction property. Thus, can express the following relationship:
[I(T,an (T)(, -TZATTA)))) Un)l =7
FF min Vina, (8441) - min Vina (00 VEN
TEN
5
min Voan, (8(711) min Vonn 2(001))))))) PeN
(15)
VEN
< > [IT QI (8t+1, āetz) llee
here ||... ll,,,, denotes the max norm over st, a. Recall the definitions of Vinan ($1+1) and Valin (st+1):
Viaa (51+1) E, EE [Q (#t+1, d++1)] + Cenn H(e ov) aatt Dy(v p).
(16)
(s1+1) E, [Q" (S141, 8141)] ar.. H( aaus D {v p).
Here, we define two distinct action-value functions: (s) E, (s, and (*(,, = (xl-Q2 We
consider the operation T
16

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
"T==""")((('' 5:) (T+ (8) 8s) (T= (2)
B BeginFor
V=, (8c+1)
V7,2 (octi))
MEN
NEN
E Fo
min Vin, (8(+1) - min NEN
V (8c+1)
DEN
(17)
S 7
min V MEX 1 - min Von MEN
Vo (0++1)
- > 11 (8-11, as-1) Q2 (st+1: ae+1) 1
We substitute Qi and Q2 and apply the relationship Vy n, (8++1) = -Va wim _ (8e+.) in step (1).
=
Or more simply, we can confirm the same result by using the result, (Tx Q.) En [- Q3)] then the following relationship holds:
(T Qi) (Tx Q2)
Qi)l.... se.de
||xx I-T QT. E. I-T- Q21
12 1l.. 11112 -2' 2041
(18)
By applying Tha infinitely many times, the action-value functions (s, a) and Q2 (s, a) converge to the same fixed point at rate governed by 7 (0.0, 1.0). Consequently, (s, and (5 6) also converge the same fixed point under the same policy.
To prove the symmetry of the value function, we demonstrate the equivalence of the fixed points:
Q- (s, (s, E, Quaa (s, ā]]
(19)
To verify this, we define the action-value function for the adversary as ( 8) = E, Q-Q (s, ā)], and action-valus for the agent, Q2 (s, a). We then compare the differences between them after applying each operation:
[TT (s)(s, 5c) - E. [- Q2) a+))
c(ss Re) yEFor max Val (#t+1) + E- (r()),a)) + yEF [nij Vina ((s(()))))))))))
(1)
Exor max Vili (8++1) + min Vyan 200 m9
?
S > max Vi,1 (se+1) + min Vinn ((0+1))))+) 2442
Fp:16 ? max E. IQ Hatch H Qutek Dy} + min Vian ((0)+1) m4t1
(20)
max [E+ ( de+1)] - anuu 71 awer Dyl + min Vona 2(
(2)
7||min min Vinn (8 min Viln 20
7
Eq.(10)
5441
|1 Q (8), āt) (Tyna Q5) (s),ae)
> (IQ a++1) - -5 (
We use the definition of c(st, &z) in step (1), the existence condition in step (2), and the y-contraction condition of the theorem in step (3). Thus, each stationary point must satisfy Eq. (19). Finally, substituting Eq. (19) into Eq. (16), we obtain the result:
Ve (st)
max 122h (st) min V DEN
you (5t) - -i (8t).
(21)
17

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
As assumed in the theorem, the adversary's action-value function is expressed the expectation of certain action-value function with respect the agent's policy: Q" 8) E= a)]- While some readers might find this assumption restrictive, it is crucial note that the desired soft optimal adversary always satisfies Eq. (19). This implies that, by deriving the soft optimal adversary's 's policy from the agent's 's action-value function, we are effectively learning within a constrained set of functions that include the desired solution, rather than searching over the entire arbitrary set of adversary action-value functions,
B.2. VALT-SOFT: Derivation and Proofs
Derivation of Lemma 4.7. In this paragraph, we describe derivation of the analytical solution of Eq. (5) as in Eq. (9). Now we consider the case of the KL-divergence as follows:
maximize <(8]a)
Vini (s) maximize =(8(s)
Σ RES
[i(ss)) Σ (3,)āā
aent Σ
v(5ls) Σ
{aa3) (-
log ক (a(3))
aeA
EES
aeA
wa.1
Ev(5(s)
logus log (((s)
Oatk
aES
(22)
Σ SES
Uzdza + Cent
Σ Vs
Σ
Ta.s log Na,s
Outth
Σ
Us (log us logps) = g(v)
SES
aEA
sE$
subject to
VB S
Vs 2 0,
Σ
1/3 1,
8ES
The Lagrangian function for this problem is given by" 3:
(v, + A(1
Evs). vz).
(23)
FES
where À is the Lagrange multiplier associated with the equality constraint. By rolling the Karush- Kuhn-Tuchee (KKT)'s stationary condition:
VeL(U, A) =
+ q3 + aent
Σ
Ta log Ta.s aatth aatt vs log + I3) - A = 0
ae.À
-Ha,
(24)
Qa aent Ha. X
log V/3 llo Ig
aattk
+ 1/3 = pa exp( 9: aent Ha,5 exp( - -D/aasth 18).
Cattk
From the equality condition, we can decide X, then:
(soft( s) p( s) exp (E, (+ aent H(T)) /aask)
1/ soft
(25)
where the partition function. This is the result of the Lemma 4.7.
To prepare the next paragraph, give this back into the objective (Eq. (22)), then we derive: Ex | (T, (s,ā)) - aene H(T)) dš.
Vr (8) Qattk log I p( 1)) exp J
(26)
Otattk
We abbreviate inequality constraint hecause can he vanished.
18

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Proof of Contraction Theorem 4.8. Recall the definition of the soft worst policy evaluation in Eq. (7), substitute the results Eq. (26) in the previous paragraph,
(Tyan (se, ae) = =r(st,ā)) + BBx min V Vean (cmn(set)))
WEN
s(+1) exp Ex 2++1))++1 Gent H(T))
min Vian ($(+1)= = Virun ($t+1) Clttck log vEN
p(5
d3
1118
Qattk
((Q5)
(27)
To represent the dependency of V Q we define f(Qz. S1+1) for simplifying notation.
We assume there are two different action-valus functions, Q (st, ā,) and (5, ā.). We abbreviate the small character and simply show Q1, Q2 as the following.
use the same metric as G-learning (Fox et al., 2016) and Soft Q-learning (Haarnoja et al., 2017), we define e
||Qi (st, āe) Q2( az) az)lls....
Since f(, 8p) is monotonically increasing function for Q, then can say:
S(Q1, 8+) f(Q2 e, 8c)
=e+ ((Q,
(28)
= =1Q1 Q21l + s() si)
++ 8() - ((Q2,8t) S ||Q1 -Q2lla,a
In the same way:
f(Q1, st) 2ʃ(Q2 - 6,) f(Q2(1))
(29)
+ f(Q1,s) f(Q2, s:) 2 - Q2lloaae
Therefore, from the both inequalities, derive:
(lf(Q1, (Q), - (sq ā.) - Q2(st, āz ae)ll
(30)
Then, consider difference between the two action-value functions after our Bellman operator:
|l (T,Tnn (st, (T,tnQ2) (St, āz) l.)
f(Q2, st+1)] llea, < < - f(Q2, $141)l
(31)
$ (4 - Q2(8++1, 0 (1+1)) alse
(30)
Therefore, we can say T is contraction operator, because we perform this operation an infinite number of times, Q1 and Q2 converge to fixed point.
B.3. VALT-EPS: Derivation and Proofs
Derivation and Approximation of VALT-EPS. If we continue to keep the f-divergence during the derivation, we can obtain a broader perspective about the relationship between choice of the divergence and the corresponding soft worst-case attack. This approach is inspired by (Belousov & Peters, 2017; 2019), which provides more detail about the derivation and related perspectives on policy improvement.
f-divergence is measurements between two distribution and defined as:
(āls))
Dy(1 (lp) & [((
p(3]s)
(s is given).
(32)
RES
Here, f() Ω R is convex function with the properties, Range(f) (0, oc) and f(x') = x' 1. An a-divergence a sub-family of the f-divergence that is defined as:
fo (x) (x" 1) aax 1)
(33)
ala 1)
19

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
The a-divergence includes many popular divergence: for example, the case, a O, results in Reverse- - derive KL--divergence, and 2 occurs (Pearson's s) -ivvieereecce. To solve dual problems, the convex conjugate function (y) for fz) is known to be useful. This is defined as:
1" (v) SuPxedom(f)) (1)5)
(34)
where (5-) denotes the inner dot product. If f and f are differentiable (Assumption 4.2), by taking the derivative of Eq. (34), (v) xx and (v( - [(x)) lemar O, we obtain the property (f*)² = (f")-.
For the a-divergence case, its conjugate function and the derivative function of the conjugate are:
(a( _ (1+ (a 1)y) - a
(35)
(F)) (1 (a 1)y) for (1 - aly 1.
(36)
To consider the case described in Eq. (7), we can rewrite the problem as:
maximize ((][s)
Woin (s)
maximize ((](s)
Σ (((())
(-(
Otent
Ev(E(s)
Σ
(a(s)
log (ū]s))
8ES
aES
âEA
Na.i
qF
Qattk
p(8
8ES
75
Σ
V348 + aent
Σ V/3 Σ
T log Ta.
Qattk
Σ
Pafa
= g(v)
Pa
SES
RES
uEA
3ES
M(sr)
subject
V5 E S,
V 2 0.
Σ V/3 1,
BES
(37)
The Lagrangian for this problem is given by:
L(v,
+ A(1 x(1-Ex)) Σ
Evi)+
KgV/s,
EES
with the KKT's condition:
Vug(v) AI K; = O,
(38)
V; 2
K31/3 0, 2 0.
Here, N; = (=((s) represents the complementary slackness for the inequality constraint. From the stationarity condition:
VB ^Ig K = Ps
q= Denn H((x)
Qutth
ys aent H - AI; + Ki
(39)
15
Catti
q3 aent - Als + K3g
= P=((()
Cattk
20

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Then, using the property, where - derive the optimal Ug as:
(s.5)
Ex (5, aent H(T)) X la k((ls)
(*(8))) - (((ss
(40)
Oattk
This result represents Lemma 4.9.
This solution can be regarded as how to asign probability mass on the prior distribution p(š]s) and it depends on n, Qui Centt Cottk- and a. If we assume only a < 1 case, from Eq. (36), the term of (fa)' must hold > 0 (due to its exponantial term), then p[8]s)
< v² ($]s) > holds. From the complementary condition in Eq. (38), in this case, the slackness parameter k(s}8 must be O.
From the constraint in Eq. (36), we get the condition for X:
V3 S, x (s, Oottk -
(41)
This inequality must hold for all 8, thus in the case of maximum of the right term. Then, we define:
x = max: (- (s, Cattk - + Eo:
(42)
where Ea is a residual that satisfies Ea > 0. Considering the case a + -o0, the optimal value of constraint term approach the bound, X > max maxppp)pppp) (- (s,88)) Therefore, Ea approaches to zero as a + -oo. Substituting this back into Eq. (40) and setting E(V";;) 5) = max (-V*(s, 8)) - (-V*(s,3)) 2 0, then we can represent as: -V* A = <.V"; + Clattk I-o E.. We can consider the cases where a < 0 as follows:
<(V";8) + Olattk T-a Eax
v² (5(s) (ff))
Qattk
p((ss)
(Ea + E(V":8))
(43)
Qattk
Qattk
1-a
p(]ss)
1- a E. + {(V*35)
From this equation, we can determine that y² (3]s) exhibits a strong peak at E(V";5) = 0, which corresponds to := arg max(-Vo 8)) arg min // (s, also shows mild probability mass for the other perturbation states due the $Esupp(p)
#Esupp(p)
term of exponential, 0 < -La <1
Now, we assume the prior p())s) is the uniform distribution over L.o norm constrained range. Then, approximate the peak of the probability by constant multiple of Dirac's 's delta function as 5 and distribute the remaining probability equally as 1 Kwarst. We represent this approximation as:
Kworst
S.
if arg min (*
n soft (5]s)
PCB.
(44)
1 Kyont
others
S
Proof of Contraction Theorem 4.10. First, We confirm the soft worst-case Bellman operator in Eq. (7) for the case of Eq. (44).
(T ()(, Ūz) Er(s, ā+) + EB [Enen V (8 81+1)
(45)
=r(st, ā+) + TB,
Rowesd V (S 874 1 + Kiwors EB ppp(- [V (8t+1, 8 Sc41]] ;
V
21

---

Off- Pol-oolicy Actor-Critic for Observation Robustness: Virtual Alternative Training
where we
8* arg min V (s, arg min (,(s(s, a)] O'ent (T)). B.
(46)
BEB,
Then, we consider difference between two action-value functions, Q1, Q2, after using the Bellman operator: [I poi Q1)(s, at) - Q2) de)l
11 woroortt + (1 K) p,l AV worst,2 - (1 x) 22 Ila
- >/ worst, + (1 - K p.l - k VV worst,2 - (1 - k) p.2 llett
X 7/0 | worss,,1 V worst, | + 7 - K) p.j 1p2llos
< >x|[V Worstt11 V Worrst,2llees + 7(1 x) -Q2llee1aren1
(47)
(1)
- 7kl1Q1 - Q2llecnidens + + - k3]1Q1 - Q21lestiāee
(2)
= (#L+1, @e+1) Q2(#1+1; 1.
For the inequality (1), we cancel entropy terms under the same policy and perturbation, then use inequality for u++1- For the inequality (2), similarly to the WocaR case (Liang et al., 2022), we utilized:
min
A min B s max A - B
(48)
aEB
SEB
8EB
and canceling entropy terms due the same fixed agent policy.
Therefore, say puan. for the case of Eq. (12), is contraction operator, because we perform this operation an infinite
number of times, Q1 and Q2 converge to fixed point.
B.4. Proof of Policy Improvement with Fixed Adversary
Here, present the policy improvement theorem (Sutton & Barto, 1998) under a fixed adversary While focus on the case of the maximum entropy scheme (Haarnoja et al., 201 8a;b) in this study, other cases, such as deterministic policy gradient (DDPG) (Lillicrap et al., 2016; Fujimoto et al., 2018), can be similarly applied as Tnew + arg max E., [Q7 (3, (s),, (B))].
x
For simplicity, we omit the entropy coefficient Oen in this discussion.
Proof.
Given a fixed adversary u and policy म, define a new policy म such that for all states 8, If we extract a new policy स by using т and the corresponding action-value function QT, a following equation holds:
E (H([((111)) + Ea-z (x, < E + Ea- (x((s,
(49)
+ H(Tov((1s) + (Q] S H(# o u(-1s)) Ea- (T((s,
(50)
This is proven by:
DRi(ToN((O)) || Rov(-ss)) Σ π Tor (()s) (log v(āss) log ã o v(āls)
aeA
--((TOv(-1s))
Σ
(logoo ((āls)
aed
-H((TOu(-ss) - Σ Tor(āls) (3( - log Σ exp((
(51)
aeA
a'EA
-H(T o (()-s)) - Ea-rou(11)) [2, []]]]]]]] + log Σ exp (s,)))
a'EA
-ILHS of (50) + |RHS of (50) 2 0.
In the third equality, we used the result defined in Theorem 4.12:
exp Q7 (s, ū)
A o v(āls) = Even exp (x,
(52)
22

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
By using this result, we can confirm:
(st @+) r(st, ae) + /Eecting BByyrwy Ba oo Q (S(+1, u++1)) H(( (-65(()))))))))))))))))())))))))
- r(5r, @t) 7Betti (Bar4 Ba Q (5++1 0c+1)] + H( ( |c+1
rollout
(53)
- (se,
Then, under fixed adversary v, we can improve our policy л by targeting Eq. (52).
In summary, we have shown that the agent's soft worst-case action-value function can be learned. By leveraging the symmetry properties of the adversary with respect to the agent, the adversary's 's (soft) optimal policy be obtained without explicitly conducting separate RL process for the adversary. By utilizing such stabilized and fixed soft optimal policies, robust policy improvement for the agent can be achieved.
C. Practical Implementation for VALT and SAC-PPO
Algorithm 1 Framework of SAC Training with Adversary
Initialize agent critic (s, a) and actor (s)
2: Initialize agent target networks Qæ; (s, a.) by setting 0/,2 01,2
3: Initialize a replay buffer R <- and entropy coefficient C'ent 6- 1.0
4: (I) Initialize the parameters for adversary, yoon if needed
5: for t = 1 to T do
6: (2) Execute action O1 behavior s1) observe (St: Tr, St11, and store in R
7: Sample mini-batch of M transitions (st, oi, rř, done)) ~ R.
8:
(3) With s update the critic parameter 01,2 by minimizing the Huber-Loss in Eq. (54)
9:
(4) With soft update the actor parameter & by maximizing the policy objective in Eq. (56)
Update Entropy Coefficient Geet by minimizing the loss:
[(Cent) - M E. M Ctent (Hearget Hourrent)
Hi Heurrent log To (a((())
11:
(5) Update the parameters for adversary, soft needed
Post Processing:
12:
Soft update the target networks: 01 + (1-T)0 + 101,2
13:
Update the scheduled parameters
14:
end for
In this section, we provide additional details about the implementation of our VALT-EPS-SAL VALT-SOFT-SAC, and the comparison method, SAC-PPO. All three methods incorporate the adversary during the Actor-Critic update processes. Thus, we first describe the common notation for policy evaluation, policy improvement the following two paragraphs, and basical procedure by the pseudo-code. Subsequently, explain the specific processes unique each method.
Policy Evaluation with Adversary. As discussed in Section 4.2, replace the original SAC policy evaluation process with the soft worst-case Bellman update. The practical procedure for policy evaluation is described as follows:
st, al, ri, sim done;; R, = 1.2,..., M,
M
L(01,2)
LHu (Qui af), min (y (6/+1): (2 (2(+1)))) Σ
M
(54)
v1.2 (5(+1) = re 7(1 done)) V ($t41)
Vu
( Eyonn
Ex [00 (8/+1) a(41) Cent log no (at+1 [5++1]]
where ~ R indicates sampling M trajectory tuples from the replay buffer R. done represents a binary flag indicating the end of the episode. T. denotes the Huber loss function, Q0 and Qø, represent the action-value function parameterized
23

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
by 01 02, and their target networks. T represents the agent policy function parameterized by Φ.. yÿ represents the function for the target value, and V denotes the value estimation for the agent under the corresponding (soft) optimal adversary yooft
Note that so is defined as the approximated solution to the following term:
min Ey [I] + aeen (((TTTTTOO) + aauu p). vEB
(55)
Policy Improvement with Adversary. As discussed in Section 4.4, replace the original SAC policy improvement with the policy improvement using soft-worst but fixed adversary. The practical procedure are follows:
R, i= 1.2..... M, 8³ r. (. 1s)),
ā ~ T (-| 8*)
M
(Ø)
( 200 (8" at) Cent log Np(ā" (z¹)
(56)
M
==1
M
1
max (TI(TT( (- Tal. 183))
J Jreg (5) Junal (6) (0) (reg(
M
S.EB.
where (o) denotes the total objective for the agent's 's policy, (M) refers to the objective derived from the (SA-)MDO MMD, (Ø) represents the policy regularization term proposed in Zhang et al. (2020b), and Kreg is the coefficient that controls the strength of the regularization.
Note that y soft is defined as in Eq. (55).
Procedure of SAC Training with Adversary. We summarize Algorithm to illustrate the common framework for SAC training with an adversary, as proposed. The processes that differ from the original SAC are colored as follows: Violet colors indicate modifications required for all three methods, and brown colors indicate modifications specific to adversary learning methods (VALT-SOFT and SAC-PPO).
In the following subsections, we provide details on the highlighted processes unique to each method, with particular focus on the implementation of ynu
C.1. Implementation Details for VALT-EPS-SAC
VALT-EPS-SAC approximates soft using distribution that combines uniform random samples and worst-case samples for the agent, weighted by the worst-case rate Kworst- We use PGD to generate the worst-case samples, eliminating the need for additional learning parameters or corresponding procedures (brown parts in Algorithm 1).
We illustrate Algorithm for this sampling process. Since SAC employs two critie networks, we use the mean of their outputs, denoted as Q2, . for PGD. We confirm that two gradient steps with randomized initial step achieve good balance between computational efficiency and robustness performance. However, this process is computationally intensive and requires GPU calculations, we use uniform distribution (2) in Algorithm I and apply (3) and (4) to incorporate the PGD outcomes.
C.2. Implementation Details for VALT-SOFT-SAC
VALT-SOFT-SAL approximates the corresponding adversary, Sooft using parameterized policy network. Thus, we define "Initialize adversary actor umodel (5 s)" in (1) of Algorithm 1. By utilizing this network, we can efficiently calculate the violet parts (2), (3), and (4) in Algorithm with computationally lightweight process.
For learning the adversary in (5), we apply the policy improvement defined in Eq. (10). The detailed implementation is as follows:
24

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Algorithm 2 VALT-EPS-SAC Adversary Distribution Procedure
Input: State s, agent policy x (a]s), action-value function Q7, (s, a), worst-case rate Kworsts attack scale e
Output: Epsilon- worst distribution p' s)
1: Function VALT-EPS--AC--DISTRII
म, Q6 Niwrest; 6)
2: Compute the worst-case state using PGD (minQ attack):
3: for = to iter iter_steps sttps do
4: Snew t proj
3
VV Fold Por, (((
5: end for
6: (5 () ((5)) + (1 Fowars ({ 6, c)
7: (where is the step size, ps is the mean action oulput of policy n. and 8 denotes the Dirac delta function)
8: return pf
R, i= 1=1,2, 1.......,,A M,
z² model 8²),
ā' ~ T.(-) 8'),
(57)
M ((b)= M
Clotkk log model
(s)) + min Qe, (s',, (3t
16,,a" ā²) Gent log (ā' 8*)
1.2
There is some arbitrariness in the frequency of updates and the number of iterations. We found that updating once per environmental interaction step works well, although slightly more stable results can be achieved by resetting every few environment steps and re-learning from the initial parameters.
We hypothesize that the adversary policy explores the agent's action-value function, and periodic resetting helps prevent the adversarial policy from converging a local minimum. This sub process for (5) in Algorithm is detailed in Algorithm 3.
Algorithm 3 VALT-SOFT-SAC Adversary Learning Procedure
Require: Environmental step I, number of learning steps per environment step: num_sleps
if (t mod num num__steps) steps) 0 then
2: Reset the adversary policy u moodel
3: for = to nwm_steps do
4: Optimize the adversary policy by minimizing the loss in Eq. (57)
5: end for
6: else
7: Do nothing (pass)
8: end if
C.3. Implementation Details for SAC-PPO
This method is based on adversarial learning and resembles VALT-SOFT-SAC described in the previous subsection. At step (I), the parameters of the PPO adersary used for RL are initialized. Steps (2), (3), and (4) involve sampling from this adversarial model. However, the key difference lies in step (5), where the adversary is trained through on-policy learning using PPO with interactions from the environment.
To balance the training of the adversary and the agent, we adopted the original adversary settings proposed in Zhang et al. (2021). Since PPO trains on mini-batch basis, the SAC training steps for the agent were evenly divided to alternate training between the PPO adversary and the SAC agent.
For the adversarial training of PPO, please refer to Algorithm in ATLA-PPO (Zhang et al., 2021), as is implemented exactly as described in the original paper.
25

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Table 2. Common SAC Hyperparameter Settings
Hyperparameter
Value
Optimizer
Adam (Kingma & Ba, 2015)
Learning Rate(Actor, Critic, O'ent)
0.0003
Discount Factor (7)
0.99
Target Smoothing Coefficient (7)
0.005
Optimization Interval per Steps
Number of Nodes
256
Number of Hidden Layers
Activation Function
ReLU
Prioritized Experience Replay (3per)
start from to 1.0 at the end
Replay Buffer Size
1,000,000 256 Auto-tuning
Batch Size
Temperature Parameter (Deent)
Target Entropy (Hsarget)
-im AA,, only for Hopper-v2: Hzerget 0.2 state normalizer only
Normalizer
D. Settings and Hyperparameters
In this section, we describe the experimental settings and hyperparameters used Section 5. The contents are divided into three subsections: training for off-policy methods, training for on-policy methods, and evaluation (attackers).
D.1. Settings for On-Policy Methods
use the code and settings provided by the original authors. For a fair comparison of robustness with SAC variants, we adopt the LSTM setting if is available in the implementation.
PPO Settings. We use the same settings and parameters those provided in Zhang al. (2021).
Robust-PPO Settings. We use the settings and parameters of PPO (+LSTM) provided in Zhang et al. (2021) and tune the regularization terms. We assume that our evaluation results for Robust-PPO appear more robust than those reported in (Zhang et al., 2020b; 2021; Liang et al., 2022), likely due to the inclusion of LSTM.
ATLA-PPO Settings. We use the same settings and parameters as those provided in Zhang et al. (2021).
PA-ATLA-PPO Settings. We use the code provided Sun et al. (2022) and tune the entropy coefficient and the adversary's hyperparameters as described their paper.
WoC Settings. We the code provided in Liang et al. (2022) and the hyperparameters as described in their paper. We used the non-LSTM settings.
D.2. Settings for Off-Policy Methods
We use SAC (Haarnoja et al., 2018a;b) the base off-policy algorithm due to its good sample efficiency, high performance. and theoritical soundness. All variants, including original SAC (SAC), Robust-SAC, SAC-PPO, WocaR-SAC, and our VALT-EPS-SAC and VALT-SOFT-SAC generally adhere to the settings and parameters outlined in the next paragraph, except for their specific adjustments.
Common SAC Settings: A Unified Basis. in our experiments. While most configurations follow the second SAC paper (Haamnja et al., 2018b), we made several adjustments to ensure stability across tasks:
The following settings form the common foundation for all SAC variants
26

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Training steps: 1M for HalfCheetah, 0.5M for Hopper, 1.5M for Walker2d, and 3M for Ant. For SAC-PPO, the SAC agent's training steps remain consistent, but additional interaction steps required for the PPO adversary.
Modifications for stability: Specific hyperparameter adjustments were made improve stability (see Table 2 for details).
Table 2 lists all hyperparameters, with underlined parts highlighting the modifications. Detailed explanations for these adjustments are provided below.
Target Entropy for Hopper-v2: We found that the target entropy Hem -im |A| = -3, as used for Hopper-v2 in the original SAC (Haarnoja et al., 201 8b), is too low to get stable final results. Consequently, scores during training oscillated drastically, phenomenon observed not only in our SAC implementation but also in a well-known open-source implementation (Raffin et al., 2021). Though a fixed temperature parameter O'ent. 0.2, as the first SAC paper (Haarnoja et al., 2018a) or decreasing learning rate for aeent also work well, we decide to tune the target entropy only for Hopper-v2 to keep stable final results.
Prioritized Experience Replay We use Prioritized Experience Replay (PER) (Schaul al., 2016) to accelerate learning outcomes. While its effect is only slightly better in some tasks, do not observe any disadvantages to using PER. Therefore, we decide to continue to use this method.
Normalizer We normalize state inputs by recording running statistics the agent receives information from environ- ments. Subsequently, when the SAC agent samples a mini-batch for learning or calculating output from observation inputs, we use the state normalizer to set the mean and standard deviation to 0.0 and 1.0, respectively. We attempt to normalize rewards by episodic returns as PPO (Zhang et al., 2020b) and other recent on-policy methods (Zhang et al., 2021; Oikarinen al., 2021; Sun et al., 2022; Liang al., 2022) did. However, we observe that the agent's learning became critically slow in HalfCheetah due delays in updating the running statistics. Therefore, we decided to abandon this normalizer.
In the following paragraphs, describe the specific settings required for each method.
Robust-SAC Settings. Originally, the robust regularizer method (Zhang et al., 2020b) was implemented for PPO, A2C, DDPG, and DQN. We extended this approach to SAC, resulting in the Robust-SAC variant, by carefully integrating key elements from both PPO and DDPG implementations.
Robust-SAC incorporates action consistency terms into the policy update to address scenarios where attackers manipulate observations to induce actions that deviate from the original ones. The loss function for the policy is given as:
[(T)= = Bg-D() Benx [aent (als) - Q* a)] + Kreg max DEL (TT(|s) x x-113)
(58)
sEB.,
where Kreg is coefficient balancing the original SAC loss and action consistency.
To solve the maximization term in Eq. (58), we evaluated two approaches: convex relaxation and Stochastic Gradient Langevin Dynamics (SGLD). We opted for SGLD due to its superior performance as observed in Robust-PPO (Zhang et al., 2020b).
Configurations were adapted by extracting the core ideas from DDPG and PPO variants. The attack scale increases linearly from the final target value, starting at = 1.0 10 and reaching the target at training-stepp - remain-steps. The values for remain-steps were as 2.0 10a. 1.0 10 2.0 10a, and 3.0- 10 for Hopper, HalfCheetah, Walker2d, and Ant, respectively. Additionally, five gradient steps were used in each calculation as in Robust-DDPG implementation (Zhang et al., 2020b).
We conducted a comprehensive search for Kreg values among (1,3, 10, 30, 100, 300}. Except for HalfCheetah, we identified effective parameters that achieve a balance between performance both unperturbed and adversarial settings. However, in HalfCheetah, a trade-off between robustness and performance in attack- free evaluations was observed, as reported in (Liang al., 2022). Even in this case, stronger regularization beyond our selected settings resulted in training instability and
27

---

Off- Oft-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Table 3. Hyperparameters for Robust-SAC.
Environment
Env. steps
Agent Ir
Ent. target
Reg, coeff.
SGLD iter.
Other settings
HalfCheetah
1.0M
3e-4
300.0
Linear schedule; attack scale for seg. lerm from 0.0 0.15 (0. (0..M)).
Hopper
0.5M
3e-4
300.0
|| Linear schedule: attack scale for reg. term from to 0.075 (0. (0-0M-).4M)
Walker2d
1.5M
3e-4
-dimi-A A|]
10.0
Linear schodule: attack scale for reg. term 0.05 (0. MM-..M)
Ant
3.0M
3o-4
di
30.0
Linear schodule: attack scale for reg. term 0.15 (0. (M-27MM)
Table 4. Hyperparameters for SAC-PPO. The second column from the left indicates the number of environment interaction steps used for training the SAC agent. The column PPO txsteps' shows the number of additional environment interaction steps used for training the PPO adversary. on top of those used for the SAC agent. We applied the PA-AD technique (Sun et al., 2022) only in the Ant task.
Envinament
Fin.
Agunt
Far, tarpe
Reg. cufff
SGLD Itr..
Adc ratio
Ads.
Adv. Vol Ads. Fat. coeff.
Other settings
Halft Theeth
1.M
Test
-din 4|
7068 x 3641
0.5
3--5
Te-5
led
1 Liear schedate: snic wale for rog ter fioe 00 15 (n IM-O SM)
Hopper
0.5
3e44
02
3000
2004 x 2441
03
36.5
36.
163
Liteer schedole stick solle for sgg toon from 80 0.05 (0.1M
30M
-Jin.A)
300
2008
led
165
Linear schetale: arack
0011 (M-MM-).7M)
degraded performance in noise-free environments. Furthermore, we found that relying solely on regularization does not improve performance against strong adversarial attacks.
Table 3 summarizes the hyperparameter settings.
SAC-PPO Settings. We introduce a novel off off-polliy alternative training framework, SAC-PPO, by incorporating the ATLA (Zhang et al., 2021) and PA-ATLA (Sun et al., 2022) implementations into our SAC framework. The SAC (agent) component retains the same settings as the base SAC, while the PPO (adversary) component follows the adversary settings of ATLA-PPO (Zhang et al., 2021). The SAC training steps (i.c., environmental interactions) are evenly divided by the PPO learning iterations to balance agent training and adversary training.
To enhance robustness, we integrate robust regularization methods (Zhang et al., 2020b), as reported in other studies (Zhang et al., 2021; Sun et al., 2022; Liang et al., 2022). Regularization settings are identical to those described in the previous paragraph, except for the use of two gradient steps improve computational efficiency.
Therefore, the tunable parameters include those for the PPO adversary (policy learning rate, value-function learning rate, and entropy coefficient) and the coefficient of the regularizer. We perform a grid search for these parameters and adopt the best configuration.
The behavior policy for collecting training samples was set such that the adversary is always present, as defined in ATLA (Zhang et al., 2021). However, this setting caused overly pessimistic behavior, hindering learning progress (especially in Ant). On the other hand, completely excluding the adversary led to overly optimistic behavior. Therefore, we adopted a 50:50 ratio incorporate information from both cases, ensuring stable learning and robustness.
Table 4 summarizes the hyperparameter settings.
WocaR-SAC Settings. WocaR (Liang et al., 2022) is originally implemented only for PPO, A2C, and DQN. We have adapted the essence of this method into our SAC implementation. In WocaR, the agent learns an additional action-value function and its target network, which estimate the worst-case scenario induced by the policy's 's action. This action-value function for the worst-case scenario is represented as the Bellman equation associated with the Worst-Case Bellman operator T as:
(IQ) at) = r(8,, 0t) + B min ((F+)1,((E-1)))
(59)
To calculate the next worst-case action value during updates practically, they calculate action bounds using convex relaxation (IBP) (Gowal et al., 2018) and determine the worst action using PGD within these bounds. We employ auto_LiRPA (Xu et al., 2020) for the convex relaxation and perform five iterations of PGD to ensure efficient efficient computation within realistic time constraints.
During policy improvement, they enhance the gradient ascent objective of the policy by incorporating the worst-case action value, aiming to balance performance under mttack-free evaluation with robustness the worst-case attack scenario. For our
28

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
HalfCheetah
Hopper
Walker2d
Ant
12500
4000
8000
Avrage Episodic Return
6000
10000
3000
6000
7500
4000
4000
2000
5000
2000
2000
2500
1000
0
0
-2000
WocaR SAC
0.00
0.25
0.50
0.75
1.00
0.0
0.5
1.0
1.5
2
3
1e6
Env. Steps
Figure 3. Learning curves of WocaR- SAC for four MuJoCo control tasks under no-attack settings. The solid lines represent the average evaluation scores, and shaded areas indicate standard deviations across different seeds. While performs well during the initial training phase, as the attack scale increases, training deteriorates across all four tasks. During training, we utilize the WocaR action-value function for deterministic gradients, set Nwans = 0.4, and apply tighter relaxation methods (Zhang et al., 2020a) than IBP.
SAC implementation, we implement two methods: mixing the worst action value with the analytical target value (soft-max of the Q) as:
[(T)- = D..)(.) [Bawa [e log π (als) ((1 ( worstt QQT a.) + Aworet (s,
(60)
combining DDPG's policy loss with the worst action-value function by using the mean of the action, ju(s), as:
[(TT) De) [(1 Kiworst Eeg [Otens log (als) Q" (s, a)] Q" Q [s)))
(61)
To mitigate poor exploration and pessimistic learning behaviors, they schedule both the attack scale and the mixture rate of the worst-case objective throughout the policy optimization process. We initiate the attack scale 0.0 and linearly remain- steps, maintaining this value until the end of the training. The values for remain-steps were set as 2.0 10", 1.0 . 105, 2.0 10", and 3.0 10 for HalfCheetah, Hopper, Walker2d, and Ant, respectively. Similarly, we schedule Kworst to increase from 0.0 to the final target value, testing among {0.2, 0.4, 0.6).
increase it to the final scale by t = training-steps
Both methods in Eq. (60) and Eq. (61) learn well in smaller tasks (Pendulum, InvertedPendullm) and the initial stages of the tasks. However, as the attack scale increases, the performance drastically declines, leading to the collapse of training, even when we use more tighter bound method (Zhang et al., 2020a) than IBP used in WocaR (See Fig. 3). We hypothesize that this is because the policy fails to learn from the worst-case scenarios due to the absence of adversaries during policy improvement. Consequently, the policy cannot produce valid actions for the next target action value, causing the worst action value to estimate increasingly poorer values as the attack scale enlarges.
similar phenomenon is observed in the ablation experiment of VAL.-EPS-SSCC in Ant-v2, which lacks an adversary during policy improvement.
Therefore, our findings indicate that WocaR is an effective technique for on-policy methods, as it utilizes gradient ascent target combining normal PPO's value and WocaR's ' action-value. However, a different approach is required to achieve policy robustness in off-policy settings.
Considering the opposite case, we were to adapt the VALT technique to on-policy methods, we would face the following challenge: since the VALT technique can only estimate off- off-policy (soft) worst-case value estimations, we would need to either abandon multi-step rollouts for the agent and perform optimization in a single step, or combine PPO's multi-step estimations with VALT's (soft) worst-case value estimation. In the latter case, its implementation may resemble WocaR-PPO. Thus, WocaR's 's worst-case ratio Kworrs be interpreted as analogous to VALT-EPS's coefficient, which represents the strength of the soft-constrained term.
VALT-EPS-SAL SAC combines uniform perturbations with an approximation of the worst attack using the mixture coefficient fowors-- We employ PGD to approximate the worst attack, similarly to the approach used in Pattanaik et al. (2018): Zhang et al. (2020b). Since estimating the worst-case states that consider the entropy term of SAC complicated and has concerns of unstable learning, we employ two approximations: (1) using the mean of the action
VALT-EPS-SAC Settings.
29

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Table Hyperparameters for VALT-EPS-SAC. Our method requires fewer hyperparameters than ATLAA as we do explicitly train an adversary policy or value function. Additional parameters, such as the ad adversarial ratio and Reporst scheduling, are less sensitive to tuning.
Faviromment
Ein, strpe
Agrat Ir
Eni, target
|| Reg coeff.
SGT D lier.
Ads. ratio
Other wettings
HattCheetah
LUM
3e-4
dimi
||
300
2
0.5
Linear schedule: atack scale for tem from 0.0 to 0.15 (0.IM 10.1M-2.800 (.8M), Aisaraei From 0.0 1.9 (084 (1.34)
Hopper
0.3M
3e-4
0.2
|
30
2
0.5
Lineur schedile: attick scale for rep. lemm from OUD to 0.075 (0.1M-0,4M), Proverai Doan 0.0 to 1.0 (0M) (.4M)
Walker2d
15M
36-4
dimiAl
snmeorr-cheesle--ttteee seale fer semm (ammm------0-00000 (aseee0----------- 1-MM, from 0.0 to 1.0 (0M-1.3M)
Ant
30M
36-4
dimlAl
II
300
2
Linear schedule: attick for semm 0.0 0.15 IM 7M), from 0.0 1.0 (0M 2.7M)
outputs, and (2) ignoring the entropy term. As the SAC-based algorithm's 's Critic comprises two networks, we average their outputs for gradient calculations.
To balance computational resources and performance, we determine that two gradient steps with a random start during PGD suffice for methods. To mitigate poor exploration during the initial training phase, we start Kworst = 0 and linearly increase it to 1.0. by training-sted steps remain-steps, maintaining this value until the end of the training. The values for remain-steps were set as 2.0 - 10", 1.0 10 2.0 10", and 3.0 - 10 for HalfCheetah, Hopper, Walker2d, and Ant, respectively.
Regarding the behavior policy for collecting training samples, we found that calculating PGD at every step for each action with the environment is computationally too heavy. Instead, we utilize uniform distribution in place of the epsilon-worg worst distribution. A 50:50 ratio of incorporating information from perturbed and non-perturhed settings works well, while for high-dimensional tasks like Ant, fully perturbed settings enhance robustness. We hypothesize that the uniform perturbation setting is not overly pessimistic and increases the likelihood of encountering state-action pairs that are challenging for the agent.
To enhance robustness, we apply the robust regularization technique in the same manner as described for SAC-PPO.
Table 5 summarizes the hyperparameter settings.
VALT-SOFT-SAC Settings. VALT-SOFT-SAC uses an inference model to represent the (soft) optimal adversary policy. The network architecture and learning are identical to the policy network in SAC, but the output layer is modified to match the dimensionality of the state space to generate noise.
The update frequency for the adversary policy is somewhat arbitrary. Although updating the adversary policy at every step yielded good results, resetting and retraining it every 1,000-2,000 steps slightly improved training stability. Thus, we chose periodic resets and retraining, with resets every 2,000 steps for HalfCheetah, Hopper, and Walker2d, and every 1,000 steps for Ant.
The tuning parameter Cattk, which determines the strength of the adversary's constraint, was set to constant values of 8.0. 2.0, 4.0, and 4.0 for HalfCheetah, Hopper, Walker2d, and Ant, respectively. Although these values may appear large, they act as entropy term to prevent the model from converging to local minima. Additionally, unlike in the PPO setting, rewards were not normalized. Given the typical scale of discounted cumulative rewards (approximately 300-500), these values are reasonable.
For the behavior policy, we used 100% adversary influence for HalfCheetah and a 50:50 ratio of adversary and non-adversary policies for the other tasks. To avoid overly pessimistic training while acquiring challenging data, we observed that linearly adjusting the proportion during training, particularly for Ant, led to stronger robustness. However, this introduces spec-fi complexity in experience collection, further exploration of behavior policies and experience memory is left for future work.
To enhance robustness, we applied the robust regularization technique in the same manner as described for SAC-PPO and VALT-EPS-SA.
Table 6 summarizes the hyperparameter settings.
D.3. Evaluation Settings
Our trainings and evaluations require much computational resources (CPUs/GPUs) therefore, we extract median seed from from eight different training seeds, then evaluate 20 episodic returns.
30

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Table Hyperparameters for VALT-SOFT-SAC. Compared VALT-EPS-SAC, VALI-SOFT-SAC introduces more hyperparameters due to adversary policy training. However, the adversary learning rate (Adv. Ir) was fixed across tasks without tuning, and others (e.g., Qattk. reset interval) showed low sensitivity.
Envirnment
Em, stee
Agust br
Ent, barget
Reg. coeff.
SGi i iter
Adv, Ir
Ads. rain
Rowt Adh. per Stuge
Other wittings
HallCheetah
1.00
3e-4
1.0
2e-4
1,0
2000
8.0
Lineu schelale; sttack scale for reg. tomo from 0.0 to 0.15 (0.135 (.SM)
Hogper
0.5M
3e44
0.2
3.0
2
3e44
0.5
2000
20
Lince scheddde: sttak waale fox reg, lemn from 0.0 0.075 (0-4M)4455)
Walker2e
1.5
3e4
din]/l]
11
3e.4
2000
4.0
Liteor schedule: attack scale for reg. term from 0.0 0.05 (0:1M 1.3M)
Ant
3,008
2e4
din]-]]
3000
2
30.
1000
4.0
Liteur schedule: attack scale for reg. Jomn droa 0.0 0.15 (0.13 2.7M)
During evaluation and adversarial training RRS/SA-RL/PA-AD), employ the same state normalizer learned by the agent to scale attacker strength, Although this approach is somewhat akin to white-box attack settings, ensure that these settings are treated consistently, thus guaranteeing that the results are evaluated in uniform manner.
Max Action Deifference (MAD) Attack. We follow as the original implementation (Zhang et al., 2020b). We utilize SGLD and confirm that ten gradient iterations after random initial start are enough for the evaluations.
PGD (minV/minQ) Attack. We follow the original implementation (Zhang et al., 2020b), rather than the method originally proposed in Pattanaik et al. (2018).
For the PPO variants, we use the implementation provided in (Zhang et al., 2021). For the SAC variants, we implement them ourselves by extracting the essence of the PPO implementation. As mentioned in Appendix D.2, the SAC critic comprises two networks, and we use the mean these two action- value estimates to compute the gradient descent.
We utilize Projected Gradient Descent (PGD) and confirm that ten gradient iterations with random initial start are sufficient for the evaluations.
RobustSarsa (RS) Attack. We apply the implementation provided in (Zhang et al., 2021) for both PPO and SAC variants. We perform multiple training runs to tune robust critic parameters (perturbation scale and regression term) around the benchmark's attack scale. Subsequently, we conduct multiple evaluations using all trained models and adopt the worst evaluation results. For fair comparisons, we use the same number of critic trainings and evaluations across all methods in a uniform manner.
SA-RL and PA-AD Attacks Implemented with PPO and SAC. For PPO (SA-RL/PA-AD) attackers, we apply the implementations provided in (Zhang et al., 2021; Sun et al., 2022). We perform approximately seventy training runs with different seeds and parameters to identify the strongest attacker for the agent. Subsequently, we conduct evaluations with these attackers and adopt the worst evaluation scores.
For SAC SS-RL//A-ADD attackers, we use the same hyperparameters and settings detailed in Appendix D.2, with two exceptions: flipping the reward term
and setting the target entropy as Hurger -dim|S] for SA-RL. To mitigate variance, we perform four training runs with different seeds for both SA-RL and PA-AD. Subsequently, we conduct evaluations with these attackers and adopt the worst evaluation scores.
For fair comparisons, we ensure the same number attacker training runs and evaluations across all methods a consistent manner.
E. Additional Experiments
E.I. Vertifying without robust regularizer term
To examine the robustness characteristics of SAC-based methods and the tendencies of our proposed approaches, we conducted evaluations with and without regularization. The results are summarized in Table 7.
The colored settings indicate the more robust configurations, which were adopted in our main results. Specifically, yellow
highlights settings where regularization was applied, while lime denotes settings without regularization.
Our method demonstrated strong robustness against adversarial attacks (SA-RL/PA-AD) in HalfCheetah, even without regularization. However, in the Ant task, VALT-SOFT-SAC suffered a significant performance drop even evaluations
31

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Table Comparison of performance with and without regularization terms, Average episodic rewards (# standard deviation) for median- seed models of our proposed methods ( (VALT-EPS, VALT- and other SAC baselines across four MuJoCo tasks. All evaluations were conducted over twenty episodes with different seeds. Yellow indicates a setting where regularization improves robustness, while
lime denotes non-regularized setting. The most robust method in the off-policy category is highlighted in hold.
Emironment
Method
Ray. Steps
Natural Reward
Uniform
MAD
PGD
uminQ)
PGD
RS
SA-RI.
(5AC)
PA-AD
(5AC)
SA-RI.
(PPO)
PA-AD
(PPO)
across attacks
Word scorn
SAC
IM
19985-117 197
6411+222
5188+154
9424+1711
4215+247
-51+267
1757--400
996--465
1171+2144
+51++767
Ribus-SAC
IM
6350=68
6127:84
5857+3112
2601=311
3183#575
2026=:477
5481-333
31974552
53675468
2026:417
HalfChretta dim: 17
SAC FPO
6M
638521190
58332870
577824475
17162117
31402310
397::65
14792155
1578
18521581
3971-68
actim-dim 0.15
SAC-200
6M
7467:1177
71548141
6000s 1064
1823+671
5836: 138
120
2052:835
2540:726
5005:-303
18239897
VAL
IM 7313-967
7345+156
6181-587
2590-137
6652 161
2416-22
43800-000
9027-264
44424211
2416+337
SAL
0027
3900884
5850
46474180
58346:07
1216:119
5336
100
10
4647
VALTSOFESAC
IM 6100-76
7769=76
4867+310
4533+80
5067+66
3339-88
4585=81
4796+135
4697+82
5120-158
VALTSOFTSMC
IM
381
5798355
5448
4490135
54341.00
3544:61
4649269
4722:88
4868
35811611
SAC
6.5
3199
321326
3194
52811112
3357-5
2979:11
3166:11
1014:1
30012
2
RABuu-SAC
0.5M
323725
323428
3237-99
2947:36
3290-2
2950:10
3114226
2952
3015
2047.66
Hepper state din: 11
SAC
SAM
3
39824
5419.117
3528-112
3527:16
5003163
3256.112
3198-2
3236:6
3196
actitn-dim:
SAC-P90 -rg.
SAM
317612
117486
1185-10
9096-119
310412
5176-4
1244-19
3157=2
3140
5006-119
0.075
VALT-EPS-SAC
0.5M
320022
319524
3196::6
25074779
26754812
3005
3109223
2974:2
199
1591#112
VALTERSS-AC wing
0.5M
2
329783
1285--
5764817
535892
336142
325549
3212
125696
9212
0.5M
331023
331926
3407:114
3478:55
5465-24
3244411
3219214
650=118
3500
650= 118
VALTSOFTSAC
0.5M
322842
1230210
124
2986
346213
3034
310219
305927
303124
2956
47
191192525
93526-2771
Robux
1.M
2840-21
5784263
51801505
57666220
8432:842
5552526
5645-50
5644167
S190E
stte-diin 17
Walker2d
SAC-PO -ry
SAC
65M
65M
3493
9182-36
400311423
50095847
426511501
9005
1423
9420=
1265-11455
4232 1640
1512:153
821
22874795
1343
1830 1306
1609
2096+-0000
1995 1666
17588175
121+11
actin-dim:
c- 0.05
VALT-EPSSACC
LSM
56
5900
5157
4777=1932
1486-11146
58301993
346243438
2593
399342371
VALT-EPS-SAC +my
1.M
5757-58
5740+128
179941783
5791-497
4972 1834
3134
923+99
2524
39976-452
3234-99
VALESOFESSC
L.M
560-24
5609
5579-85
53074995
5490171
5349-18
+981+1567
5485-19
S458+20 +20
VALE vigg
1.5
1655-11
1638
3679-228
2838
1451:1129
5572-857
5568
5515-335
4487
2838 2003
SAC
3M
1852
996 1076
271-74
23:55
106:2370
195221211
10814700
725
71:102
1953
6790 1548
61174:2226
48476-2578
1687=1125
937
21709
1016
1045-1135
3351
stute 111
astiondiin:
PA SAC PPO
13M
635121730
1207
6308::1518
5854111952
5680:
336:1102
17784660
1846272
1760-11798
00
4169::2144
1431
3552:468
1461:237
37542661
336:135
407762110
K371866
144:237
<= 0.15
VALT-EPS-SCC
6850
46434:2557
3006-92263
1901:1185
1252:11419 1419
3851 =3208
4481433277
741:8744
1374
741:874
errg.
63324748
512912297
5019:/1929
1687:1176
3356-2408
4710:2559
09644/991
1736: 1486
45856-2022
VALTSOFTSAC
1703
2456:+1524
8734:588
10074-331
1009=1010 1010
392:5411
745-71
41365441
5001:300
7c
VALESOFT:SCC 1102
6719
685242200
52875:2035
27576
1718
518241033
4280:1007
2594-82590
without noise.
At first glance, this result may seem unusual. However, since our method generates adversarial outputs in the state space, Ant, with its 11 1-ddimensional state space, presents a particularly large and complex setting. We propose two hypotheses: first, without a certain degree of robust adjustment (in this case, regularization), the agent cannot effectively counter the adversary: second, since the adversary policy is trained through value estimation via the agent's 's policy, successful adversary training requires the agent's policy to be sufficiently smooth.
As noted in the main section, the impact of regularization is relatively small for tasks with smaller noise scales, such as Hopper and Walker. Eysenbach & Levine (2022) noted that SAC inherently possesses robustness properties, allowing it to handle relatively small perturbations without explicit regularization, and to perform comparably to the standard baseline in such settings. However, larger perturbations require deeper insights into the MDP structure, as addressed by methods like ATLA and our proposed VALT. In environments with high- sigh-dimensional input spaces, regularization becomes essential for stabilizing adversary learning. Analyzing the behavior of non-max-en DRL methods, such as TD3 (Fujimoto et al., 2018), in this context also an interesting direction, which we leave for future work.
E.2. A Ablation Study for VALT-EPS and VALT-SOFT
Effects of Policy Evaluation and Policy Improvement with Soft-Worst Adversary. We conducted an investigation into the importance of Policy Evaluation (PE) and Policy Improvement (PI) among the update methods we introduced in Section 4, which assume adversary during training. This analysis focused on the Ant task, where particularly distinctive trends were observed for VALT-EPS-SAC and VALT-SOFT-SAC. The training curves are shown in Fig. 4, and the evaluation results are summarized in Table 8.
In the ablation setting where the adversary is not assumed during policy evaluation (w/o PE), there is little change in evaluation scores. However, in the ablation setting where the adversary is not assumed during policy improvement (w/ PI), training collapses in both methods, resulting in catastrophic performance even in frese-free evaluations.
32

---

Off-Policy Actor-Crithe for Observation Robustness: Virtual Alternative Training
Ant
Ant
8000
8000
Avrage Episodic Return
7000
Avrage Episodic Return
7000
6000
6000
5000
5000
4000
VALT-EPS VALT-EPS WOOEE VALT-EPS WooP!
VALT-EPS-reg.
4000
VALT-SOFT-H +eg..
SALT-SOFT
3000
3000
VALT-SOFT w/oPE
VALT-SOFT w/oFl
2000
2000
1000
1000
0
0
1e6
1e6
Env. Steps
Env. Steps
(a) From the ablation study VALT-EPS
(b) From the ablation study on VALT-SOFT
Figure 4. Learning curves from the ablation study on VALT-EPS and VALT-SOFT for the Ant task under no-attack settings. We ablate the policy regularization lerm (+reg.), the adversary during policy evaluation (w/o PE), and the adversary during policy improvement (w/o PI). For both methods, w/o Pl is crucial for training stability.
Table Comparison of performance with and without the soft worst-case adversary during Policy Evaluation (PE) and Policy Improvement (PI). We denote these settings as w/o PE and w/o PI. Average episodic rewards (+ standard deviation) for median-seed models. All evaluations were conducted over twenty episodes with different seeds. Yellow indicates the setting used in the main results (with regularization).
Enviranment
Method
Em. Steps
Natural Reward
Unitforn
(MIAV)
PGD
PGD (min4))
SA-RI [SAU)]
PA-AD ISAC)
SA-RI. (PPC)
(PP3)
across attncks
Wors! score:
SAC
3M
6583
196-11076
.277744
-2:::33
370
19836-1213
-1047:7000
-25
-7:1022
12953
SAC +ny
3M
743
5129:22197
5019219923
1925
1457:1176
338
471082519
298421891
17382146
4585522022
145731175
VALT-EPS
3M
6251 802
4553
30266:2263
1691:1188
125281419
3351#2208
4481-22327
741
1425
741:8874
VALT- SAC
JM
6118-1681
6418-1188 1783
27794 28
1144 1044
2155
226911880
5248-62137
1178-
2952 7079
1128-11 102
SAC
3M
24:36
24.3
10025
10:25
actiondian:
VALESOFESAC
3M
6719-66
6532 760
52876-3015
2757 1297
3098 1521
381811718
515261683
42811-101 167
3084--2900
2787 1297
c: 0.15
SALTSOFFT.SC
3M
3292
2458:1152
873--555
10772731
1009
3921:411
4132441
74:271
'ALT-SOFT-SAC
3M
4917:61405
3816:12081
1420=880
4245:440
3472-11822
157:1100
205:176 176
LS7:110
VALI-SFFI-AA wib
JM
554-1601
12452
3801-306
3483-457
-795-6044
-795-014
We hypothesize that this occurs because the adversary learns to generate difficult noise through the agent's 's value function, whereas the agent's 's policy does not adapt to such noise during training. As result, the value estimation may gradually become excessively pessimistic.
This trend resembles the learning collapse observed in WocaR-SAC, as discussed in Section D.2 and illustrated in Fig. 3, further suggesting that assuming the presence of an adversary during policy improvement is crucial for off- ool-col methods.
Some readers may wonder why the w/o PE settings perform better than the standard settings (especially in the case of VALT-SOFT-SAC), as the Natural Reward and Worst Score are not worse-and in some cases even better-than those of the standard settings without regularization. To investigate this phenomenon, we conducted an additional evaluation described in the next paragraph.
Additional Evaluation of VALT-EPS and VALT-SOFT with Regularization under the w/o PE Condition. To confirm the robustness performance under the w/o PE condition, we conducted additional evaluations of VALT-EPS-SAC and VALT-SOFT-SAC with regularization. The results are summarized in Table 9.
Although the w/o PE condition shows better natural performance than the unregularized settings, its worst-case robustness is lower than that of the standard settings (i.e., with policy evaluation adversaries). Interestingly, the strongest adversary against the standard settings is PGD (minQ), while for the w/o PF settings, it is SA-RI. PPO.
These results suggest that omitting adversaries during policy evaluation stabilizes training and leads to better performance under clean conditions. However, the agent's s action-value function lacks sufficient pessimism, which limits its ability to capture worst-case robustness.
33

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Our VALT framework is designed to learn robust policies by embedding soft worst-case estimation directly into a unified action-value function. Consequently, this value function not only guides policy learning under adversarial perturbations, but also serves as a robustness-awar evaluation metric under white-box settings. This dual role highlights that the learned value function functions both as target for adversarial exploitation and as reliable indicator of worst-case performance.
Table 9. Additional comparison of performance with and without the soft worst-case adversary during policy evaluation (PE). We denote
the ablation setting as w/o PE. and regularization is applied all settings, including the ablation. Values indicate average episodic rewards (+ standard deviation) for models trained with the median performance seed. All evaluations were conducted over twenty episodes using
different evaluation seeds.
Yellow highlights the setting used in the main results (i.e., without the w/o PE ablation).
Environment
Method
Eax. Stups
Nacurol Reward
Uniform
(minV)
PGD
PGD (nmaQ)
SA-RL
(SAC)
PA-AD (SAC)
SA.RL (PPO)
PA-AD (90))
senne attacks
Worst scure
Ant dim. 131 oction din: 5 0.15
SAC
XM
6563 1892
996
-2
-258558
306
1953:-1215
-10474700
-725
-3:102
-1053-1213
VALLERS SAC TaE VALT FE vegg.
SM
63824748
5129822397
1039-1993
1437411776
3356:2248
471882319
404441199
1788-1466
458542202
1187
3M
020861411
6327741345
$2962:43
340241537
1538 1649
3486522000
4724442306
1155-1129
233241929
1155=1129
VALE-OFFES knng
6918
280
2015
1299
1521
5152-1183
4281
VALE ESAC wao PS -my
3M
6800 1018
639911887
$215:
1977
1902:1226
1719:11600
1603
41561:2183
109012236
1518
312512295
1998 1518
E.3. Considerations for Behavior Policy
Table 10. Comparison of performance under different adversary rates in the behavior policy. We denote these settings Ady where * indicates the adversary rate. Values represent average episodic rewards (1. standard deviation) for models trained with the median seed. Yellow highlights the setting used in the main
All evaluations were conducted over twenty episodes using different evaluation seeds.
results (with a 100% adversary rate and regularization).
Environment
Fas. Steps
Reward
Uniform
(minV)
(min()
PGD
SA-RI. (SAC)
(SAC)
SA-RI. (PFO)
PA-AD (PPO)
across attacks
HallChertah state dim: 17 action dim:
SAC
IM
10985+117
6411-223
3268+154
3424-171
4215+247
-51+267
1757+409
936-4655
1171+214
-51+267
VALT-SFFT AdviL aog.
IM
5581+45
5798-55
5448+93
4480+81
5434+80
3584-61
4449-699
4722-81
4858+57
VALT-OFI T AdvL.
IM
6100116
5769_78
48671310
4533.180
50671:65
3329_88
4385.181
4796__135
4697182
3329-85
0.15
VALT-OFF Adv0.5
IM
7982199
6736 923
45521:276
44961. 178
196611006
16641:0011
34391199 /92
2660 197
18291212
1970:1197
VALESOFT Ad-0.0
IM
90633800
6502::545
37284333
2471=2203
2261:2299
1290:2103
1708&1150
3332:-8
32242:400
1290:200
We evaluated the impact of different adversary assumptions in the behavior policy, as shown in the training curves in Fig. 5 and summarized in Table 10 for robustness evaluation. The Adv1.0+reg. and Adv1.0 settings exhibit similar training curves, likely due to the small regularization coefficient.
HalfCheetah Case
In the Adv0.0 setting, performance is high in a noise-free environment, but it significantly deteriorates under strong adversarial attacks. This may occur because, if the replay buffer contains only favorable samples, policy evaluation assumes an adversary only for one-step predictions from stable states, and policy improvement further enhances actions from these favorable states. As result, HalfCheetah achieves high scores in the noise-free setting, but at the cost of reduced robustness.
Since HalfCheetah is less prone to training collapse, we adopted a 100% adversary influence. However, future research should explore adaptive balancing mechanisms to mitigate task dependency and correct distributional shifts in the replay buffer.
Table 11. Comparison of performance under different adversary rates in the behavior policy. We denote these settings Ady where * indicates the adversary rate. Values represent average episndic rewards (+ standard deviation) for models trained with the median seed.
All evaluations were conducted over twenty episodes using different evaluation seeds. Yellow highlights the setting used in the main
results (with 100% adversary rate and regularization).
Rnvinoment
Metbod
Steps
Kas.
Uniferrn
(mirQ)
(SAC)
(SAC)
(100)
acrse
Aat stre-dic 11I adice-dim: = 0.16
SAC
1M
6593:1882
106
27a24
23:35
405:2300
1953311213
1047:22 700
3254332
-22102
-193841213
VALISOOFT Adved breatt - coust))
3M
6862:1400
3406-2200
400-4722
65c171
217:1667
135:110
55:388
40:106
111:56
40:116
VALESOFT ABVIS Gregtionn const)
JM
67194666
6002-260
52874-2035
3757-11597
3088-1131
3806-17118
4251:1167
36861-2500
278741297
VALISOFT Ad angious (mas))
3M
50092-408
5955-512
5436:1055 1054
2008:11544
4722:11066
1814
$121-4455
33071-2065
44381:2099
2H0111314
Adv1.8 evrg drucs 1000
Ant Case We further evaluated the impact of different adversarial influence ratios during agent behavior on the Ant task. In addition to fixed adversarial strengths, we introduced a configuration that anneals the adversary temperature parameter aakk from a high value (1000.0) to a low value (4.0), aiming to gradually increase the adversarial impact over the course of training, while keeping the adversarial influence ratio constant (Ady 1.0).
34

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
HalfCheetah
12000
Avrage Episodic Return
10000
8000
6000
4000
SAC
Robust-SAC
2000
VALT-SOFT Adv1.0
VALT-SON SOFT Adv1.0
VALT-SOFT AdvO.5
VALT-SOFT Adv0.0
0.0
0.2
0.4
0.6
0.8
1.0
le6
Env. Steps
Figure 5. Impact of different adversary assumptions in the behavior policy. Training curves for various adversary influence rates in VALT-SOFT-SAC on the HalfCheetah task under no-attack settings. Adv1.0 represents a setting where the behavior policy is fully influenced by the adversary (i.c., v' st)), while Adv0.0 assumes no adversary influence (i.c., as π(- 8t)).
Figure 6 shows the clean evaluation curves averaged over eight different seeds. Most configurations achieved successful learning; however, the setting with full adversarial influence during behavior (Adv 1.0) and fixed temperature (O 4.0) showed large variance across seeds and an increased chance of converging to suboptimal policies (local minima), occurring in out of 8 seeds compared to the typical out of 8. This degradation likely stems from the fact that, under this setting, training trajectories are collected assuming strong adversaries even in the early stages-before before the agent has learned sufficiently good behaviors thus increasing the chance of getting stuck in poor local solutions.
To further assess robustness, we report adversarial evaluation scores in Table 11. Similar to the HalfCheetah case, models trained without adversarial consideration in the behavior policy suffered severe performance drops under attack even when regularization was applied. This suggests that in high-dimensional input spaces, failing to account for adversarial perturbations during data collection leads to insufficient coverage of pessimistic scenarios. degrading robustness.
Interestingly, the setting with full adversarial influence (Adv = 1.0) and regularization produced the robust policies terms of median performance, and further improvements were observed when the adversary temperature, Cattke was annealed from 1000 This schedule consistently resulted in stable learning across all seeds and yielded robust behavior even under worst-case evaluations.
These findings imply that introducing adversarial signals gradually-i.e. allowing the agent collect sufficient data for reasonable behavior early on and emphasizing stronger perturbations later-can improve both optimization and data collection strategies. Although further fine-grained scheduling of adversarial influence or temperature might enhance learning efficiency and robustness, off-policy methods inherently face the challenge policy-data mismatch. Therefore, we focused on systematic experiments with straightforward and consistent configurations in this study.
F. Possible Framework for Discrete Action Domains in the Case of VALT
In this work, we focused on continuous action domains, especially in the case of SAC. Though we believe that our proposed methods can be extended to discrete action environments, here we discuss the possible framework for discrete action environments in the case of VALT and the possible challenges for the implementation. For the simplicity, we discuss the DQN base framework (deterministic policy), but the same idea can applied to the other cases like SAC- Discrete (Christodoulou, 2019).
35

---

Off-Policy Actor-Critie for Observation Robustness: Virtual Alternative Training
Ant
8000
Avrage Episodic Return
6000
4000
2000
SAC
V-50FT-reg//dv=00000=== 0 const.)
W-S0FT=rcg/Adv==05cc = const.)
0
V.S0FT-reg/Advv=10.e 0= const.)
V.SOFT-reg(Ad==100 a= 1000 1=00:000=00000000000000000000000 4)
1
1e6
Env. Steps
Figure 6. Training performance of VALT-SOFT-SAC the Ant task under no-attack evaluation, comparing different behavior policy assumptions and Csa schedules. Adv1.0 assumes full adversarial influence (āg ~ т )" while Adv0.0 uses the agent policy alone (as ~ IT) 4 const. denotes fixed attack temperature, and 1000 - 4 indicates scheduled decay from 1000 over 0-2.7M steps. The main result uses Adv0.5 with Carkk = 4 (blue linc).
To consider VALT framework, assume the agent action as:
Oagent π (8) argmax ac Qngent a),
(62)
where Qagent (3, a) is the Q-function for the agent. Then, define the (soft) optimal adversary for the agent as:
y tsol {) = argmin,went Eggee (Qugem (s, a)]l + Ctattk Dy (v p)-
(63)
Lastly, the action-valus function for the agent is updated under the fixed adversary, Eq. (63), by minimizing the following loss:
Loss(Qugem)) = Loss LLos-Loossfee Sunc(Qugenti (8, u(, a, y(s, a, s') = r(s, a) + 7lE Ex O (s', a')
(64)
where Oagen the target Q-function for the agent, Loss_fure represents the loss function, e.g., L2-mean and Huber loss, under the distribution, s, s,a, D(-), and y(s, a, s') represents the one-step target value for the agent.
The main challenge in practical implementation is the development of an algorithm to compute the following expectation term in Eq. (64):
Byrsen [ [Qagen ā.) ā')]]=
(65)
which is handled differently the VALT-SOFT and VALT-EPS cases, as described below.
VALT-SOFT for Discrete Actions In the VALT-SOFT case, can derive the soft worst-case adversary distribution (as in VALT-SOFT-SAC, Eq. (9)) as:
*5oft (1s) p( 5) exp (- -Vagent (s, 8)/kk))))
(66)
7
where Z is the partition function, and Vagent (s, Ex (Qagent (s. (s, a)] is the agent's 's value function. The inner expectation E, can be computed in Eq. (62).
36

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Similar to VALT-SOFT-SAC, this soft worst-case adversary distribution can be approximated using surrogate model such as neural network. However, in high-dimensional environments like Atari (e.g., 80x80 inputs), training the adversary model becomes challenging due to the curse of dimensionality in the adversarial action space.
Although we have not yet experimented with this, we assume that regularization plays an important role in stabilizing adversary training, as observed in the ablation study of VALT-SOFT-SAC on Ant (with state dimension 11 111), Additional techniques, such as PA-AD (Sun et al., 2022), may also be required.
VALT-EPS for Discrete Actions In the VALT-EPS case, we can expand Eq. (65) into:
Eg [Ex Qagent ss', a']]]] (1.0 Kowerst Epp En, (Qugen (s', a')l] + Kwors/ming Ex (Qagam a')]
(67)
Here, the first termeetttttteeee the expected value under the prior distribution (uniform in this study)-can can be easily computed, while the second term-the worst-case value within the c-ball c-ball-requires requires approximation techniques to compute the minimum value,
since Qugem is represented by neural network.
In WocaR-DQN. DQN, Liang et al. (2022) proposed two methods to estimate the worst-case value: (1) deriving the available action set via convex relaxation bounds (upper/lower) of the c-ball, from which the worst-case action is selected; or (2) using the Projected Gradient Descent (PGD) method to approximate the worst-case perturbation that degrades the agent's best action.
If we adopt these techniques, VALT-EPS for DQN becomes similar to WocaR-DQN, although some differences remain. WocaR-DQQ -QN introduces an additional Q-function model the worst-case, whereas in our approach, the worst-case is unified into the agent's Q-function as soft worst-case while preserving the contraction property.
In the case of VALT-EPS, high-dimensional state spaces may not pose significant challenge, as the worst-case values are computed through the Q-function using convex relaxation or gradient descent techniques.
Actor-Critic Cases Although the above discussion assumes value- iillee-ttraaiio-stye style method (e.g., DQN), where Eq. (64) can be interpreted form of value iteration, a policy-iteration-sty framework (e.g., Actor-Critic) may offer advantages in computing Eq. (65). Instead of requiring an additional procedure to estimate the arg max in Eq. (62), we can directly compute the expectation term [x][]] that appears in Eqs. (65), (66), and (67), under a fixed policy assumption. Then, both the adversary's 's optimization in VALT-SOFT (Eq. (66)) and the worst-case value estimation in VALT-EPS (Eq. (67)) can be approximated as demonstrated in this study: the former directly utilizes the fixed policy for the expectation, while the latter approximated by applying PGD either the mean action of deterministic policy or the value prediction under the softmax policy (as in SAC-Discrete).
Summary In summary, we have presented a possible framework for VALT in discrete action domains. While the proposed approaches show promise, further progress is expected in integrating advances areas such as the vulnerability of DNN in high-dimensional input spaces, computational efficiency, and the practical application to modern discrete-action domains.
G. Computational Resources
Wall-clock Time to Train To clarify the time and resources required for our algorithms and other baselines, we present Table 12, which summarizes the total training time for each algorithm. For a fair comparison, all training runs were conducted on Tesla V100 32GB GPUs.
Notably, our implementation can also run on CPU with as little as 1GB of memory (3GB for Ant), albeit approximately 1.5 times the training duration compared to GPU-based execution. However, observed that PALT-EPS-SAC fails to proceed with learning on CPUs, potentially due to the computational overhead of PGD.
PPO variants (e.g., PPO, ATLA-PPO) have been reported in prior work (Liang et al., 2022), and we confirm that our results are consistent with theirs, as we use the same publicly available codebase.
Analysis of Computation Time To analyze the computational efficiency of our methods and guide future improvements, we provide a breakdown of computation time in Table 13, based on training various SAC variants on the HalfCheetah task. Here, cur-q. next_o, q_back, and q-opt respectively represent the computation of current Q-values during policy evaluation, value estimation for the next state, backpropagation for action-value function optimization, and parameter
37

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Table 12. Computation times for each method
HalfCheetah
Hopper
Walker2d
Ant
Model
Time (hour)
SAC
4.2
2.1
6.2
12.8
Robust-SAC
7.7
3.7
11.2
23.4
SAC-PPO
13.5
10.3
17.8
47.9
VALT-EPS-SAL
9.7
4.8
14.1
30.1
VALT-SOFT-SAC
10.1
5.0
14.8
31.0
Table 13. Computation time (seconds) per 10k steps on HalfCheetah.
Policy Evaluation
Policy Improvement
Other Components
Model
act
cur.q
next_q
q_back
q-opt
p_fwd
reg
p_back
p.opt
a_learn
adv_learn
SAC
21
22
24
Robust-SAC
19
6
22
17
22
24
125
7
SAC-PPO
25
6
27
18
27
31
71
35
16
9
229
VALT-EPS-SAC
20
6
150
18
24
156
69
53
14
8
VALT-SOFT-SAC
35
38
24
43
43
8 90
updates for the action-value network. Similarly, p-Ewd, reg, p.back, and pLopt correspond to action computation during policy improvement, regularization term computation, backpropagation of the policy loss, and policy parameter updates. a alpha_lears rn and adv_leare denote the time spent on temperature parameter adjustment and adversary training, respectively.
For SAC-PPO, we normalize the computation time based on the number of learning steps equivalent SAC's 10k steps, since it consists of PPO adversary training, 2048 steps over 2441 iterations in total.
We omit simulation time in MuJoCo, replay buffer operations (e.g., storing and sampling), and post-processing such as logging, these components did not significantly vary across methods and were not major contributors to overall computation.
Note that the reported values reflect peak computation time, and are subject to fluctuation due to scheduling (e.g., of regularization terms) and variations in overall server load.
Among the most notable factors:
SAC-PPO: As the forward passes of the adversary (PPO) are computationally light, the agent-side optimization also relatively inexpensive. However, the large number adversary updates required by the alternating training scheme results in a significant overall time cost. While reducing the number of adversary updates is a possible approach, achieving balance between SAC's training efficiency and proper tuning of the adversary proves be highly challenging and likely impractical.
This method incurs substantial overhead during both policy evaluation and improvement due to the requirement PGD computations at each iteration. As a result, next_q and p_fwd show markedly high values. Since reducing the number of PGD iterations (currently two) would severely compromise robustness, further efficiency gains would require parameter tuning alternative methods.
VALT-EPS:
VALT-SOFT:
The adversary is trained at fixed intervals (every 1000-2000 steps) using trajectories sampled from the replay buffer. We observed that robustness performance was relatively insensitive to the choice of interval size. While reducing the number of update steps per interval could further improve efficiency, adversary training was not major computational bottleneck and thus remained unchanged.
Other Observations: Methods incorporating adversarial signals into the behavior policy (c.g., SAC-PPO, VALT-SOFT) show minor additional costs proportional to computational time. Robust-SAC's regularization takes 70-125 seconds due to SGLD iterations (following SA-DDPG (Zhang et al., 2020b)), while other methods use only 2. For better efficiency, Robust-SAC could also adopt fewer iterations.
38

---

Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training
Summary As discussed in the main results and computational analysis, framework achieves observation robustness through formulation naturally integrated into the effectilllly addressing challenge for existing off- off-policy methods. Despite its effectiveness, our approach entails additional computational overhead, described in this section. These aspects leave room for further improvement and serve as promising direction for future research.
One possible direction for broader adoption by general off-policy actor-critic users- while keeping additional cost minimal- might be to introduce assumed environment noise during behavior policy execution, policy evaluation, and policy updates. This idea is analogous to how TD3 adds noise actions during policy evaluation.
H. Reproducing Statement
We provide detailed procedures, experimental settings, and hyperparameters in Appendix C and Appendix D. In addition, we release our code-including implementations of SAC variants and evaluation settings-for reproducibility and further research.
Limitations
As discussed in Section 4, proposed methods grounded in solid theoretical foundation and can be applied across variety of scenarios. However, there are two primary limitations to be noted.
First, although we outline possible extension of the framework to discrete action domains in Appendix F, detailed algorithm and implementation are not yet provided.
Second, the proposed methods are less compatible with on- on-policy algorithms, as the VALT framework fundamentally relies on off-policy value estimation, which is not well aligned with the core principles of on-policy learning. (See the discussion on "WocaR-SAC Settings" in Appendix D.2 for further details.)
39


Reviews:
[{"review_id": "6eSjy8Byn5", "reviewer": "Reviewer_weVU", "summary": "The paper \"Robust Off-Policy Actor-Critic: Virtual Alternative Training via Symmetric Policy Evaluation\" addresses the challenge of training reinforcement learning (RL) agents that are robust to adversarial perturbations in their input observations. Existing methods often rely on alternating training between the agent and an explicitly learned adversary, which can be sample-inefficient and difficult to integrate with off-policy algorithms. This paper proposes a novel off-policy framework called Virtual ALternative Training (VALT). The key idea of VALT is to reformulate adversarial learning as a soft-constrained optimization problem that eliminates the need for additional environmental interactions to train the adversary. Instead, the adversary's policy and value function are implicitly derived by leveraging the agent's value estimation and a symmetric property of policy evaluation between the agent and the adversary.\n\nThe paper presents a way to construct an alternative adversarial training framework without explicitly learning an RL policy for the adversary. This is achieved by exploiting the symmetry in policy evaluation. The authors present two concrete algorithms based on the Soft Actor-Critic (SAC) that implement the VALT framework. These algorithms demonstrate both sample efficiency and robustness against various adversarial attacks. VALT-EPS-SAC uses an epsilon-worst case approach to approximate the adversary and VALT-SOFT-SAC employs a parameterized policy network to model the (soft) optimal adversary.\n\nThe paper presents experiments on challenging MuJoCo continuous control tasks (HalfCheetah, Hopper, Walker2d, and Ant) show that the proposed VALT-based algorithms achieve significantly better sample efficiency compared to on-policy adversarial training methods (like ATLA-PPO) and demonstrate superior robustness against a wide range of heuristic and learning-based adversaries, often outperforming existing robust off-policy baselines (like Robust-SAC).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate your detailed review and insightful suggestions.**  \nWe understand that your main concerns center around:  \n(1) adaptation to discrete action domains,  \n(2) analysis of computational cost, and  \n(3) performance degradation of VALT on HalfCheetah in the absence of noise.  \nBelow, we address each point in detail.\n\n---\n\n### (1) Adaptation to Discrete Action Domains and Visually Challenging Environments\n\n> _\"The paper acknowledges that VALT currently lacks support for off-policy algorithms in discrete action environments... Have you tried other more visually challenging environments?\"_\n\nThank you for highlighting this important direction.  \nWe agree that both discrete action settings and visually challenging observations are valuable for broadening the applicability of robust RL methods.  \nAlthough we have not yet explored this direction, we plan to further develop our current approach to pursue significant improvements in this area.\n\nAs briefly noted in line 1441, we observe a conceptual connection between VALT-EPS and WocaR. In the case of discrete action spaces, we believe that a discrete variant of VALT-EPS would resemble WocaR-DQN. Specifically, our variant would rely on a **single soft-worst Q-function** (compared to the two used in WocaR-DQN: vanilla and worst), updated using **both uniformly random attacked Q-values and PGD/convex-relaxed worst-case Q-values**.\n\nWhile this variation offers interesting theoretical properties—such as **contraction** and **policy improvement**—we chose not to further discuss it due to the lack of supporting experiments at this stage. However, if the reviewer finds it appropriate, we would be happy to include this theoretical discussion and point to it.\n\nThat said, even without experiments on discrete action tasks with visual inputs, we believe the contributions of this work are significant enough to justify publication:\n\n- A demonstration that adversarial robustness can be achieved without explicitly training an adversary.\n- Theoretical guarantees (e.g., contraction, policy improvement under fixed adversary assumptions).\n- The first extensive benchmark for off-policy robust RL with SAC variants, evaluated under a PPO-compatible framework. \n- We are committed to releasing our code and evaluation environments to support reproducibility and facilitate future research on observation-robust off-policy methods.\n\n---\n\n### (2) Computational Cost Analysis\n\n> _\"Mentioning computational cost is good, but a more thorough analysis of wall-clock time or FLOPs compared to baselines would be valuable.\"_\n\nWe agree and appreciate this suggestion. While we report wall-clock time in Appendix F, we acknowledge that a more detailed breakdown (e.g., time per processes or FLOPs per step) would be valuable. We plan to include this analysis in a future revision and will provide code and scripts (currently being refactored and documented) that enable monitoring of computation times to support full reproducibility.\n\n---\n\n### (3) Performance on HalfCheetah without Noise\n\n> _\"Why do you think the results without noise are so much worse on HalfCheetah for VALT than the SAC baseline?\"_\n\nThis is a great point. As noted in prior robust RL studies (e.g., [1]), there exists a fundamental **trade-off between clean-environment performance and robustness** under adversarial perturbations. This trade-off applies not only to VALT but also to methods like Robust-SAC (see line 1359), where reducing regularization restores SAC-like performance (\\~10,000) but sacrifices robustness.\n\nFor VALT, similar trade-offs are governed by:\n\n- The adversarial ratio in the behavior policy (e.g., assuming no attack vs. assuming adversarial conditions), and\n- The strength of the soft constraint coefficient (e.g., considering near-random vs. near-worst-case attacks).\n\nThe effect of the adversarial ratio in the behavior policy is illustrated in Table 5 and Figure 5.\n\nAlthough WocaR-PPO [1] also reports such a trade-off, it appears less drastic due to PPO's lower clean performance (\\~5,000-6,000), whereas SAC achieves higher scores(\\~8,000-10,000), thus making the drop more visually noticeable.\n\n[1] Liang, Y., Sun, Y., Zheng, R., and Huang, F. Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning. *Advances in Neural Information Processing Systems*, 35:22547–22561, 2022.\n\n---\n\nOnce again, thank you for your constructive feedback. We believe your suggestions will help us further improve the clarity, applicability, and impact of our work.", "claims_and_evidence": "The first claim is that it is theoretically possible to construct an adversarial framework without requiring an explicit RL process for the adversary. This is achieved by a theoretical proof showing that the adversary's optimal value function has a simple relationship to the agents optimal value function.\n\nThe second claim is that this new framework is viable and robust experimentally. This is supported by empirical results in figure 2 and table 1. Experiments are run on some OpenAi gym control problems.", "methods_and_evaluation_criteria": "The method is tested on OpenAI gym which has relatively simple state representation. The observations are visually very simple which somewhat limits the potential impact.\n\nMentioning computational cost is good, but a more thorough analysis of wall-clock time or FLOPs compared to baselines would be valuable, especially considering VALT claims to improve sample efficiency without increased computation.", "theoretical_claims": "I only had a look at the proof of theorem 4.6 because it is the most important piece. It seems correct.", "experimental_designs_or_analyses": "I focused on the main results from sec. 5. The main results seem sound and valid. The authors compare against a wide range of both on-policy and off-policy methods, including state-of-the-art robust RL algorithms, allows for a strong assessment of VALT's relative performance. \n\nThey evaluated robustness against various attackers:\n* Heuristic Attacks: Random (Uniform), Max-ActionDiff (MAD), PGD (minQ for SAC and minV for PPO), and RobustSarsa (RS). This provides a range of attack strategies, from simple random noise to gradient-based and more sophisticated attacks.\n* Learning-Based Adversary Attacks: SA-RL (PPO and SAC) and PA-AD (PPO and SAC). These are more challenging attackers, as they are learned RL agents themselves. \n\nThis provides a good picture of the performance of performance.", "relation_to_broader_scientific_literature": "I think it's novel enough as a contribution to the robust RL literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "* The paper acknowledges that VALT currently lacks support for off-policy algorithms in discrete action environments. The experimental evaluation is therefore limited to continuous action domains. Showing results in discrete action domains, or at least discussing potential adaptations for discrete action spaces more thoroughly, would be beneficial for the broader applicability of VALT.\n\n* Mentioning computational cost is good, but a more thorough analysis of wall-clock time or FLOPs compared to baselines would be valuable, especially considering VALT claims to improve sample efficiency without increased computation.\n\n* In some of the experiments without any perturbation e.g. half cheetah the proposed method gets much worse results than SAC. This is not desirable in general.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "* Have you tried other more visually challenging environments. do the results still hold ?\n\n* why do you think the results without noise are so much worse on half cheetah for VALT than the SAC baseline.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MNDBviElWa", "reviewer": "Reviewer_wL61", "summary": "This paper proposed an off-policy VALT framework for SA-MDP based on Symmetric Property and Soft Optimization. Compared with the existing ATLA framework, this framework improves sample utilization efficiency as it does not require additional training for the Adversary.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and interest in our work.\nBelow, we address your main concerns. We also acknowledge that some notational details may have been omitted due to space limitations.\n\n---\n\n## Claims and Evidence\n\n### **Motivation and Advantage of this work**\n\nWe believe investigating off-policy adversarial learning in SA-MDPs is valuable due to its sample efficiency and performance across diverse tasks. This aligns with the broader trend in RL since the 2010s, where off-policy methods like DQN and DDPG are widely adopted despite distribution shift.\n\nWhile off-policy methods do suffer from distribution mismatch, their efficiency has justified continued use—alongside many efforts to mitigate such issues. We apply the same reasoning in the adversarial setting. VALT inherits distribution shift challenges like any off-policy method, but provides clear theoretical guarantees and achieves strong empirical results—surpassing Robust-SAC in robustness on HalfCheetah and Ant (especially under learned attacks such as SA-RL and PA-AD) while maintaining sample efficiency.\n\nWe emphasize that our method grounds robustness within the MDP framework rather than relying on local smoothness. Our formulation supports both theoretical clarity and practical effectiveness, aligning with ICML’s emphasis on principled contributions.\n\n\n\n### **What does \"enhance generalization\" refer to**\n\n“enhance generalization” refers to the agent's improved robustness under previously unseen or perturbed observations.\n\n---\n\n## Methods and Evaluation Criteria\n\n### **Update criterion (Proposition 4.11):**  \nWe consider a fixed adversary and maximize the value function in Eq. (7). Since $D_f$ does not depend on the policy, it is omitted in the optimization: \n\n$\\mathbb{E}\\_{\\nu} [ \\mathbb{E}_{\\pi}[Q] + \\mathcal{H}(\\pi) ]$,\n\nwhose analytical solution (as in SAC) is $\\pi^{\\star}\\_{\\text{old}}$ in Eq. (13).  \nTo approximate $argmax\\_{\\pi} V^{\\pi}\\_{\\nu^{\\text{soft}}}(s_t)$, we minimize the KL divergence between $\\pi$ and $\\pi^{\\star}_{\\text{old}}$, using the joint distribution $\\pi \\circ \\nu$. This serves as a soft policy improvement criterion.\n\n### **SAC-PPO vs PPO variants:**  \nSAC-PPO uses SAC as the agent, while the PPO variants use PPO.  \nWe introduce SAC-PPO to provide a fair min-max adversarial baseline for SAC agents, as such comparisons have not been explored in prior work.\n\n---\n\n## Theoretical Claims\n\n### **Relaxation via $f$-divergence:**  \nThe $f$-divergence allows optimization over a broader perturbation set $\\mathcal{N}$, while softly constraining the adversary to stay near a prior distribution $p$. Under Assumption 4.2, the choice of $p$ implicitly bounds the effective perturbations.\n\n### **Gap to Nash Equilibrium (Eq. 2):**  \nAs $\\alpha_{\\text{attk}} \\to 0$, the relaxed soft max-min game converges to the exact Nash equilibrium. Similar to entropy regularization in PPO, a small positive $\\alpha_{\\text{attk}}$ often improves learning stability in practice.\n\n### **Missing proofs (Propositions 4.4 & 4.5):**  \nDue to space constraints, we omitted full derivations. Similar to SAC, we define a modified reward:\n\n$\\hat{r} = r + \\mathbb{E}_{\\mathcal{F}}[\\mathcal{H} + D_f]$,\n\nwhich leads to the soft value update rule under an assumed soft-worst adversary at the next state, yielding Eq. (7) and (8).  \nHere, the $Q$-function differs from that in Section 3.1 as it incorporates the adversary’s influence.\n\n---\n\n## Experimental Analysis\n\n### **Variance in Hopper (Fig. 2b):**  \nThis is due to one unstable seed among the eight used. Similar fluctuations are reported in SAC's paper(Fig. 1 in [1]) for Hopper. As noted in line 1316, we apply adjustments, but Hopper's sensitivity makes some variance unavoidable.\n\n[1] Haarnoja et al., \"Soft Actor-Critic Algorithms and Applications\", arXiv 2018.\n\n---\n\n## Supplementary Material\n\n1. Line 8 updates the critic; Line 12 updates the target network.\n2. Learning curves show mean scores over 8 seeds; tables report median seed performance. Lower-scoring outliers affect averages, hence the difference. \n\nIn the Ant ablation, the lack of regularization in VALT-SOFT likely leads to a weak agent policy, which in turn results in an inconsistent adversary. In such cases, excluding adversary effects from value estimation (w/oPE) may prevent overly pessimistic updates and yield better training performance. However, we observe that under regularized conditions (+reg.), w/oPE fails to provide correct policy evaluation and leads to lower robustness. We plan to include this analysis in the revised manuscript.\n\n3. PPO’s training times match those in [2]; we will add our timing results in the revised version.\n\n[2] Liang et al., “Efficient Adversarial Training without Attacking”, NeurIPS 2022.\n\n---\n\n## Broader Scientific Context and Weaknesses\n\n- Concerns on theoretical gaps are addressed in the Theoretical Claims section.\n- The motivation and theoretical justification for off-policy SA-MDPs are detailed above (Claims and Evidence).", "claims_and_evidence": "+ In Line 50, it is mentioned that \"there is currently no off-policy actor-critic method\". Then, what are the advantages of adopting the off-policy approach in SA-MDPs? In the more sensitive setting of state adversarial, off-policy seems to cause severe impacts due to distribution shift (the authors seem to have also mentioned this issue in Line 285). Is such a sacrifice worthwhile for the sake of the advantages of off-policy?\n+ What does \"enhance generalization\" mentioned in Line 67 refer to?", "methods_and_evaluation_criteria": "+ How is the update criterion for policy improvement in Proposition 4.11 obtained?\nWhat are the differences between the SAC-PPO method mentioned in the appendix and the series of PPO methods in the baseline? Why is this additional algorithm introduced?", "theoretical_claims": "+ How does the general $f$-divergence term relax Equation (3) with constraint (1)? What is the gap between the final Nash equilibrium obtained after relaxation and the Nash equilibrium condition (2)?\n+ It seems that the derivations or proofs of Proposition 4.4 and Proposition 4.5 are missing. Is the $Q$ here the same as the $Q$ in Section 3.1? How are $\\mathcal{H}$ and $D_f$ introduced?", "experimental_designs_or_analyses": "+ Why does the phenomenon of a sudden increase in standard deviation occur in the VALT-EPS algorithm in Hopper in Figure 2(b)? Do the authors have any understanding of this?", "relation_to_broader_scientific_literature": "+ The authors mention in Line 59 that \"the proposed robust methods were not robust enough against stronger attacks\". Are there any practical examples of such SA-MDPs with \"stronger attacks\"? Providing such examples would help readers better understand the motivation of this paper.", "essential_references_not_discussed": "There are no necessary references known that this paper fails to discuss.", "other_strengths_and_weaknesses": "## Strengths\n+ The Symmetric Property proposed in this paper effectively reduces the amount of environmental interaction required by the existing SA-MDPs framework.\n+ The VALT framework proposed in this paper fills the gap of the missing off-policy algorithms for solving SA-MDPs.\n## Weaknesses\n+ Due to the lack of derivations or proofs for Proposition 4.4 and Proposition 4.5, I am not entirely certain about the correctness of the Symmetric Property proposed in this paper.\n+ I'm not quite sure whether there are SA-MDPs environments in practice that are highly aggressive, making robust methods insufficiently robust.", "other_comments_or_suggestions": "For the main comments and suggestions, please refer to the previous sections. Additionally, there are no further comments or suggestions.", "questions_for_authors": "Refer to the previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BOJxBvreXJ", "reviewer": "Reviewer_AMCX", "summary": "This paper proposes a method to address the observation robustness, which does not rely on interacting with the environment and making the algorithm off-policy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and detailed feedback. Below, we respond to each of the concerns and suggestions.\n\n---\n\n## Claims and Evidence\n\n### **Scope Clarification (Title/Abstract/Introduction)**\n\nThank you for pointing this out. We agree that our work specifically addresses **adversarial robustness in state observations**, and we will explicitly clarify this scope in the **title**, **abstract**, and **introduction** of the revised manuscript. \n\nIn particular, we plan to revise the title to:  \n**\"Off-Policy Actor-Critic for Adversarial Observation Robustness: ...\"**  \nto more accurately reflect the focus of our study.\n\n---\n\n### **ATLA in Introduction**\n\nThank you for your comment regarding the historical positioning of the ATLA framework.\n\nOur intention in referring to ATLA was to highlight its influence in the **observation robustness** setting, where it has seen wide empirical use in this domain. We did not mean to suggest that ATLA introduced adversarial training prior to earlier works such as **GANs** or **RARL (Pinto et al., 2017)**. We sincerely apologize for any confusion this may have caused.\n\nIn the revised manuscript, we will:\n- Properly cite **GANs** and **RARL** as foundational works in adversarial training.\n- Position **ATLA** as a more recent **formalization** of this framework, specifically within observation perturbation settings.\n\n---\n\n### **On Sample Inefficiency of Adversarial Training**\n\n> \"The mutual dependency between the victim agent and the adversary doubles the required sample size for training...\"\n\nWe appreciate this concern. \n\nWe agree that not all robust RL approaches incur this cost—especially **offline** or **model-based** methods, as noted. Our original statement specifically referred to **model-free online RL**, where the agent and adversary must interact with the environment during training. We will revise the manuscript to clarify this scope and explicitly note the exceptions.\n\n---\n\n## Experimental Clarity\n\n### **Figure 2 Clarification**\n\nThank you for the helpful comment. Due to space constraints, the caption of Figure 2 lacked sufficient detail. We will revise it to explicitly state that the evaluations are conducted under nominal (non-adversarial) conditions, as also mentioned in the main text. To further avoid confusion, we will update the caption to direct readers to Table 1 for robustness metrics under attack.\n\n---\n\n### **Table 1 Interpretation**\n\nThank you for the valuable suggestion. We agree that average performance under attacks may obscure important aspects of robustness. In the revision, we will use the worst-case scores instead of average scores. Please see the revised table at the following anonymous link:\n\nhttps://gofile.io/d/G0l2mW\n\nAlong with the revised table, we will clarify that VALT-EPS and VALT-SOFT consistently maintain high robustness across all tasks. For complex tasks such as HalfCheetah and Ant, incorporating MDP-level considerations—as in our methods—is essential for high robustness, especially off-policy. SAC-based methods generally outperform PPO-based methods on Hopper. While PPO variants achieve the highest score in Ant, they require over three times more environment interactions, highlighting our methods' potential for robust RL with significantly improved sample efficiency.\n\nThank you again for your insightful feedback.\n\n---\n\n## Essential References Not Discussed\n\nWe greatly appreciate the additional references. In the revised manuscript, we will address topics that were not sufficiently covered, such as **attacks on actions and rewards**.\n\nWe also recognize that **offline robust RL** has become increasingly important. We plan to incorporate the literature you suggested, along with recent developments in this area.\n\nDue to space constraints, this discussion may be included in **Appendix A**.\n\n---\n\n## Other Comments or Suggestions\n\n### **Typos**\n\nThank you for pointing these out. We will correct them in the revised manuscript.\n\n---\n\n## Questions for Authors\n\n> **Does your off-policy approach provide benefits in online RL?**\n\nThank you for this insightful question.\n\nYes, we believe our off-policy approach can offer significant benefits in **online RL settings**. In real-world applications such as **robotic learning**, improved sample efficiency leads to **reduced interaction time with the physical environment**, which lowers the need for human monitoring and enables **faster, safer deployment**.\n\nThat said, if the reviewer is referring to **non-stationary or rapidly changing environments**, we agree that **on-policy methods** or **adaptive formulations** may be more appropriate. Exploring hybrid or continual learning strategies is a promising direction for future work.\n\n---\n\nThank you again for your constructive and thoughtful feedback.", "claims_and_evidence": "**General:**\n\nBy looking at the formulation in sec. 3 and related work, it seems this work aims only at state adversarial robustness. Then the scope should be explicitly written early in the title/abstract/introduction.\n\n\n\n**Introduction:**\n\nWhy is ATLA framework mentioned that frequently? It seems less novel to me. As pointed out by the authors, ATLA framework by Zhang et al. and Sun et al. is done in 2021. But then, the authors claim this framework is adopted by Pinto 2017. It is not sensible to claim a later-proposed framework is adopted by an algorithm proposed 4 years ago. RARL (Pinto et al.) directly use such adversarial training framework (similar to GAN as well) without theory, thus this so-called ATLA framework, is just normal adversarial training protocol. Back to this work, the authors should not introduce such method by citing Zhang et al. but better citing RARL/GAN first and then, say sth like formally concluded by Zhang et al.\n\n\"The mutual dependency between the victim agent and the adversary doubles the required sample size for training, leading to inefficiencies and increased computational costs.\" Not sure about this, isn't every work studying adversarial robust needs such additional cost?", "methods_and_evaluation_criteria": "The graphical demonstration in Fig 1 is clear and nice-looking!", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiments are comprehensive, but there are concerns.\n\nThe first one is Fig 2, where the caption is not clearly demonstrating the experimental tasks are under attack or not, and it is under which attack. Besides, SAC is significantly better than others, which might also be a concern.\n\nThe issue of Table 1 is that it is hard to conclude the proposed method is superior or not. By looking at the average score, the proposed method only works in Halfcheetah, but this way of concluding is definitely biased. Considering the defense effect against all attacks are important. My proposal will be, to add an additional column/plot of worst-case performance under these attacks. I think it's a way better metric to conclude the defense effect than average performance.", "relation_to_broader_scientific_literature": "The paper is based on a general adversarial training framework for robustness, namely VALT [Zhang et al, 2021]. In this work, the authors remove the interactions with environment by providing analytical solution to the optimal adversary.", "essential_references_not_discussed": "The main issue is related work seems only include \"Adversarial Attack and Defense on State Observations.\" There are lots of other type of attack such as action attack:\n\nPinto, Lerrel, et al. \"Robust adversarial reinforcement learning.\" *International conference on machine learning*. PMLR, 2017.\n\nTessler, Chen, Yonathan Efroni, and Shie Mannor. \"Action robust reinforcement learning and applications in continuous control.\" *International Conference on Machine Learning*. PMLR, 2019.\n\n\n\n\n\nThe minor one is that authors claim about the novelty of leveraging off-policy notion to address robustness. But off-policy seems often required in offline robustness problems. Would be good to include some of them in the related work such as but not limited to:\n\nPanaganti, Kishan, et al. \"Robust reinforcement learning using offline data.\" *Advances in neural information processing systems* 35 (2022): 32211-32224.\n\nRigter, Marc, Bruno Lacerda, and Nick Hawes. \"Rambo-rl: Robust adversarial model-based offline reinforcement learning.\" *Advances in neural information processing systems* 35 (2022): 16082-16097.\n\nTang, Xiaohang, et al. \"Adversarially Robust Decision Transformer.\" *arXiv preprint arXiv:2407.18414* (2024).", "other_strengths_and_weaknesses": "Paper is well-written. However, it is hard to obtain conclusion from the experiments. Need better metric.", "other_comments_or_suggestions": "For now, I will offer a weak reject. But will consider increasing the score if the concerns and questions are addressed, especially the experiments, specifically the results are hard to draw conclusion.\n\nTypos:\nAppendix C: basical procedure\nBottom of page 21: pratical procedure\nAlgorithm 1: minimiging", "questions_for_authors": "The paper tries to motivate by making the algorithm for robustness off-policy. Will this be beneficial if you are doing online RL?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A81eU2A5BK", "reviewer": "Reviewer_Vgnr", "summary": "The paper presents a novel off-policy reinforcement learning approach that addresses adversarial input observations without requiring additional environmental interactions, thus enhancing sample efficiency and avoiding inefficiencies in agent-environment interactions. By reformulating adversarial learning as a soft-constrained optimization problem, the method eliminates mutual dependencies between the agent and adversary. The approach is theoretically supported by the symmetric property of policy evaluation and shows consistent success and strong sample efficiency in evaluations, making it a promising contribution to the field.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and detailed feedback. Below, we respond to each point.\n\n---\n## Methods and Evaluation Criteria\n### About Baseline [1]  \n> [1] Reddi et al., *Robust adversarial reinforcement learning via bounded rationality curricula*, ICLR 2024\n\nThank you for the suggestion.\nWe considered adapting [1]—originally proposed for dynamics robustness—to the observation robustness setting. However, we found key theoretical mismatches. [1] trains the adversary using a separate replay buffer under stationary dynamics. Under observation perturbations, the effective dynamics become $\\mathcal{F} \\circ \\pi(s' \\mid s, \\tilde{s})$, leading to inconsistency between the adversary's buffer and the trajectory induced by the current policy, thereby violating convergence assumptions.\n\nThis reveals a structural gap: [1] assumes dynamics that receive two actions in parallel, whereas our setting involves a cascade structure (perturb, then act). Bridging this gap is an interesting direction, but it is beyond the scope of this work.\n\n---\n\n## Relation to Broader Literature\n\n> **Code Release**\n\nThank you. Given the multiple baselines and training procedures, additional time is needed for cleanup and documentation. We are committed to releasing the code with the camera-ready version.\n\n---\n\n## Weaknesses\n\n- **[W1] Missing Citations in Section 5.2:**\n\nThank you. We will revise the manuscript to include relevant citations.\n\n- **[W2] Mixture Coefficient (line 283):**\n\nOur heuristic choice was made for simplicity, but we agree it may cause miscalibration. We will add a discussion and cite works on distributionally robust optimization.\n\n---\n\n## Other Comments\n\n- **arXiv vs. Published Citations:**\n\nWe will update arXiv references to their published versions where applicable.\n\n---\n\n## Questions for Authors\n\n- **[Q1] Identifying Robust and Sample-Efficient Methods**\n\nIn Table 1, the highest scores within each on-policy or off-policy category are highlighted in bold, and the best-performing methods among the most sample-efficient are shaded in gray. Our methods consistently maintain strong robustness, especially in complex tasks (HalfCheetah, Ant), emphasizing the value of MDP-aware modeling. Following Reviewer AMCX's suggestion, we will use worst-case metrics to better capture robustness.\n\n- **[Q2] Why Robust Methods Vary Across Attacks**\n\nWe attribute this to equilibrium differences across training methods. Robust RL does not guarantee global optimality, and policies may settle into different robustness profiles depending on the training dynamics.\n\n- **[Q3] Variance in Hopper**\n\nThis is addressed in our response to Reviewer wL61.\n\n- **[Q4] When to Use VALT-EPS vs. VALT-SOFT**\n\nTwo key factors:\n\n1. **Computation**: VALT-EPS uses PGD-based attacks across two networks, requiring GPU acceleration (line 1689).\n2. **Task Sensitivity**: In failure-sensitive environments, VALT-SOFT better explores around risky observations. In tolerant domains like HalfCheetah, VALT-EPS is more effective.\n\n- **[Q5] Computation Time in Table 6**\n\nTable 6 is intended as reference, not comparison. We agree that step count affects time. In the revision, we’ll include per-update time and breakdowns to improve fairness.\n\n- **[Q6] Hyperparameter Complexity**\n\nOur method introduces fewer tunable parameters than ATLA. For example, we avoid learning adversary networks. Most hyperparameters follow SAC; new ones (e.g., mixture rate) are easier to tune. We will include hyperparameter tables for clarity as:\nhttps://gofile.io/d/8Eq728\n\n- **[Q7] Hyperparameter Tuning and Early Stopping**\n\nThank you for the question. We set the number of training steps based on when the policies achieve sufficiently high scores under non-attacked conditions. While some environments may require fewer steps, we adopted a consistent training schedule to ensure fair comparisons across methods.\n\nIn off-policy settings, longer training can help refresh the replay buffer with newer data, mitigating distribution shift and potentially improving robustness—albeit at the cost of sample efficiency. However, we did not adopt this strategy in the current work and instead consider addressing distribution shift an important direction for future research.\n\nTuning hyperparameters across multiple attacks is challenging. During training, we apply simple heuristic attacks to identify promising candidates, and later refine them under various attacks.\n\nWhile trade-offs exist, our experiments indicate that robustness tends to be more influenced by the choice of method than by hyperparameter settings. We view robust hyperparameter tuning under various attacks as an important and open challenge for future work.\n\n---\nWe hope our responses help clarify your concerns, or will be addressed in the revision. If you find the direction and potential contributions meaningful—especially as a baseline for off-policy adversarial RL—we would sincerely appreciate your consideration in revisiting the score.", "claims_and_evidence": "The claims of this work is supported by both theories and empirical results.", "methods_and_evaluation_criteria": "Yes, I think the proposed method and evaluation are reasonable. However, this paper also mentions multiple of [1], this baseline should be included. \n\n\n\n\n\n#### [1] Reddi, A., T¨ olle, M., Peters, J., Chalvatzaki, G., and D’Eramo, C. Robust adversarial reinforcement learning via bounded rationality curricula. ICLR 2024", "theoretical_claims": "I went through the theories and proofs. I did not go into details of the proofs in the appendix, but overall it sounds reasonable.", "experimental_designs_or_analyses": "Please refer to \"Questions for authors\"", "relation_to_broader_scientific_literature": "* This work opens an alternative way to implicitly learn adversarial training without two-player game. It will be better for contributing to community with the open-source code.", "essential_references_not_discussed": "I think the authors cover most of the references.", "other_strengths_and_weaknesses": "Strengths\n* well-written\n* comprehensive evaluation on different attacks\n* proposed method is impactful and concise\n* algorithms are supported with theories\n\nWeaknesses\n* In 5.2. Attacker Settings, it is mentioned that the common attack scales are used in previous studies. Please cite those works. \n* Although, this works briefly shows how to choose the mix in line 283, I still have the concern about the over-optimism/pessimism, which is pointed out from a lots of robust optimization works, e.g., [2]. Further defense or mentioning explicit limitations can enhance this work.\n\n#### [2] Juncheng Dong et al. Variational Adversarial Training Towards Policies with Improved Robustness. AISTATS 2025", "other_comments_or_suggestions": "Please avoid citing paper in arXiv version if it was accepted. For example, the paper below.\n\n\n\n\n\n#### Reddi, A., T¨ olle, M., Peters, J., Chalvatzaki, G., and D’Eramo, C. Robust adversarial reinforcement learning via bounded rationality curricula. ICLR 2024", "questions_for_authors": "* How can we tell the most robust sample-efficient method from the values shown in table 1? \n* I thought that different heuristic attacks is only for evaluation. Then what is the reason that the the most robust sample-efficient method under different attacks are different training methods?\n* Can you elaborate more why both SAC and VALT-EPS has higher variance only at the end for hopper task in figure 2?\n* Could you discuss when people should use between VALT-EPS and VALT-SOFT from both theoretical and empirical perspectives?\n* Table 6 records the computation time among different off-policy methods. In the experiments, SAC-PPO took more training steps from previous tables. Then how will be a fair comparison regarding computation time in table 6?\n* I have the concern if there increases a lot more hyper-parameters for the two variants of the proposed methods compared with existing robust RL methods.\n* It seems that each proposed method has been evaluated under different attacks. While during training, it requires hyper-parameter tuning for each method, including baselines, how do you decide what kind of hyper-parameters should be adopted and when the training can be stopped? In other words, there may happen that the learning curves perform better for hyper-parameter A while it may be more robust to some attacks in evaluation for hyper-parameter B, and maybe be more robust to the other attacks in evaluation for hyper-parameter C.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an interesting off-policy framework for robust reinforcement learning, addressing a key limitation of existing methods that require additional environment interactions to train an adversary. The reviewers generally appreciate the paper’s technical contributions, including the formulation of adversarial learning as a soft-constrained optimization problem supported by a symmetric property of policy evaluation. The proposed algorithms also show strong empirical performance across multiple challenging environments and attack settings, achieving better sample efficiency and robustness compared to prior baselines.

While some concerns were raised about the performance in clean environments, computational cost analysis, and applicability to discrete action spaces, the authors have responded thoroughly and clarified the limitations and trade-offs involved. Overall, the paper provides a valuable contribution to the robust RL literature with sound theoretical foundations, comprehensive experiments, and promising future directions.

Meanwhile, the review team would like to encourage the authors to carefully incorporate the reviewers’ constructive feedback and suggestions to further strengthen the clarity, experimental analysis, and broader applicability of their work. In particular, the authors may provide a more detailed analysis of the wall-clock time or FLOPs compared to baselines to better support their claims on computational efficiency. It would also be valuable to expand the discussion on the potential extension of the proposed framework to discrete action spaces, addressing both theoretical and practical challenges. Additionally, covering more related works suggested by the reviewers, particularly those on offline robust RL and action perturbation attacks, would help position the paper more comprehensively within the literature. Finally, the review team recommends discussing the importance of exploring hybrid or continual learning strategies as a promising direction for enhancing robustness and adaptability in dynamic or non-stationary environments.
Decision: Accept (poster)