Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations
Authors: Yongyuan Liang
Yanchao Sun
Ruijie Zheng
Xiangyu Liu
Benjamin Eysenbach
Tuomas Sandholm
Furong Huang
Stephen Marcus McAleer
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Robustness
Adversarial Learning
Primary area: reinforcement learning
TL;DR: We introduce a temporally-coupled adversary considering the temporal coupling between perturbations over time and propose a game-theoretic response approach for adversarial defense against these adversaries.

Abstract: Deploying reinforcement learning (RL) systems requires robustness to uncertainty and model misspecification, yet prior robust RL methods typically only study noise introduced independently across time. However, practical sources of uncertainty are usually coupled across time.
We formally introduce temporally-coupled perturbations, presenting a novel challenge for existing robust RL methods. To tackle this challenge, we propose GRAD, a novel game-theoretic approach that treats the temporally-coupled robust RL problem as a partially-observable two-player zero-sum game. By finding an approximate equilibrium within this game, GRAD optimizes for general robustness against temporally-coupled perturbations. Experiments on continuous control tasks demonstrate that, compared with prior methods, our approach achieves a higher degree of robustness to various types of attacks on different attack domains, both in settings with temporally-coupled perturbations and decoupled perturbations.

GAME GHERRTEEO ROBUST RL
HANDLES
HHIPORALLY-CORRLEE PERTURBATIONS
Yongyuan Liang
Yanchao Sun
Ruijie Zhengi
Xiangyu Liu Stephen McAleer! $ Princeton University
Benjamin Eysenbach
Tuomas Sandholm Furong Huang!
f University of Maryland, College Park
Carnegie Mellon University
ABSTRACT
Deploying reinforcement learning (RL) systems requires robustness to uncertainty and model misspecification, yet prior robust RL methods typically only study noise introduced independently across time. However, practical sources of uncertainty are usually coupled across time. We formally introduce temporally-coupled pertur- bations, presenting novel challenge for existing robust RL methods, To tackle this challenge, we propose GRAD, novel game-theoretic approach that treats the temporally-coupled robust RL problem as a partially-observabbl two-player zero-sum game. By finding an approximate equilibrium within this game, GRAD optimizes for general robustness against cemporally-couples perturbations. Experi- ments on continuous control tasks demonstrate that, compared with prior methods, our approach achieves a higher degree of robustness to various types of attacks on different attack domains, both in settings with semporally-couplee perturbations and decoupled perturbations.
INTRODUCTION
In recent years, reinforcement learning (RL) has demonstrated success in tackling complex decision- making problems in various domains. However, the vulnerability of deep RL algorithms to test- sit- changes in the environment or adversarial attacks has raised concerns for real-world applications. Developing robust RL algorithms that can defend against these adversarial attacks is crucial for the safety, reliability and effectiveness of RL-based systems.
In most existing research robust RL (Huang et al., 2017; Liang et al., 2022; Sun et al., 2022; Tessler et al., 2019; Zhang al., 2020), the adversary is able to perturb the observation or action every timestep under static constraint. Specifically, the adversary's 's perturbations are constrained within a predefined space, such as an Lp norm, which remains unchanged from one timestep to the next, This standard assumption in the robust RL literature can be referred to as assumption. However, this static constraint can lead to unrealistic perturbations: for example, the attacker may be able blow the wind hard southeast time but northwest at time + 1. Providing robustness against such an perturbations may result in an overly conservative policy.
However, the set of perturbations faced in the real world are typically remporally-couleed if the wind blows in one direction at one time step. it will likely blow in a similar directly at the next step. partially-observable two-player game and
In this paper, we will treat the robust RL problem use tools from game theory to acquire robust policies, both for the con-temporally-copple and the temporally-couple settings.
In this paper, we propose a novel approach: Game-theoreds Response approach for Adversarial Defense ( (GRAD) that leverages Policy Space Response Oracles (PSRO) (Lanctot et al., 2017) for robust training. GRAD is more general than prior adversarial defenses in the sense that it does not target certain adversarial scenarios and converges to the approximate equilibrum training with an adversary policy set. While prior methods often assume the worst case and aim improve against them, they lack adaptability to specific attacks such as these adversaries under temporally-coupled constraints. We formulate the interaction between the agent and the cemporally-couple adversary as a two-player zero-sum game and employ PSRO to ensure the agent's 's best response against the learned
"Corresoonding author, cheryun! @umd edu

---

Standard Attacks
Temporally-coupled Attacks
Our robust model: more robust and stable under different types of attacks
Robust baselines: robust under standard attacks, fall down under the temporaliy-coupled attacks
Figure The robust GRAD agents (top) and the statc-of-the-ar robust WocaR-RLL (Liang et al., 2022) (bottom) exhibit distinct learned behaviors. Under standard non-temporally-couphhe attacks, both agents maintain basic body stability, with the GRAD agent making effort avoid lateral rotations. Notably, WocaR-RI RL focuses on enhancing robustness in worst-case scenarios, but experiments reveal its vulnerability temporally-coupled attacks, leading to tendency to fall towards one side. contrast, GRAD showcases superior robustness both non-tenn caappporall--ouplee and temporally coupled adversarial settings.
adversary and find an approximate equilibrium. This game-theoretic framework empowers our approach to effectively maximize the agent's 's performance by adapting to the adversary's 's strategies.
Our contributions are three-fold. First, we propose a novel class of temporally-couple attacks to identify the realistic pitfalls of prior threat models challenge for existing robust RL methods. Second, we introduce a game- theoretic response approach, referred to as GRAD. We high- light the significant advantages of GRAD in terms of convergence and policy exploitability. Notably, GRAD demonstrates adaptability to adversaries in both temporally-cous colllyy-cupeedd and non-temponn non-temporally- coupled settings. Furthermore, GRAD serves as versatile and flexible solution for adversarial RL, enhancing robustness against diverse types of adversaries.
adversarial
Third, provide empirical results that demonstrate the effectiveness of approach in defending against both cemporally-couple and non-temporally coupled adversaries on various attack domains. Figure illustrates how robustness temporally-coupler perturbations induces different behavior than robustness to standard perturbations.
PRELIMINARIES
Notations and Background. A Markov decision process (MDP) can be defined as tuple (S, A, P, R. 7), where and A represent function: R S A R, P S A A(S) represents the set of probability distributions over the state space S and 7 E 10, is the discount factor. The agent selects actions based on its policy, S A(A), which is represented by function approximator (e.g. neural net- work) that is updated during training and fixed during testing. The value function is denoted by (s) := P 12>00= 7 HR(ss,, ar) so = s). which measures the expected cumulative discounted reward that an agent can obtain from state s E S by following policy π.
state space and the action space, R. is the reward
State Action Adversaries. State adversary is type of test-time attacker that perturbs the agent's state observation returned by the environment at each time step and aims to reduce the expected episode reward gained by the agent. While the input the agent's policy perturbed. the underlying state in the environment remains unchanged. State adversaries, such as those presented in (Zhang al., 2020; 2021; Sun et al., 2022), typically consider perturbations on a continuous state space under certain attack budget c. The attacker perturbs state 8 into E B, (s), where (s)s) is a Ep norm ball centered at s with radius e. Moreover, Action adversaries' goal is to manipulate the behavior of the agent by directly perturbing the action executed by the agent to ā with the probability a as uncertainty constraint before the environment receives it (altering the output of the agent's 's policy), causing it to deviate from the optimal policy (Tessler et al., 2019). In this paper, we focus solely on continuous-space perturbations and employ an admissible action perturbation budget as a commonly used f, threat model.
2

---

Algorithm Policy Space Response Oracles (Lanctot et al., 2017)
Result: Nash Equilibrium
Input: Initial population IT
repeat (for & 0,
TT <- NE in game restricted to strategies in II
for i E {1, do
Find best response 3, (R,(
11 11 U {
end for
until Approximate exploitability is less than or equal to zero Return:d x²'
Problem formulations as zero-sum game. We model the game between the agent and the adversary as a two-player zero-sum game that is a tuple (S, Ila, II, P, RR,), where II, and II, denote the sets of policies for the agent and the adversary, respectively. In this framework, both the transition kernels P and the reward function R of the victim agent depend on not only its own policy Ta II, but also the adversary's policy Nr II.. The adversary's 's reward R( az) is defined as the negative of the victim agent's ' reward R(st, 0z). reflecting the zero-sum nature of the game. The expected value u.- (h) for the agent is the expected sum of future rewards in history h and the robuat RL problem as two-player zero-sum game has u,-" (h) + u (f) O for all agent and adversary strategies. Nash equilibrium (NE) is strategy profile such that, if all players played their NE strategy, no player could achieve higher value by deviating from it. Formally, T is a NE if ua ( maxx, (Ta ).) best response BE strategy BI (Te) for the agent a to a strategy ne is a strategy that maximally exploits म, (B,,,.T)) arg max,, Un (Ta: T.). In this paper, our goal to converge to the approximate NE for the zero-sum game.
Double Oracle Algorithm (DO) and Policy Space Response Oracles (PSRO). Double ora- cle (McMahan et al., 2003) is an algorithm for finding Nash Equilibrium (NE) in normal-form games. The algorithm operates by keeping population of strategies IIe at time t. Each iteration, NE ** & is computed for the game restricted to strategies in IT'. Then, a best response BR, (RR() to this NE is computed for each player and added to the population, II II; (x for i {1.2). Although in the worst case DO must expand all pure strategies before ' converges to an NE in the original game, in many games DO terminates early and outperforms alternative methods. interesting open problem is characterizing games where DO will outperform other methods.
Policy Space Response Oracles (PSRO) (Lanctot et al., 2017; Muller et al., 2019; Feng et al., 2021; McAleer et al., 2022b;a), shown Algorithm are a method for approximately solving very large games. PSRO maintains population of reinforcement learning policies and iteratively trains the best response to mixture of the opponent's 's population. PSRO is a fundamentally different method than the previously described methods in that in certain games it can be much faster but in other games it can take exponentially long in the worst case.
ROBUSTNESS TO TEMPORALLY-COUPLE. ATTACKS
In this section, we first formally define cemporally-couplee attacks. Then, we introduce our algorithm, a tane-theoreetc response approach for adversarial defense against the proposed attacks.
3.1 TEMPORALLY-COUPLIS ATTACK
Robust and adversarial RI. methods restrict the power of the adversarial by defining a set of admissible perturbations:
Definition 3.1 (e-Admissible Adversary Perturbations). An adversarial perturbation pt. is considered admissible in the context of a state adversary if. for a given state sı at timestep t, the perturbed state s, defmeed as 3, St + Pt satisfies ||s, &ell S c, where is the state budget constraint. Similarly, if Pt. is generated by an action adversary. the perturbed action ā, defined as à = a, + Pt should be under the action constraint I|at ăell
3

---

While the budget constraint is commonly applied in prior adversarial attacks, it may not be applicable in many real-world scenarios where the attacker needs to consider the past perturbations when determining the current perturbations. Specifically, in the temporal dimension, perturbations exhibit certain degree of correlation. To capture this characteristic, we introduce the concept of temporally- coupled attackers. We propose a temporally-couples constraint defined in Definition 3.2, which sets specific limitations on the perturbation at the current timestep based on the previous timestep's perturbation.
Definition 3.2 (E- reeporally-coupppe
Perturbations). A temporalyy-cuuple state perturbation Pt is deemed acceptable if it satisfies the temporally constraint E: || 5t ($1+1 $ I ē where &e and 8t+1 are the perturbed states obtained by adding Pt and p++1 to 8t and 8441. respectively. For action adversaries, the temporally-coupiee constraint is similarly denoted as ||at - 0e - (0c+1 -a++1)l S E, where ūt and u++1 are the perturbed actions.
We illustrate this definition in Fig. 2. When an ad- versary is subjected to both of these constraints, it is referred to as a temporally-coupe this paper. For a camporallyy-copplee adversary, each timestep's 's perturbation is restricted within certain range e, similar to other regular adversarial attacks. However, it is further confined within a smaller range é based on the previous timestep's 's perturbation. This design offers two significant benefits.
dee1
adversary in
6
Standard Perturbations
Temporally-coupled
Perturbations
Firstly, it enables the adversary to consider the tem- poral coupling between perturbations over time. By constraining the perturbations to smaller range and discouraging drastic changes in direction, the adversary can launch continuous and stronger attacks while preserving a certain degree of stability. Intuitively, if the adversary consistently attacks in one direction, it can be more challenging for the victim to preserve balance and defend effectively compared to when the perturbations alternate between the left and right directions.
Figure 2:
Standard perturbations and temporally- coupled perturbations in 2d example.
Then, the semporally-coulle attack strategy by narrowing down the range of choices for each timestep's perturbation. Reducing the search space does not necessarily weaken the adversary; in fact, it can potentially make the adversary stronger if the optimal attack lies within the temporally -ettrmllly search space, which is supported by our empirical results. By constraining the adversary to a more focused exploration of attack strategies, the cimporally-couplle constraint facilitates the discovery and exploitation of more effective and targeted adversarial tactics that exhibit less variation at consecutive timesteps. This characteristic enhances the adversary's ability launch consistent and potent attacks.
constraint also enables the adversary to efficiently discover the optimal
Practically, is crucial to carefully determine to guarantee that this additional cemporally-coupler constraint does not impede the performance of attacks but rather amplifies their effectiveness. The effectiveness of different choices for was empirically evaluated in our empirical studies, highlighting perturbations represent novel
the benefits brings adversarial learning. cemporally-couplee case to challenge the existing methods. Our empirical experiments demonstrate that, even with the introduction of temporally-come coupled constraints, these perturbations can have a notable impact on existing robust models, showcasing the need for addressing such scenarios in robust RI..
3.2 GRAD: GAME-THEORER APPROACH FOR ADVERSARIAL DEFENSE
Building upon prior robust RL methods, we develop a robust RL algorithm for the cemporally-couplee setting. Our resulting method uses tools from game theory to enhance robustness against adversaries with different settings including non-temporallyyccoplee and temporally-coupler constraints.
In our Game-theoretic Response approach for Adversarial Defense ( (GRAD) framework as mod- ification PSRO (Lanctot et al., 2017), an agent and a temporally-coupler adversary are trained as part of two-player game. They play against each other and update their policies in response to each other's 's policies. The adversary is modeled as a separate agent who attempts to maximize the impact of attacks on the original agent's performance and whose action space is constrained by both e and ε. Our method adapts the e-budget assumption from prior work (Liang et al., 2022) to handle temporally-coupled constraints. Meanwhile, the original agent's objective function is based
4

---

Algorithm 2 Game-theoretic Response approach for Adversarial Defense (GRAD)
Initial policy sets for the agent and adversary II {Ila Hot Compute expected utilities empirical payoff matrix U for each joint π {Tas Tob E 11 Compute meta-Nash equilibrium da and or over policy sets (Ia, II,)) for epoch in (1, do for many iterations Nr, do
Input:
Sample the adversary policy n. dn:
Train To with trajectories against the fixed adversary To: Dag == of 8842))1
(when the fixed adversary only attacks the action space, &t 84.)
end for
II. = II (
for many iterations N=, do
Sample the agent policy ma da
Train the adversary policy ni with trajectories: Dr; := (s at +f+))(+))) )==
(r) applies attacks to the fixed victim agent Ta based on a using different methods)
end for
11 HL {
Compute missing entries in U from
Compute new meta strategies oa and og from U
end for
Return: current meta Nash equilibrium whole population da and de
on the reward obtained from the environment, taking into account the perturbations imposed by the adversary. The process continues until approximate equilibrium is reached, at which point the original agent considered to be robust to the attacks learned by the adversary. We show our full algorithm in Algorithm 2.
Under some assumptions (see Appendix A for details), GRAD convergence to an approximate Nash Equilibrium (NE):
Proposition 3.3. For to an approximate Nash Equilibrium (NE) of the two-player zero-sum adversarial game.
MDP with fixed number of discrete actions, GRAD converges
In GRAD, both the agent and the adversary have two policy sets. During each training epoch, the agent aims to find an approximate best response to the fixed adversary, and vice versa for the adversary. This iterative process promotes the emergence stable and robust policies. After each epoch, the new trained policies are added to the respective policy sets, which allows for a more thorough exploration of the policy space.
For different types of attackers, the agent generates different trajectories while training against fixed attacker. If the attacker only targets the state, then the agent's training data will consist of the altered state å after adding the perturbations from the fixed attacker. If the attacker targets the agent's action, the agent's 's policy output will be altered as â by the attacker, even if the agent receives the correct state s during training. As for the adversary's training, after defining the adversary's 's attack method and policy model, the adversary applies attacks to the fixed agent and collects the trajectory data and the negative reward train the adversary. The novelty of GRAD lies its scalability and adaptability in robust RL which converges to the approximate equilibrium without only considering certain adversarial scenarios. Our work formulates the robust RL objective as zero-sum game and demonstrates the efficacy of game-theoretic RI. in tackling this objective, rather than being solely reliant on the specific game-theoretic RI algorithm of PSRO or focusing on defense against specific types of attackers. GRAD provides more versatile and adaptive solution.
As Definition 3.2, which is actually a generalization of the original attack space. We have ||st 5; (s++1 1el1)ll < ||s, 3e + ||s++11 š < 2c, when E > 2e, it converges to the non-coupled attack scenario. Consequently, our defense strategy not specific to narrow attack set In the next section, empirically demonstrate that our approach exhibits superior and comprehensive robustness, which is capable of adapting to various attack scenarios and effectively countering different types of adversaries continuous control tasks.
5

---

Robusiness under Strongest Non-tempenlllcccreee Siate Aftecks
Robusiness under Stroggett coopooii--oopngg Stats Altacks
PAATLA-PFO
WocaR-PPO
PA-ATLA-PPO PAATLA/FPO
WocdR.PPO
Eqiedd (cenwat
5000
PAATLA-FFO'
GRAD
5000
GRAD (Durs)
2000
4000
5000
5000
(a) Non-teemeee con-temporally temporalll--Coulling State Attacks
(b) Temporally-Coupling State Attacks
Figure 3: Average episode rewards + standard deviation over 100 episodes under the strongest non temporally- coupling and caaporally-coupliin state attacks for state robust baselines and GRAD on five control tasks.
4 EXPERIMENTS
In our experiments, we investigate various types of attackers on different attack domains including state perturbations, action perturbations, model uncertainty and mixed perturbations. We will study diverse set of attack and compare with state-of-the-art baselines.
Our experiments are conducted on five various and challenging MuloCo environments: Hopper, Walker2d, Halfcheetah, Ant, and Humanoid, all using the v2 version of MuJoCo. We the Proximal Policy Optimization (PPO) algorithm the policy optimizer for GRAD training. For attack constraint e, we use the commonly adopted values for each enviromment. We set the cemporally-couplee constraint c/5 (with minor adjustments in some environments). Ablation experiments study the choice ofOther choices will be further discussed in the ablation studies. Our experiments conducted five various and challenging MuJoCo environments: Hopper, Walker2d, Halfcheetah, Ant, and Humanoid, all using the v2 version of MuJoCo. We the Proximal Policy Optimization (PPO) algorithm the policy optimizer for GRAD training. For attack constraint we use the commonly adopted values for each environment. We set the temporally-coupp constraint c/5 (with minor adjustments in some environments). Ablation experiments study the choice of c.
Experiment setup.
We report the average test episodic rewards both under no attack and against the strongest adversarial attacks to reflect both the natural performance and robustness of trained agents, by training adversaries targeting trained agents from scratch. For reproducibility, we train each agent configuration with 10 seeds and report the one with the median robust performance, rather than the best one. More implementation details are in Appendix C.I.
Case I: Robustness against state perturbations. In this experiment, our focus is on evaluating the robustness of our methods against state adversaries that perturb the states received by the agent. Among the alternating training (Zhang et al., 2021; Sun et al., 2022) methods, PA-ATLA-PPPO is the most robust, which trains with the standard strongest PA-AD attacker. As modification, we train PA-ATLA-PPO* with a temporally-couple PA-AD attacker. WocaR-PPO (Liang et al., 2022) the state-of-the-ar defense method against state adversaries. Our GRAD method utilizes the temporally-couph PA-AD attacker for training. Figure 3 presents the performance of baseline and GRAD under both non-tem cempoorally-couplee and temporally-couplet state perturbations.
Despite being trained handle temporally-couple adversaries, method also demonstrates strong performance in the non-robust ("natural") setting, expecially on the nigh-dimensional Humanoid task. Under temporally-coupled attacks, the average performance of GRAD is 45% higher than the strongest baseline.
Case I: Robustness against action uncertainty. Beyond assessing the susceptibility of GRAD to state attacks, we also investigate its robustness against action uncertainty, where the agent intends to execute action but ultimately takes a different action than anticipated. We scrutinize two specific forms of action uncertainty, as outlined in prior work (Tessler et al., 2019). The first one is action perturbations, introduced by action adversary, which strategically adds noise the agent's intended action. The second scenario revolves around model uncertainty, where, with probability denoted a, alternative action replaces the originally planned action output by the agent. These

---

Robusiniss under Strongest Non-tampormllco..ee wwwppoooa-oooooddaooo Action Aftaces
Roousthess under Strongest Temperaal-cuptiee cocplliioopiinggg.. Action Attokks
WocoR, GHAD (O.m)
Eqieoda (cenwath
5000
CHAD (Gur))
5000
MC-UP
2000
4000
5000
5000
(a) Non- Cemporalyy-Couppeee Action Attacks
(b) Temporally-Coupling Action Attacks
Figure 4: Average episode rewards + standard deviation over 100 episodes for GRAD and action robust models
against the strongest non--eepporally-couulee and camporally-coupled action perturbations on five MuJoCo tasks.
L100pe Bicce
Jabeatit San soleos uoachinz
Jumaacoc: athert SOOK JOOOcO
Mhace
1100
(a) Hopper
(b) Walker2d
(c) Halfcheetah
(d) Ant
(e) Humanoid
Figure 5: Robustness to Model Uncertainty Across Various a Values. The noisy probability represents the
likelihood of a randomly sampled noise replacing the initially selected action.
scenarios closely parallel real-world control situations, such as dealing with mass uncertainty (e.g.. when a robot's weight changes) or facing sudden, substantial external forces (c.g.. when an external force unexpectedly pushes a robot).
In our baseline comparisons, we include PR-MDP and NR-MDP (Tessler et al., 2019), which are robust to action noise and model uncertainty. We also incorporate WocaR-PPO into our baseline evaluations. We train GRAD using temporally-coupled action adversary and evaluate its robustness in both action perturbation and model uncertainty scenarios.
Action Perturbations. To obtain a stronger evasion action perturbation rather than OU noise and parameter noise, we are the first to train an RL-bbased action adversary following the trajectory outlined in Algorithm This strategy aims showcase the worst-case performance of our robust agents under action perturbations. For evaluation, we train both temporally-coupled and non-temporally-cuppe action adversaries for each robust model. In Figure 4, we present the exceptional performance of GRAD against standard and cemporally-coupliin action perturbations. GRAD demonstrates high degree robustness. For example, on the Humanoid task it outperforms the basclines by a 17% margin for standard attacks and by 40% advantage against temporally-coupling action attacks. These results provide evidence of GRAD's defense mechanism against various types of adversarial attacks in the action space.
To evaluate robustness under model uncertainty, we consider a range of noise probabilities denoted as in the range of [0, 0.05, 0.1, 0.15, 0.2]. These values represent the probability of randomly generated noise replacing the action selected by the victim agent. As depicted in Figure GRAD exhibits superior robustness compared action-robust baselines across spectrum of a uncertainty value without explicit exposure to model uncertainty noises during training.
Model Uncertainty.
Case III: Robustness against mixed adversaries. In prior works, adversarial attacks typically focused on perturbing either the agent's observations or introducing noise to the action space. However, in real-world scenarios, agents may encounter both types of attacks simultaneously. To address this challenge, agent's state and action at each time step. We employ alternating training to create a baseline as Mixed-ATLA using this mixed adversary type. Our GRAD model and Mixed-ATLA are trained with semporally-coupler mixed attackers. The detailed algorithm for the mixed adversary is provided in Appendix 5.
propose mixed adversary, which allows the adversary to perturb the
Our results in Figure 6 indicate that the combination of two different forms of attacks can target robust agents in most scenarios, providing strong evidence of their robustness. GRAD outperforms other
7

---

Robusiness under Strongest Non-demooraay-ooppeeee Attecks
Robustnoss under Stroncest "smporally- Mived Attacks
Vited-ATLA
WooaR-PPO
GRAD (Olus)
Mited-ATLA
WodaR-PPO
GRAD (Curs)
6000
3000
5000
Pee Reesye
2000
1000
(a) Non-temporally CopplyylppalyyyCoupiinggg Mixed Attacks
(b) Campooraly--Cuplhie Mixed Attacks
Figure 6: Average episode rewards + standard deviation of GRAD and baselines over 100 episodes under the strongest www.eeeoooallycoopplli and temporally cooplliin mixed attacks.
(a) Hopper
Walker2d
(c) Halfcheetah
(d) Ant
(e) Humanoid
Figure GRAD achieves higher returns in the robust setting without sacrificing performance in the non-robust ("natural") setting.
methods all five environments against peeeporrllyycoupleee mixed adversaries, with a margin of over 20% in the Humanoid environment. Moreover, when defending against temporally-couple mixed attacks, GRAD outperforms baselines by 30% in multiple environments, with a minimum improvement of 10%.
Natural Performance. We also evaluate the natural performance of GRAD and the baselines, as shown in Figure 7, which compares natural rewards vs. rewards under the strongest temporally- coupled attacks. It is evident that while achieving robustness, GRAD maintains a comparable natural performance with the baselines; the agent's performance does not degrade significantly in environments without adversaries. The natural performance comparing GRAD with action-robust models can be found in Appendix C.5.
Ablation studies for temporally coupled constraint E. As defined in our framework, the temporally-coupled constraint ε limits the perturbations within range that varies between timesteps. When set too large, the constraint becomes ineffective, resembling setting E close to zero overly restricts perturbations, leading to decline in attack performance. An appropriate value for ε is critical for effective temporally-couple attacks.
WocaR-PPO
GRAD (Durs)
Reserrds tomporl y
yccupted
standard attacker. Conversely,
Figure illustrates the performance of robust models against temporally-coupled state attackers trained with different attacker
2000
maximum é. For PPOaR-PPO. the temporally-couphe achieves good performance when the values of E are set to 0.02.
Temporaly-coo cocupled constraints
the values increase and the compoedlly-ooppled constraint weakens, the agent's performance improves, indicating decrease the adversary's attack effectiveness. the case of GRAD agents, they consistently maintain robust performance as the E values become larger. This observation highlights the impact of temporal coupling on the vulnerability of robust baselines to such attacks. In contrast, CRAD agents consistently demonstrate robustness against these attacks.
Figure 8: Ablated studies for E.
RELATED WORK
Robust RL against adversarial perturbations. Existing defense approaches for RL agents primarily designed to counter adversarial perturbations in state observations. These methods encom- pass wide range of strategies, including regularization techniques (Zhang al., 2020; Shen et al.,
8

---

2020; Oikarinen et al., 2021), attack-driven approaches involving weak strong muddeett-based attacks (Kos & Song, 2017; Behzadan & Munir, 2017; Mandlekar et al., 2017; Pattanaik et al., 2018; Franzmeyer et al., 2022; Vinitsky et al., 2020), RL-based alternating training methods (Zhang et al., 2021; Sun et al., 2022), and worst-case motivated methods (1 (I.ing et al., 2022). Furthermore, there is line of research that delves into providing theoretical guarantees for adversarial defenses in RL (Lütjens et al., 2020; Oikarinen et al., 2021; Fischer et al., 2019; Kumar et al., 2022; Wu et al., 2022; Sun et al., 2023), exploring variety of settings and scenarios where these defenses can be attacks can take various forms. For instance, perturbations can affect the actions executed by the agent (Pan et al., 2022; Tessler et al., 2019; Lanier al., 2022; Lee et al., 2020). Additionally, the study of adversarial multi-agent games has also received attention (Gleave et al., 2020; Pinto et al., 2017).
effectively Applied.Aversartii
Robust Markov decision process and safe RL. There are several lines of work that study RL under safety/risk constraints (Heger, 1994; Gaskett, 2003; Gareia & Fernández, 2015; Bechtle et al., 2020; Thomas et al., 2021) or under intrinsic uncertainty of environment dynamics (Lim et al., 2013; Mankowitz et al., 2020). In particular, several works discuss coupled or non-rectangular uncertainty sets, which allow less conservative and more efficient robust policy learning by incorporating realistic conditions that naturally arise in practice. Mannor et al. (2012) propose to model coupled uncertain parameters based on the intuition that the total number of states with deviated parameters will be small. Mannor et al. (2016) identify "k-rectangular" uncertainty sets defined by the cardinality of possible conditional projections of uncertainty sets, which can lead more tractable solutions. Another recent work (Goyal & Grand-Clement, 2023) proposes to model the environment uncertainty with factor matrix uncertainty sets, which can efficiently compute robust policies.
Two-player zero-sum games. There are a number of related deep reinforcement learning methods for two-player zero-sum games. CFR based techniques such as Deep CFR (Brown et al., 2019), DREAM (Steinberger et al., 2020), and ESCHER (McAleer et al., 2023). use deep reinforcement learning to approximate CFR. Policy-gradient techniques such as NeuRD (Hennes et al., 2020), Frietion-FoRel. (Perolat et al., 2021; 2022), and MMD (Sokota et al., 2022), approximate Nash cquilibrium via modified actor-critic algorithms. Our robust RL approach takes the double oracle techniques such as PSRO (Lanctot et al., 2017) as the backbone. PSRO-based algorithms have been shown to outperform the previously-mentinee
algorithms in certain games (McAleer et al., 2021).
A more detailed discussion of related works in robust RL and game-theoretie RL are in Appendix B.
6 CONCLUSION AND DISCUSSION
Motivated by the perturbations that arise in real world scenarios, introduce new attack model for studying deep RL models. Since existing robust RL methods usually focus on a traditional threat model that perturbs state observations or actions arbitrarily within an Lp norm ball, they become too conservative and can fail to perform a good defense under the cemporally-coupeed attacks. In contrast, we propose game-theoreticn against attacks with various constraints including temporally-coupled ones. Experiments across range of continuous control tasks underscore the good performance of our approach over previous robust RL methods for both temporally-couplee attacks and temporally-coupe attacks across diverse attack domains.
response approach GRAD, which finds the best response
The current PSRO-based approach may require several iterations to converge to the best response, which can pose limitations when computational resources are constrained. We leverage distributed RL tools to expedite the training of RL agents within GRAD, enabling efficient learning of the best response. Detailed computational cost analysis can be found in Appendix C.6.
Limitations.
Regarding scalability concerns, we have demonstrated the GRAD in addressing robust RL problems on hign-dimensional tasks. In principle, alternative game-theoretic algorithms (Perolat et al., 2022), known for their practical efficiency, can be considered for defense in different game scenarios. As part of our future research directions, we plan to explore methods to further enhance the scalability of GRAD. This exploration may involve harnessing parallel training techniques and drawing insights from other scalable PSRO approaches (McAleer et al., 2020; Lanctot et al., 2017). Additionally, we aim extend the applicability of our method to pixel-based RL scenarios and real-world situations with increased practicality and complexity.

---

ACKNOWLEDGEMENTS
Liang Zheng, Liu and Huang are supported by National Science Foundation NSF-IIS-214722 FAI, DOD-ONR-Offic of Naval Research under award number N00014-22-1-2335, DOD-AFOSR-Air Force Office of Scientific Research under award number FFA9550-23-1-0048, DOD-DARPA-Defense Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD) HR001 12020007, Adobe, Capital One and JP Morgan faculty fellowships. McAleer is funded by NSF grant #2127309 to the Computing Research Association for the CIFellows 2021 Project.
REFERENCES
Sarah Bechtle, Yixin Lin, Akshara Rai, Ludovic Righetti, and Franziska Meier. Curious ilqr: Resolving uncertainty in model-based rl. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 162-171. PMLR, 30 Oct-01 Nov 2020. URL https: //proceedings mlr press /
00 wew.:tthhhhhheeeeeeeeeeeeee .a.00a.hhm....... html.
Vahid Behzadan and Arslan Munir. Whatever does not kill deep reinforcement learning, makes stronger. CoRR,
2017. 9, 15
Noam Brown and Tuomas Sandholm. Libratus: The superhuman Al for no-limit poker. In IJCAI, pp- 5226-5228, 2017a. 16
Noam Brown and Tuomas Sandholm. Safe and nested subgame solving for imperfect-information games. Advances in neural information processing systems, 30, 2017b. 16
Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats
top professionals. Science, 359(6374):418 2018. 16
Noam Brown, Tuomas Sandholm, and Brandon Amos. Depth-limited solving for imperfect- information games. Advances in neural information processing systems, 31, 2018. 16
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret mini- mization. In International Conference on Machine Learning, pp. 793-802, 2019. 9, 16
Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforcement learning and search for inperrect-infoomatioon games. Advances in Neural Information Processing Systems, 33: 13:17797-17069 2020. 16
Neil Burch, Michael Johanson, and Michael Bowling. Solving imperfect information games using decomposition. In Twenty-eighth AAAI conference on artificial intelligence, 2014. 16
Yonathan Efroni, Chi Jin, Akshay Krishnamurthy, and Sobhan Miryoosefi. Provable reinforcement learning with short-term memory. ICML, volume 162 of Proceedings of Machine Learning Research, pp. 5832-5850. PMLR, 2022. 18
Xidong Feng, Oliver Slumbers, Yaodong Yang. Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, and Jun Wang. Discovering multi-agent auto-curricula in two-player zero-sum games. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Mare Fischer, Matthew Mirman, Steven Stalder, and Martin T. Vechev. Online robustness training for deep reinforcement learning. CoRR, abs/1911.00088,, 2019. 9, 15
Tim Franzmeyer, Stephen McAleer, João F Henriques, Jakob N Foerster, Philip HS Torr, Adel Bibi, and Christian Schroeder de Witt. Illusory attacks: Detectability matters in adversarial attacks on sequential makers... arXiv preprint 22t.::2207.10100000 2022. 15
Haobo Fu, Weiming Liu, Shuang Wu, Yijia Wang, Tao Yang, Kai Li, Junliang Xing, Bin Li, Bo Ma, Qiang Fu, and Yang Wei. Actor-critic policy optimization in large-scale imperfect-infeo informattnn game. In Proceedings of the Tenth International Conference on Learning Representations (ICLR), 2022. 16
10

---

Javier Garcia and Fernando Feránndez. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015. 9
Chris Gaskett. Reinforcement learning under cireumstances beyond its control. In International Conference Computational Intelligence for Modelling Control and Automation,, 2003. 9
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations, 2020. 9
Vineet Goyal and Julien Grand-Clemen Robust markov decision processes: Beyond rectangularity.
Mathematics of Operations Research, 48(1):203- 2022222,,,,, 2023. 9
Jonathan Gray, Adam Lerer, Anton Bakhtin, and Noam Brown. Human-level performance no-press diplomacy via equilibrium search. In International Conference Learning Representations, 2020. 16
Audrūnas Gruslys, Mare Lanctot, Rémi Munos, Finbarr Timbers, Martin Schmid, Julien Perolat, Dustin Morrill, Vinicius Zambaldi, Jean- can-Baatiste Lespiau, John Schultz, et al. The advantage regret-matching actor-critic. arXiv preprint arXiv:2008. 12234, 2020. 16
Matthias Heger. Consideration of risk in reinforcement learning. In International Conference on Machine Learning, 1994. 9
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect- information games. arXiv preprint arXiv:1603.0112,, 2016. 16
Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei, Rémi Munos, Julien Perolat, Mare Lanctot, Audrunas Gruslys, Jean- Baptiste Lespiau, Paavo Parmas, Edgar
et al. Neural
replicator dynamics: Multiagent learning via hedging policy gradients. In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Systems, pp. 492-501, 2020. 16
Peide Huang, Mengdi Xu, Fei Fang, and Ding Zhao. Robust reinforcement learning as stackelberg game via reguulielyyregullarizedd adversarial training. In IJCAI, pp. 3099-3106. ijcai.org, 2022. 16
Sandy H. Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. Adversar- ial attacks on neural network policies. In International Conference on Learning Representa- tions (tons(Workshop), 2017.
Garud N. Iyengar. Robust dynamic programming. Math. Oper. Res., 30(2):257-280, 2005. 15
Ezgi Korkmaz. Investigating vulnerabilities of deep neural policies. In Uncertainty in Artificial Intelligence, pp. 1661-1670. PMLR, 2021. 15
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. In International Conference Learning Representations (octittttttiosooosooooooooo 2017. 9, 15
Aounon Kumar, Alexander Levine, and Soheil Feizi. Policy smoothing for provably robust reinforce-
ment learning. In International Conference on Learning Representations, 2022. 9, 15
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theo approach to multiagent reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2017. 1, 3,4,4.9 9
JB Lanier, Stephen McAlcer, Pierre Baldi, and Roy Fox. Feasible adversarial robust reinforcement learning for underspecified environments. arXiv preprint arXiv:2207.0959/) 2022. 9
Xian Yeow Lee, Sambit Ghadai, Kai Liang Tan, Chinmay Hegde, and Soumik Sarkar. Spatiotempo- rally constrained action space attacks on deep reinforcement learning agents. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 4577-4584 2020. 9
Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang. Li Zhao, Tao Qin, Tie- Yan Liu, and Hsiao- Wuen Hon. Suphx: Mastering mahjong with deep reinforcement learning. arXiv preprint 2003. /3590, 2020. 16

---

Yongyuan Liang. Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial training without attacking: cos---case-aware robust reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:/ openreview. net/ forum? f Blyt-llttoog n. 1, 2, 4, 6, 9, 15
Shiau Hong Lim, Huan Xu, and Shie Mannor. Reinforcement learning in robust markov decision processes. Advances in Neural Information Processing Systems, 26:701-709, 2013. 9
Weiming Liu, Bin Li, and Julian Togelius. Model-free neural counterfactual regret minimization with bootstrap learning. IEEE Transactions on Games, 2022. 16
Bjöm Lütjens, Michael Everett, and Jonathan P How. Certified adversarial robustness for deep reinforcement learning. In Conference on Robot Learning, pp. 1528-1337) PMLR, 2020. 9,
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically-plausinne perturbations. In 2017 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3932-3939, IEEE, 2017.
9, 15
Daniel J. Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Yuanyuan Shi, Jackie Kay. Todd Hester, Timothy Mann, and Martin Riedmiller. Robust re- inforcement learning for continuous control with model misspecification. In International Confer- ence on Learning Representations, 2020. URL https: wwwoopeeeviiieeeee net. forum? d= HJg 9
Shie Mannor, Ofir Mebel, and Huan Xu. Lightningg does not strike twice: Robust mdps with coupled uncertainty. arXiv preprini arXiv: fttv:/1200 12oo:4646.4 2012. 9
Shie Mannor, Ofir Mebel, and Huan Xu. Robust mdps with k-rectangular uncertainty. Mathematics of Operations Research, 41(4):1484-1500 2016. 9
Stephen McAleer, John Lanier, Roy Fox, and Pierre Baldi. Pipeline PSRO: A scalable approach for finding approximate Nash equilibria in large games. In Advances in Neural Information Processing Systems, 2020. 9, 16
Stephen McAleer, John Lanier, Pierre Baldi, and Roy Fox. XDO: A double oracle algorithm for extensive-form games. Advances in Neural Information Processing Systems (NeurIPS), 2021. 9
Stephen McAleer, JB Lanier, Kevin Wang, Pierre Baldi, Roy Fox, and Tuomas Sandholm. Self-play psro: Toward optimal populations in two-player zero-sum games. arXiv preprint arX1v:2207.065.1, 2022a. 3
Stephen McAleer, Kevin Wang, Mare Lanctot, John Lanier, Pierre Baldi, and Roy Fox. Anytime optimal psro for two-player zero-sum games. arXiv preprinl arXiv:2201.07700 2022b. 3
Stephen McAleer, Gabriele Farina, Mare Lanctot, and Tuomas Sandholm. Escher: Eschewing impor- tance sampling in games by computing a history value function to estimate regret. International Conference on Learning Representations, 2023. 9, 16
H Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by an adversary. Proceedings of the 20th International Conference on Machine Learning (ICML), 2003. 3
Matej Moravcik, Martin Schmid, Karel Ha, Milan Hladik, and Stephen Gaukrodger. Refining subgames in large imperfect information games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016. 16
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisy, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in hcads-up no-limit poker. Science, 356(6337):508-515) 2017. 16
Paul Muller, Shayegan Omidshafiei, Mark Rowland, Karl Tuyls, Julien Perolat, Siqi Liu, Daniel Hennes, Luke Marris, Mare Lanctot. Edward Hughes, et al. A generalized training approach for multiagent learning. In International Conference on Learning Representations, 2019.
12

---

Arnab Nilim and Laurent EI Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Oper. Res., 53(5):780-798, 2005. 15
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and T'sii-Wei Weng. Robust deep reinforcement learning through adversarial loss. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL https: /tppereeeevveeeeeeeeeeeeeeeeeeee ew net/ / forum? i d=eadM_ bdwOQ. 9, 15
Xinlei Pan, Chaowei Xiao, Warren Hc, Shuang Yang, Jian Peng, Mingjie Sun, Mingyan Liu, Bo Li, and Dawn Song. Characterizing attacks deep reinforcement learning. In AAMAS, pp. 1010-1018. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022. 9
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In AAMAS, pp. 2040-2042. International Foundation for Autonomous Agents and Multiagent Systems Richland, SC, USA / ACM, 2018. 9,
15
Julien Perolat, Remi Munos, Jean-Baptisn Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, et al. From Poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization. In International Conference on Machine Learning, pp. 8525-8535. PMLR, 2021. 9, 16
Julien Perolat, Bart de Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome Connor, Neil Burch, Thomas Anthony, et Mastering the game of stratego with model-free multiagent reinforcement learning. arXiv preprint arXiv:2206. 15378, 2022. 9, 16
Lerrel Pinto, James Davidson, Rabul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference Machine Learning, pp. 2817-2826. PMLR, 2017. 9, 16
Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.0560 2019. 16
Martin Schmid, Matej Moravcik, Neil Burch, Rudolf Kadlec, Josh Davidson, Kevin Waugh, Nolan Bard, Finbarr Timbers, Mare Lanetot, Zach Holland, et al. Player of games. arXiv preprint
12.01721 2021. 16
Jack Serrino, Max Kleiman-Weines David C Parkes, and Josh Tenenbaum. Finding friend and foe in multi-agent games. Advances in Neural Information Processing Systems, 32, 2019. 16
Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference Machine Learning, pp. 8707-8715 18. PMLR, 2020. 8, 15
David Silver, Julian Schritwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of without human knowledge. nature, 550(7676):354-333 2017. 16
Samuel Sokota, Ryan D'Orazio, J Zico Kolter, Nicolas Loizou, Marc Lanctot, Ioannis Mitliagkas, Noam Brown, and Christian Kroer. A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games. arXiv preprint arXiv:2206.05825 2022. 9
Sriram Srinivasan, Marc Lanctot, Vinicius Zambaldi, Julien Pérolat, Karl Tuyls, Rémi Munos, and Michael Bowling. Actor-critic policy optimization in partially observable multiagent environments. Advances in neural information processing systems, 31, 2018. 16
Eric Steinberger. Single deep counterfactual regret minimization. arXiv preprint urXiv:1901.0766 2019. 16
Eric Steinberger, Adam Lerer, and Noam Brown. DREAM: Deep regret minimization with advantage baselines and model-free leaming. arXiv preprint arXiv:2006. 10410, 2020. 9, 16
13

---

Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. In Interaational Conference on Learning Representations, 2022. 1, 2, 6, 9, 15, 18
Yanchao Sun, Ruijie Zheng, Parisa Hassanzadeh, Yongyuan Liang, Soheil Feizi, Sumitra Ganesh, and Furong Huang. Certifiably robust policy learning against adversarial multi-agent communication. In The Eleventh International Conference on Learning Representations, 2023. URL https: // /pppeeeeeewwweo net
9, 15
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- tions in continuous control. In International Conference on Machine Learning, pp. 6215-6224.
PMLR, 2019. 1, 2, 6, 7, 9, 15, 16, 17
Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: htt.ttts://oppeeeereeeee //pp///eeeeewwoee//////ee net forum?
id=vIDBSG13v2L. 9
Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.01822 2020. 9, 15
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782)350-3344 2019. 16
Fan Wu, Linyi Li, Zijian Huang, Yevgeniy Vorobeychik, Ding Zhao, and Bo Li. CROP: certifying ro-
bust policies for reinforcement learning through functional smoothing. In International Conference on Learning Representations, 2022. 9, 15
Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896): 223 228, 2022. 16
Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, et al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310. 19668, 2023a. 15
Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil, Improved sample complexity bounds for distri- butionally robust reinforcement learning. In ATSTATS, volume 206 of Proceedings of Machine Learning Research, pp. 9728-9754. PMLR, 2023b. 15
Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. Douzero: Mastering doudizhu with self-play deep reinforcement learning. In International Conference on Machine Learning, pp. 2233-12334, PMLR, 2021. 16
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsich. Robust deep reinforcement learning against adversarial perturbations on state obser- vations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 21024-21037. Curran As- neurips http//pppe/22//////11/ sociates, Inc., 2020. URL httpstt///rocceediii Paper. paff 1, 2, 8, 15 60eb6568ea11 10w111060333333333/322232
Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning state observations with learned optimal adversary. In International Conference Learning 2,
Representations, 2021. URL https: h/tpp//////////ii..////// http://peeeeeeeeeeeeeoorurraaaaa oett://ooooommt1sssceeerrree 6, 9, 15
Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, R. Kumar, and Chao Tian. Natural actor-critic for robust reinforcement learning with function approximation, 2023. 16, 18
14

---

A PROOF OF PROPOSITION 3.3
Proof. The exploitability e(T) of strategy profile æ is defined as
e(TT)
Σ max
u T-1).
"
EN
A best response (BR) strategy (R, for player to strategy n- is a strategy that maximally exploits ---
BR, T
arg max (T T-1).
An C-Nash equilibrium (C-NE) is strategy profile which, for each player i, n, is an c-BR to π-
We can define the approximate exploitability of the pair of meta-Nash equilibrium strategies for the agent and the adversary (a,, σ.) as the sum of the expected reward their opponent approximate best responses achieve against them:
O() = va (T o.) + vo (T o.),
where u; (z -i) denotes the expected value of a player's 's approximate best response vs. the opponent meta-NE. Now assume that each approximate best response is within & of the optimal best response. Then
Uy ( a- 2 e BR; (o-i), 4'
As result, upon convergence, when the approximate exploitability ê(O., o..) is less than 5. then the exploitability of the pair of meta-Nash equilibrium strategies for the agent and the adversary (oay o.) is less than and the pair of strategies are in an c-approximate Nash equilibrium.
Every epoch where GRAD does not converge an approximate equilibrium, must add a unique deterministic policy to the population for either the agent or the adversary because if both players added policies already included in their populations, those policies would not be approximate best responses. Given that the MDP has a finite horizon and operates in a discrete action space, there exists only finite set of deterministic policies that can be added to the populations II, and II, Since the meta-Nash equilibrium over all possible deterministic policies is equivalent to the Nash cquilibrium of the original game, in the worst case where all possible deterministic policies are added, the algorithm will terminate an approximate Nash equilibrium.
B ADDITIONAL RELATED WORK
Robust RL against adversarial perturbations. Regularization-busee methods (Zhang et al., 2020; Shen et al., 2020; Oikarinen et al., 2021) enforce the policy to have similar outputs under similar inputs, which can achieve certifiable performance for visual-input RL (Xu et al., 2023a) on Atari games. However, in continuous control tasks, these methods may not reliably improve the worst-case performance. Recent work by Korkmaz (2021) points out that these adversarially trained models may still be sensible to new perturbations. Attack-driven methods train DRL agents with adversarial examples. Some early works (Kos & Song, 2017; Behzadan & Munir, 2017: Mandlekar al., 2017; Pattanaik et al., 2018; Franzmeyer et al., 2022; Vinitsky et al., 2020) apply weak or strong gradient-based attacks on state observations to train RL agents against adversarial perturbations. Zhang et al. (2021) and Sun et al. (2022) propose to alternately train an RL agent and strong RL adversary, namely ATLA, which significantly improves the policy robustness against rectangle state perturbations. A recent work by Liang et al. (2022) introduces more principled adversarial training framework that does not explicitly learn the adversary, and both the efficiency and robustness of RI. agents are boosted.
Additionally, a significant body of research has delved into providing theoretical guarantees for adversarial defenses in RL (Lütjens et al., 2020; Oikarinen et al., 2021; Fischer et al., 2019; Kumar et al., 2022; Wu et al., 2022; Sun et al., 2023), exploring various settings and scenarios. Robust RL faces challenges under model uncertainty et al., 2023b). The main goal of GRAD is to address the adversarial RL problem with an adversary that is adaptive the agent's policy. Works like Tessler al. (2019) on action perturbations and
prior works (lyengar, 2005; Nilim & Ghaoui, 2005; Xu
15

---

Zhou et al. (2023) model mismatch uncertainty, are hard to defend against the strongest adversarial perturbations and only empirically evaluated on uncertainty sets. This vulnerability arises due to the inherent difficulty in estimating the long-term worst-case value under adaptive adversaries. Distributional robust optimization (RahO)Ra & Mehrotra, 2019) is also challenging to apply to this challenging problem, especially against state adversaries in the high-dimensional state space. At the same time, adversarial training methods also struggle to effectively deal with model uncertainty or model mismatch problems. However, GRAD, leveraging game-theoretic methods, demonstrates robustness against adversarial perturbations and model uncertainty, as more effective and general solution for high-dimensional tasks.
Robust RL formulated as a zero-shot game. Considering robust RL formulated as zero-sum games, several notable contributions have emerged. Pinto et al. (2017) proposed robust adversarial reinforcement learning (RARL), introducing the concept of training an agent in the presence of a destabilizing adversary through a zero-sum minimax objective function. Huang et al. (2022) extended this paradigm with RRL-Stack, hierarchical formulation of robust RL using general-sum Stackelberg game model. Tessler et al. (2019) addressed action uncertainty by framing the action perturbation problem approaches by achieving approximate equilibriums on an adversary policy set. While existing works often focus on specific adversaries or worst-case scenarios, limiting their adaptability, GRAD does not specifically target certain adversaries, which makes it adaptable to various adversaries, including both cemporally-couples and con-tempooalll-couplle. adversarial settings. Moreover, GRAD's broad applicability extends to diverse attack domains, presenting more practical and scalable solution for robust RL compared to prior works.
zero-sum game. GRAD distinguishes itself from conventional robust RL
Game-Theoe Theoretic Reinforcement Learning, Superhuman performance in two-player games usually involves two components: the first focuses on finding a model-free blueprint strategy. which the setting we focus on in this paper. The second component improves this blueprint online via model- based subgame solving and search (Burch et al., 2014; Moraveik et al., 2016; 66; Brown et al., 2018; 8;; 2020; Brown & Sandholm, 201 7b; Schmid et al., 2021). This combination of blueprint strategies with subgame solving has led to state-of-the-art performance in Go (Silver et al., 2017), Poker (Brown & Sandholm, 2017a; 2018; Moravčík et al., 2017), Diplomacy (Gray et al., 2020), and The Resistance: Avalon (Serrino et al., 2019). Methods that only use a blueprint have achieved state-of-the-art performance on Starcraft (Vinyals al., 2019), Gran Turismo (Wurman et al., 2022), DouDizhu (Zha et al., 2021), Mahjohng (Li et al., 2020), and Stratego (McAleer et al., 2020; Perolat et al., 2022). In the rest of this section focus on other model-free methods for finding blueprints.
Deep CFR (Brown et al., 2019; Steinberger, 2019) is general method that trains a neural network on buffer of counterfactual values. However, Deep CFR uses external sampling, which may be impractical for games with a large branching factor, such as Stratego and Barrage Stratego. DREAM (Steinberger et al., 2020) and ARMAC (Gruslys al., 2020) are model-free regret-based deep learning approaches. ReCFR (Liu et al., 2022) proposes a bootstrap method for estimating cunulative regrets with neural networks. ESCHER (McAleer et al., 2023) removes the importance sampling term of Deep CFR and shows that doing so allows scaling to large games.
Neural Fictitious Self-Play (NFSP) (Heinrich Silver, 2016) approximates fictitious play by progres- sively training the best response against an average of all past opponent policies using reinforcement learning. The average policy converges to an approximate Nash equilibrium in two-player zero-sum games.
There is an emerging literature connecting reinforcement learning to game theory. QPG (Srinivasan et al., 2018) shows that state-conditioned Q-values are related to counterfactual values by a reach weighted term summed over all histories in an infostate and proposes an actor-critic algorithm that empirically converges to NE when the learning rate is annealed. NeuRD (Hennes et al., 2020), and F-FoReL. (Perolat et al., 2021) approximate replicator dynamics and follow the regularized leader, respectively, with policy gradients. Actor Critic Hedge (ACH) (Fu et al., 2022) is similar NeuRD methods do not have
but uses an information set based value function. All of these policy-gradice theory proving that they converge with high probability in extensive form games when sampling trajectories from the policy. In practice, they often perform worse than NFSP and DREAM on small games but remain promising approaches for scaling to large games (Perolat et al., 2022).
16

---

Algorithm 3 Action Adversary (AC-AD)
Input: Initialization of action adversary policy e; victim policy initial state so for = 0, 2,.... do
adversary e samples an action perturbations ūr ~ v(-(se),
victim policy и outputs action a₁ =T(((s))
the environment receives â de + ag returns 8+++1 and rr
adversary saves (st, āe -T,, 8141) to the adversary buffer
adversary updates its policy
end for
C EXPERIMENT DETAILS AND ADDITIONAL RESULTS
C.I IMPLEMENTATION DETAILS
We provide detailed implementation information for our proposed method (GRAD) and baselines.
Training Steps For GRAD, specify the number of training steps required for different environ- ments. In the Hopper, Walker2d, and Halfcheetah environments, we train for 10 million steps. In the Ant and Humanoid environments, extend the training duration 20 million steps. For the ATLA baselines, we train for million steps and 10 million steps environments of varying difficulty.
Network Structure Our algorithm (GRAD) adopts the PPO network structure the ATLA baselines to maintain consistency. The network comprises a single-layer LSTM with 64 hidden neurons. Additionally, and an output layer is used to project 64 to the output dimension. Both the agents and the adversaries use the same policy and value networks to facilitate training and evaluation. Furthermore, the network architecture for the best response and meta Nash remains consistent with the aforementioned configuration.
input embedding layer is employed to project the state dimension 64,
Schedule of e and غ During the training process, we gradually increase the values of e and č from to their respective target maximum values. This incremental adjustment occurs over the first half of the training steps. We reference the attack budget used in other baselines for the corresponding environments. This ensures consistency and allows for a fair comparison with existing methods. The target value of is determined based the adversary's 's training results, which is set c/5. In some smaller dimensional environments, é can be set to c/10. We have observed that the final performance of the trained robust models does not differ by more than 5% when using these values for č.
Observation and Reward Normalization To ensure consistency with PPO implementation and maintain comparability across different codebases, apply observation and reward normalization. Normalization helps to standardize the input observations and rewards, enhancing the stability and convergence of the training process. We have verified the performance of vanilla PPO different implementations, and the results align closely with our implementation of GRAD based on Ray rllib.
Hyperparameter Selection Hyperparameters such as learning rate, entropy bonus coefficient, and other PPO-specific parameters are crucial for achieving optimal performance. Referring to the results obtained from vanilla PPO and the ATLA baselines as references, a small-scale grid search is conducted to fine-tune the hyperparameters specific to GRAD. Because of the significant training time and cost associated with GRAD, we initially perform simplified parameter selection using the Inverted Pendulum as a test environment.
C.2 ADVERSARIES IN EXPERIMENTS
State Adversaries Aimed to introduce the attack methods utilized during training and testing in experiments. When it comes to state adversarics, PA-AD as Alogrithm 4 stands out as the strongest attack compared other state attacks. Therefore, report the best state attack rewards under PA-AD attacks.
Action Adversaries In terms of action adversaries, RL-based action adversary Alogrithm can inflict more severe damage on agents' rewards compared to OU noise and parameter noise in (Tessler et al., 2019).
17

---

Mixed Adversaries When dealing with mixed adversaries capable of perturbing both state and action spaces, becomes crucial to design the action space for the adversary. Algorithm 5, we extend the idea of PA-AD (Sun et al., 2022), which learns policy perturbation direction generate perturbations. In our case, the mixed adversary director only needs to learn the policy perturbation direction dz. For various attack domains, the actor functions then translate the direction de into state or action perturbations. This design approach ensures that our mixed adversary doesn't increase the complexity of adversary training. as it deploys mixed perturbations using different actor functions as required by distinct attack domains.
Algorithm 4 Policy Adversarial Actor Director (PA-AD)
Input: Initialization of adversary director's policy e; victim policy æ, the actor function g for the state space 8, initial state so
for == 0, 1, .... do
Director samples a policy perturbing direction and perturbed choice, ā v1-(ss)
Actor perturbs to š = St),,
Victim takes action a, T-((s)). proceeds $1+1, receives r,
Director saves (8t, (s,, -Yt, S1+1) to the adversary buffer
Director updates its policy u using any RI. algorithms
end for
Algorithm 5 Mixed Adversary
Input: Initialization of adversary director's policy u; victim policy æ, the actor function gs for the state space and go. for the action space A, initial state so
for f 2, do
Director samples a policy perturbing direction de u(-1s)).
Actor perturbs 8, to 5, = g, (dr. 8:)
Victim takes action a (-115)). proceeds to 8141, receives victim policy outputs action 0₁ ~ xT-|st)
Actor perturbs to ã; 9a (d),
The environment receives â,, returns 81+1 and r.
Director saves (st, ag -T,, 1c+1) to the adversary buffer
Director updates its policy u using any RL algorithms
end for
Transition Adversaries. In addition to addressing adversarial perturbations, we extend the eval- uation of GRAD to consider transition uncertainty, mitigating the mismatch problem between the training simulator and the testing environment. Robustness under transition uncertainty crucial for real-world applicability. To assess this aspect, experiments JoCo environments (Hopper. Walker2d, and HalfCheetah) by modifying their physical parameters ("leg joint stiffness' value: 30, 'foot joint stiffness" value: 30, and bound 'back actuator range'. 0.5) following the protocol established by Zhou et al. (2023). Comparative evaluations are performed against robust natural actor-critic (RNAC)(Zhou et al., 2023) trained with Double Sampling (DS) and Inaccurate Parameter Models (IPM) uncertainty. The results presented in Table consistently demon- strate that GRAD achieves competitive or superior performance compared to baseline methods in each perturbed environment, showcasing its effectiveness in robustly handling transition uncertainty.
conducted on perturbed Mu-
Attacks. While our temporally-conn coupled setting consid- cring perturbation from the last time step aligns with the common practice of state adversaries, which typically perturb the current state without explicitly attacking short-term memory, we recognized the importance of exploring a more general scenario akin to general partially observable MDP (Efroni et al., 2022). We introduced a short-term memorized temporally-coupp the mean of perturbations from the past 10 steps and applying the temporally-coupler constraint to this mean. The results in Table 2 from these additional experiments against short-term memorized cemporally-couplee attacks underscore the efficacy of GRAD under this extended setting. GRAD consistently demonstrates heightened robustness compared to other robust baselines when confronted with memorized temporally-couu adversary. These findings provide valuable insights into the
Short-term Memorized Temporall-coupe
attacker by calculating
18

---

Perturbed Environments
RNAC-PPO (DS)
RNAC-PPO (IPM)
GRAD
Natural reward
3502 1 256
3254 138
3482 1 209
Hopper
leg.
2359 + 182
2289 + 121
2692 + 236
Natural reward
4322 + 289
4248 = 89
4359 - 141
Walker
foot_joint..c "oofioosntstiffhesss
4078 = 297
4129 = 78
4204 = 132
Natural reward
5524 1 178
5569 + 232
6047 1 241
Halfcheetah
'back actuator range'
768 + 102
1143 + 45
1369 + 117
Table 1: Comparison of cumulative reward in Perturbed Environments with changed physical parameters.
Short-term Memorized Temporally-Couee Cocplle Attacks
Hopper
Walker2d
Halfcheetah
Ant
Humanoid
PA-ATLA-PPO
2334 +: 249
2137 +: 258
3669 + 312
2689 + 189
1573 = 232
WocaR-PPO
2256 332
2619 + 198
4228 283
3229 = 178
2017 = 213
GRAD
2869 228
3134 + 251
4439 + 287
3617 : 188
2736 269
Table 2: Performance Comparison under Memorized Hamporally-Couplee Attacks
-E- AU-ATLA-IMI Wic.R PRP 2RAD (Uans)
SA SA-ATLA-PTOO
4,000
Wecak-rr1
Wi PPO
(Dans)
4/301
Awerge aptewde mooneee
GRAD (Chnn)
3,001
00000
3,000
2,000
9,000
1,00
3,000
Out 0.07S
0.15
0.16
0.1
0.55
0.05
0.15
0.0
1..5
Texeporall
Texpprally
Tanperally
(a) State attacks: Hopper
(b) Action attacks: Hopper
(c) Mixed attacks: Hopper
7/000
4,100
6,001
SA SA-AIT A A-Pr- Weesk-I9
Wecsi-0191
0,001
Weest-PMi
GRAD (Ouns)
GRAD (Durs)
GRAD (Oues)
5/001
4,000
2,000
3,000
2,00
3000
11.00
0.10
0,405
0.25
10.0
Texaporally orapiei b
Texpperally coaplel badadd
Terpperally couppoll stack badack
(d)
State attacks: Humanoid
(c) Action attacks: Humanoid
(f) Mixed attacks: Humanoid
Figure 9: Comparisons under state or action or mixed temporally-coupled attacks w.r.t. diverse attack budgets e's on Hopper and Humanoid.
temporal scope of perturbations, contributing to more comprehensive understanding of GRAD's capabilities in handling diverse adversarial scenarios.
C.3 ATTACK BUDGETS
In Figure 9, we report the performance of baselines and GRAD under different attack budget e. As the value of increases, the rewards of robust agents under different types of attacks decrease accordingly. However, our approach consistently demonstrates superior robustness as the attack budget changes.
C.4 CAMPORALLY-COUPLAEE CONSTRAINTS
We also investigate the impact of semporally-coupler constraints غ on attack performance, as we explained in our experiment section.
19

---

Avrage mwann www.03000000
3,83)
GRAD
3,000
GRAD
5,000
2,000
3,000
3,000
7,000
2,000
D.22
0.16
0.66
C 0.03
0.1
0.00
6.33
0.06
State attacks: Ant
(b) Action attacks: Ant
(c) Mixed ottacks: Ant
6,000
0,000
-Bi- PI-XTLA-PPO
Wheat-FFO
6,000
-i- AC AC-XTLA-PPOO
Wheak-I1Y)
5,000
SA-ATLA RPO*
Weeak-Iti
GRAD
5,000
GRAD
4,000
4,000 -
3,000
3,000
0.:00
0:25
6.00
0.6
6.13
Tenspasalt
(d) State attacks: Humanoid
(e) Action attacks: Humanoid
(f) Mixed attacks: Humanoid
Figure 10: Comparisons under state or action or mixed temporally-coupled attacks with diverse temporally-
coupled constraints c's on Ant and Humanoid.
+
Plaate
Plane
MMA Durd
beng dird
UMV
(a) Hopper
(b) Walker2d
(c) Halfcheetah
(d) Ant
(e) Humanoid
Figure 11: Natural performance Robustness against temporally-coupled action perturbations
C.5 NATURAL REWARD VS. ROBUSTNESS
We presents the natural performance comparison of GRAD and action robust baselines in Figure 11.
C.6 COMPUTATIONAL COST
The training time for GRAD can vary depending on the specific environment and its associated difficulty. Typically, single V100 GPU, training GRAD takes around 20 hours for environments like Hopper. Walker2d, and Halfcheetah. However, for more complex environments like Ant and Humanoid, the training duration extends approximately 40 hours. It's worth noting that the training time required for defense against state adversaries or action adversaries is relatively similar.
20


Reviews:
[{"review_id": "5rDurpcmTn", "reviewer": "Reviewer_mjop", "summary": "The classic robust RL focuses on worst-case scenarios, which may result in an overly conservative policy. Instead, this paper introduces temporally-coupled perturbations. Additionally, this paper proposed an adversarial training approach named the game-theoretic response approach for adversarial defense. Finally, the authors show the robust performance of the proposed methods in several MuJoCo tasks compared with several baselines.", "questions": "Please refer to the \"weakness\" section for further information.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* This paper is easy to follow.\n* This paper does thorough experiments for state attacks and action attacks and compares the proposed method with several baselines. \n* The temporally-coupled adversarial perturbation seems new.", "weaknesses": "* Although the temporally-coupled adversarial perturbation seems new, it is quite limited. Definition 3.2 only considers the temporally-coupled perturbation from the last time step. Even if the authors don't consider the general partially observable MDP, they should consider a more general case, e.g., m-order MDP [Efroni et al. 2022, Provable Reinforcement Learning with a short-term memory].\n* The zero-sum game-based approach is not new for robust training in RL, e.g., [Tessler et al., 2019].\n* This paper misses one classic setting of robust MDP (i.e., transition adversaries) [e.g., Iyengar'05, Robust Dynamic Programming] as well as related baselines [e.g., Zhou et al. 2023, Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation].\n* Figure 1 doesn't make sense to me. The robust baseline considers the worst-case scenario, which should be more stable for different kinds of attacks compared with the less conservative model that is proposed in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jGRwI2L0qv", "reviewer": "Reviewer_JmX3", "summary": "This work introduces a novel-framework of temporally-coupled robust RL problem that is closer to the real-world setting. This work proposes GRAD, a game-theoretic approach to provide robust policies played against an adversary which attacks states and actions fitting in the temporally-coupled robust RL problem setting. This work also gives extensive complementing experiment results.", "questions": "-n/a-", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I think this work really pushes the robust RL community research efforts further by answering:\n\n> can we design robust RL algorithms for realistic nature attacks?\n\nThe main contribution of game-theoretic algorithm with temporal-based nature attacks (robust RL problem) is a really nice idea worthy for publication. But the score reflects my weakness section.", "weaknesses": "I have only a few weakness for this work as follows:\n\n> The current framework considers robustness against state and action uncertainty. More closer work [1] and thereafter are not included. Model uncertainty is justified in the framework mentioning the evolution of the environment depends on the perturbed actions. Model uncertainty in robust RL is defined in more generality [2-10]. So it will be better to include more detailed Related Works including [2-10] and more relevant works in the revision. I agree this work includes experiments with model uncertainty, but the baselines are also only action robust algorithms. I'd rather see more extensive writing and experiments for model-uncertainty OR the current work just focusing on state-action uncertainty is a big step forwards in itself. I've also stopped at '10' since you get the idea of inadequate related work discussion.\n\n> GRAD shares similar idea of RARL algorithm (Pinto et al., 2017), that is, zero-sum structure to get the robust policy against the nature adversary. More details than below must be added to point out key differences (like state-action uncertainty inbuilt) and due references need to be given.\n` Pinto et al. (Pinto et al., 2017) model the competition between the agent and the attacker as a zero-sum two-player game, and train the agent under a learned attacker to tolerate both environment shifts and adversarial disturbances `\n\nI am open to discussions with the authors and reviewers to increase/maintain (already reflects the positive impact) my score. All the best for future decisions!\n\n[1] Robust Multi-Agent Reinforcement Learning with State Uncertainty Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Transactions on Machine Learning Research, June 2023.\n\n[2] Xu. Z, Panaganti. K, Kalathil. D, Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learning. Artificial Intelligence and Statistics, 2023.\n\n[3] Nilim, A. and El Ghaoui, L. (2005). Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798\n\n[4] Iyengar, G. N. (2005). Robust dynamic programming. Mathematics of Operations Research, 30(2):257–280.\n\n[5] Panaganti, K. and Kalathil, D. (2021). Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In Proceedings of the 38th International Conference on Machine Learning, pages 511–520.\n\n[6] Panaganti, K. and Kalathil, D. (2022). Sample complexity of robust reinforcement learning with a generative model. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pages 9582–9602.\n\n[7] Roy, A., Xu, H., and Pokutta, S. (2017). Reinforcement learning under model mismatch. In Advances in Neural Information Processing Systems, pages 3043–3052.\n\n[8] Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M. (2022). Robust reinforcement learning using offline data. Advances in Neural Information Processing Systems (NeurIPS).\n\n[9] Shi, L. and Chi, Y. (2022). Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05767\n\n[10] L Shi, G Li, Y Wei, Y Chen, M Geist, Y Chi  (2023) The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model, NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K0CSRP4WA9", "reviewer": "Reviewer_GZCq", "summary": "This paper addresses the challenge of deploying reinforcement learning (RL) systems that can withstand uncertainties, particularly those that are temporally coupled. Recognizing that conventional robust RL methods may falter against such temporally-linked perturbations, the authors introduce a game-theoretic approach called GRAD (Game-theoretic Response approach for Adversarial Defense). GRAD conceptualizes the robust RL problem as a partially observable two-player zero-sum game and uses Policy Space Response Oracles (PSRO) to achieve adaptive robustness against evolving adversarial strategies. The study's experiments confirm GRAD's performance in ensuring RL robustness against both temporally coupled and standard adversarial perturbations.", "questions": "Please see the Weaknesses, I will decide the final rating after the rebuttal.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors formulate the robust RL objective as a zero-sum games and demonstrating the efficacy of game-theoretic RL in tackling this objective.", "weaknesses": "1. This paper does not have a clear mathematical representation of the problem it intends to address. \n\n2. The article claims its primary contribution lies in using zero-sum games to formulate the robust RL problem. However, employing zero-sum games to account for uncertainties, whether in single-agent or multi-agent RL, is well-established, as seen in works like Robust Adversarial Reinforcement Learning, Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training, and Robust Multi-Agent Reinforcement Learning with State Uncertainty. Given this widespread application, the paper's stated novelty becomes questionable.\n\n3. In terms of algorithmic design, the proposed method is largely an application of the Policy-Space Response Oracles (PSRO). The novelty seems limited, and it's unclear how PSRO uniquely addresses the issue of temporally-coupled perturbations.\n\n4. Considering that the PSRO algorithm converges to an NE in two-player, zero-sum games and has seen recent extensions to other equilibria types [1, 2], the paper's proposed method, essentially a reiteration of PSRO, makes the convergence proof for GRAD appear somewhat lackluster in its contribution.\n\n5. This paper has ample room for improvement in writing, problem formulation, and the method itself. For instance, by simply using the triangle inequality, we could give the range of $\\bar{\\epsilon}$ that break the temporally-coupled property, rather than merely stating, \"By setting \\epsilon ̄ to a large value, it converges to the non-coupled attack scenario.\" Additionally, the motivation behind temporally-coupled perturbations lacks clarity and persuasiveness, leaving me unconvinced of its pressing relevance.\n\n[1] Lanctot, Marc, et al. \"A unified game-theoretic approach to multiagent reinforcement learning.\" Advances in neural information processing systems 30 (2017).\n\n[2] McAleer, Stephen, et al. \"Pipeline psro: A scalable approach for finding approximate nash equilibria in large games.\" Advances in neural information processing systems 33 (2020): 20238-20248.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)