Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning
Authors: Shangding Gu
Laixi Shi
Muning Wen
Ming Jin
Eric Mazumdar
Yuejie Chi
Adam Wierman
Costas Spanos
Venue: ICLR 2025 Poster (2025)
Keywords: Robust reinforcement learning
benchmark
reinforcement learning
multi-agent reinforcement learning
Primary area: datasets and benchmarks

Abstract: Driven by inherent uncertainty and the sim-to-real gap, robust reinforcement learning (RL) seeks to improve resilience against the complexity and variability in agent-environment sequential interactions. Despite the existence of a large number of RL benchmarks, there is a lack of standardized benchmarks for robust RL. Current robust RL policies often focus on a specific type of uncertainty and are evaluated in distinct, one-off environments. In this work, we introduce Robust-Gymnasium, a unified modular benchmark designed for robust RL that supports a wide variety of disruptions across all key RL components—agents' observed state and reward, agents' actions, and the environment. Offering over sixty diverse task environments spanning control and robotics, safe RL, and multi-agent RL, it provides an open-source and user-friendly tool for the community to assess current methods and foster the development of robust RL algorithms. 
In addition, we benchmark existing standard and robust RL algorithms within this framework, uncovering significant deficiencies in each and offering new insights.

Published as a conference paper at ICLR 2025
ROBUST GYMNASIUM: A UNIFIED MODULAR BENCH- MARK FOR ROBUST REINFORCEMENT LEARNING
Shangding Gu' Laixi Shi Muning Wen Ming Jin', Eric Mazumdar
Yuejie Chi,, Adam Wierman', Costas Spanos'
University of California, Berkeley
California Institute of Technology
Shanghai Jiao Tong University Virginia Tech 5 Carnegie Mellon University
ABSTRACT
Driven by inherent uncertainty and the sim-to-real gap, robust reinforcement learning (RL) seeks to improve resilience against the complexity and variabil- ity in eeveroenviro sequential interactions. Despite the existence of large number of RL benchmarks, there a lack standardized benchmarks for ro- bust RL. Current robust RL policies often focus specific type of uncertainty and are evaluated in distinct, one-off environments. In this work, we introduce Bobust-Gyymasiiud
a unified modular benchmark designed for robust RL that ob-
supports wide variety of disruptions across all key RL camponents--gene served state and reward, agents' actions, and the environment. Offering over sixty diverse task environments spanning control and robotics, safe RL, and multi-agent RL, provides an open-source and user- seer-friendly tool for the community to assess current methods and foster the development of robust RL algorithms. In addition, we benchmark existing standard and robust RL algorithms within this framework, uncovering significant deficiencies in each and offering insights. The code is available at this website
INTRODUCTION
Reinforcement learning (RL) is popular learning framework for sequential decision-making based on trial-and-error interactions with an unknown environment, achieving success in a variety of appli- cations, such games (Mnih al. 2015: Vinyals et al.) (2019), energy systems (Chen et al. [0022),
finance and trading (Park& Van Roy 2015 Davenport & Romberg 2016). and large language model alignment OPenAA 2023 Ziegler al. 2019).
Despite recent advances in standard RL, its practical application remains limited due to concerns over robustness and safety. Specifically, policies learned in idealized training environments often fail catastrophically in real-world scenarios due to various factors such as the sim-to-real gap (Pinto) al. 2017). uncertainty (Bertsimas al. 2019), noise, and even malicious attacks (Zhang al., 2020 Klopp et al. 2017; Mahmood et al. 2018). Robustness is key to deploying RL in real-worl applications. especially in high-stakes or high-cost fields such as autonomous driving (Ding al.) 2023b), clinical trials (Liu et al. 2015), robotics (Li et al. 2021). and semiconductor manufacturing
(Kozak et al. 2023). Towards this, Robust RL seeks ensure resilience the face of the complexity and variability of both the physical world (Bertsimas et al. 2019) and human behavior (verskyy &
Kahneman 1974 Arthur [991).
Robust RL policies currently fall short of the requirement for broad deployment. Disruptions or interventions can occur various stages of the agent-environment interaction, affecting the agent's observed state (Zhang al. 20200 2021b. Han al. 2022, Sun et al. 2021 Xiong et al.) 2022 observed reward (Xu & Mannor (006). action (Huang et al. (017)) and the environment (transi-
tion kernel) (Iyengar] 2005 Pinto al. 2017) and existing robust RL policies are vulnerable to
Equual Contribution.
WWe appreciate any constructive comments and suggestions corresponding to shangding gu berkeley.edu and laixis caltech.edu.

---

Published as a conference paper at ICLR 2025
Robust RL Paradigms in Robust Gymnasium
Standard Single-Acent RL
Standard Multt-Agent RL
Gymnnum-ffostt
Oisr
Standard
Fand
Obnerventon Diarugnior
Disruption Module
Safety MuitCC
Task Bases
Figure 1: The overview of Robust-Gymmasium. For more details, please visit the website
such real-world failures (Mandlekar al. 2017). This vulnerability is, part, result the fact that policies are designed to address only one specific type of disruption (e.g., over the observed state), among other technical limitations (Ding et al.) 2023a). More critically, robust RL policies are often evaluated in distinct, one-off environments that can be narrow over-fitted to the proposed algorithms. The absence of standardized benchmarks Ideally, benchmark should offer wide range of diverse tasks for comprehensive evaluation and account for uncertainty and disruptions over multiple stages throughout the interaction process.
key bottleneck to progress robust RL.
While numerous RL benchmarks exist, including recent one focused robustness environment shifts (Zouitine et al. 2024), none are specifically designed for comprchensively evaluating robust RL algorithms. To address this gap, present Robust-Gymmnsiumm] lar benchmark for robust RL. This open-source tool enables flexible construction of diverse tasks, facilitating the evaluation and development robust RL algorithms. Our main contributions are:
unified, highly modu-
We introduce a unified framework for robust RI,, encompassing diverse disruption types within modular agent-environment interaction process (detailed in Sec. (2). This framework enables the development of Bobbus-Gyynasium a benchmark that comprises over sixty diverse tasks in roboties and control, safe RL, and multi-agent RL; and includes a wide range of disruptions targeting different stages/sources (agent observations, actions, and the environment) with varying modes (e.g.. random adversarial disturbances, en- vironmental shifts) and frequencies. This is a unified benchmark specifically designed for robust RL, providing foundational tool for evaluating and developing robust algorithms.
We conduct comprehensive evaluation of several state-of-the-aa (SOTA) baselines from standard RL, robust RL, safe RL, and multi-agent RL using representative tasks in nobust-Gymnasiun
Our findings reveal that current algorithms often fall short expectations in challenging tasks, even under single-stage disruptions, highlighting the need for new robust RL approaches. Furthermore, by encompassing tasks with disruptions across all
experiments demonstrate the
flexibility Robust-Gymnasinn stages and four disturbance modes, including an adversarial model using large language model (LLM). This illustrates the potential of LLMs in robust RL research.
2 A UNIFIED ROBUST REINFORCEMENT LEARNING FRAMEWORK
We begin by presenting robust RL framework that unifies various robust RL tasks explored in the literature, including combinations of these paradigms. We outline the framework in the context of single-agent RL and then extend to encompass broader classes of RL tasks, such as safe RL and multi-agent RL.
Website with the introduction, code, and examples: https:://oobuss ceccbbuusttyy....mm gitm.g io/
2

---

Published as a conference paper at ICLR 2025
Background: Markov decision process (MDP). A single-agent RL problem is formulated finite-horizon Markov decision process (MDP), represented by the tuple M A,T, Po; rr), where S and A denote the (possibly infinite) state and action spaces, and T is the horizon length. The nominal transition kernel pa = { (P igieT defines the environmental dynamics: ( 8, gives the probability of transitioning from state to state s' given action time step t. The reward function rd = frissT represents the immediate reward at time step t, given the current state and action a.
action de ~ nt(()8))
action āg
Action- Disruptor
(PI) rr))
(Pe.
Environment- Disruptor
Perturbed Environment
Agent (Poliyy)
Environment
reward Fr Fa(8t,
next state P2.
Observed reward
Observation- Disruptor
Observed state S2
Figure 2: The overview of a finite-horizon MDP with disruptors.
2.1 UNIFIED ROBUST RL FRAMEWORK: MDPS WITH DISRUPTION
To proceed, introduce additional disruption module that represents potential uncertainties disturbances that impact different stages of the agent-environmen interaction process (MDP). This module provides a categorized summary of the types of uncertainty addressed prior robust RL studies.
Disruptors. We introduce each type in detail follows:
Observation-divuupto An agent's observations may not perfectly reflect the true status of the environment due factors like sensor noise and time delays. To model this sens- ing inaccuracy, we introduce an additional modulc-the termines the agent's observations from the environment: Agents' observed state ăt The observation-dee observation-disrupoor takes the true current state as input and outputs perturbed state 5; Ds(s)). The agent uses 8, as input its policy select an action; Agents' ob- served reward 7r: The observation-disrupcc takes the real immediate reward and outputs perturbed reward F, = (r.).. The agent observes 7, and updates its policy accordingly.
dicervation-disuuutor--www
de-
input
Action-disrupte The real action d, chosen by the agent may be altered before or during execution in the environment due implementation inaccuracies system malfunctions.. The action-disruptor models this perturbation, outputting a perturbed action ā = (a,a.). which is then executed the environment for the next step.
Recall that a task environment consists of both the internal dynamic model and the external workspace it interacts with, characterized by its transition dynamics P and reward function The environment during training can differ from the real-world environment due factors such the sim-to-real gap. human and natural variability, exter- nal disturbances, and more. We attribute this potential nonstationarity to environment- disruptor, which determines the actual environment (P, the agent interacting with at any given moment. These dynamics may differ from the nominal environment (PO r that the agent was originally expected to interact with.
Enviromment-disrupe
MDPs with Disruption. As shown in Fig. robust RL problem can be formulated as finite- horizon MDP with an additional disruption module Mais = AS, P.r. D,(.), D,(.), D.(()....)) D.).). abbreviated DDP. It consists of three potential disruptors introduced above. Specif- ically, the interaction process between an agent and an MDP with disruption (Fig. 2 unfolds follows: at each time step & [T), the (possibly perturbed) environment outputs the current
3

---

Published as a conference paper at ICLR 2025
state 9t and reward r2. The observation-disrupii then perturbs these, sending the modified state S, = =D(ss) and reward Ft = (r1):) ) to the agent. Based on these, the agent selects an action a, .T(- 5t), according to its policy
S + A(A) defines the probability distribution over actions in A given the observed state &t- The action-disruptor then a perturbed environment governed by the citrromment-disruppee based on the reference nominal environment (Po) ,r)) The environment then transitions the next state St+1 Pt(- St, āt) and provides the reward r (St, oz), which becomes the input for the observation-disrupto in the next step t +
{ where
perturbs this action de (ax))), which then
Goal. For any Disrupted-MDD the objective learn a policy (action selection rule) (TisSIsT that maximizes long-term cumulative rewards, represented by the value function (V." JiseT :
max V (s) = E
Σ (sk: ax) n, (Prr),
(1)
Here, the expectation is taken the trajectories generated by executing the policy x under the perturbed transition kernels and reward functions rP.
In addition to disruption modes, the Disrupted-MDP allows disruptors to operate flexibly over time during the interaction process. Disruptors can act at different frequencies, such as step-wise, wiss varying intervals.
Robust-Gymnasium: A UNIFIED ROBUST RL BENCHMARK
We now introduce our main contribution, a modular benchmark Brobust-Gymnasiun designed for evaluating Robust RL policies in robotics and control tasks. Each task is constructed from three main components: an agent model (the robot object), an environment (the agent's 's workspace), and task objective (such as navigation manipulation). Robust-Gymnaas integrating various disruptors of different types, modes, and frequencies with these task bases. Not all task bases support every type of disruption. A detailed list of the robust RL tasks implemented in this benchmark is available in Figure [7 In the following sections, we introduce over 60 task bases from eleven sets, outline the design the disruptors, and describe the construction of a Disrupted- MDP_ robust RI. tasks.
offers robust RL tasks by
3.1
TASK AND ENVIRONMENT BASES
Gymmasium-Box211 (three relative simple control tasks in games).
These tasks are from Gymnasium (Towers et al., 2024). including three robot models from different games, such as the Bipedal Walker 4-joint walking robot designed to move forward and Car Racing navigating track by learning from pixel inputs (Parberry 2017 Brockman al. 2016).
Symmisium-MuJJoC) (eleven control tasks).
It includes various robot models, such bipedal and quadrupedal robots. This benchmark is widely used various RL problems, including standard online and offline RL, with representative examples like Hopper, Ant, and HalfCheetah Todorov et al. 2012 Brockman et al. 2016).
Maze (two navigation environments).
Maze comprises environments where an agent must reach specified goal within maze (Gupta et al., 2020p. Two types of agents are available: 2-degrees of freedom (DoF) ball (Point-Maze) and more complex 8-DoF quadruped robot (Ant-Maze) from Gymnasium- Symmasium--MuloCo MuloCo. Various goals and maze configurations can be generated to create tasks of varying difficulty.
Fetch (four tasks for Fetch Mobile Manipulator robot arm). Fetch features 7-degrees of freedom (DoF) Fetch Mobile Manipulator arm with a two-fingered parallel gripper (Plappert et al.) 2018). The environment consists of table with various objectives, resulting in four tasks: Reach, Push, Slide,
4

---

Published as a conference paper at ICLR 2025
and PickAndPlace, which involve picking up moving the objects to specified locations.
Franka Kitchen (tasks need long-holl hong-horizon, multi-task planning for a robot arm). This environment based on a 9-degrees of freedom (DoF) Franka robot situ- ated in kitchen containing common household items like microwave and cab- inets (uupta et al. 2020p. The task goal is to achieve specified configuration, which may involve planning and completing multiple sub-tasks. For example, goal state could have the microwave open, a kettle inside, and the light over the burners turned on.
Dexterous Hand (five dexterous hand manipulation tasks).
is based on the Shadow Dexterous Hand - an anthropomorphic 24-DoF robotic hand with 92 touch sensors palm and phalanges of the fingers (Plap- pert et al. 2018 Melnik et al., 2021). The tasks involve manipulating various objects, such pen, egg. blocks.
Adroit (four manipulation tasks for a dexterous hand attached to a free arm). This environment features freee arm equipped with a Shadow Dexterous Hand, providing to 30-DoF (Rajeswaran et al. 2018). The high degree of freedom enables the robot perform more complex tasks, such as opening door with a latch (AdroitHandDoor),
HumanoidBench (four tasks for nighh-dimensiona humanoid). We incorporate four tasks from the recent HumanoidBench (Sterrazza et al. 2024) designed mainly for a Unitree HI humanoid robot P which is equipped with two dexterous Shadow Hands. Specifically, tasks (push, truck) and two locomotion tasks (reach, slide), all of which require sophisticated coordination among various body parts.
include two manipulation
Robosuite (twelve tasks for various modular robot platforms).
Robosuite is popular modular benchmark (Zhu al.) 2020) that supports seven robot arms, eight grippers, and six controller modes. The manipulation tasks are conducted environments with doors, tables, and multiple robot arms, with goals such as wiping tables or coordinating to transfer hammer. Additionally, we introduce new task Mok--MutiRoouutDDor feattuu an adversarial arm that impedes another arm's 's success to test robustness.
Safety MuJoCo (nine control tasks with additional safety constraints). Built standard robot models in Gymnasium- MuJooCo, the Safety MuJoCo tasks are designed for scenarios that prioritize both long-term returns and safety. These tasks incorporate safety constraints, such limiting velocity and prevent- ing robots from falling (Gu et al.) 20245).
MAMuJoCo (twelve multi-agent cooperation tasks).
MAMuJoCo is based on multi- platform from the factorizations of Momnisiu--MJJoGe robot models (Peng al., (021). The tasks need to be solved by cooperations of multiple agents. This set of tasks are vulnerable to disturbance like one leg of a quadruped robot malfunctioning, or all dynamics of legs are contaminated by system noise.
Lagerd
3.2 DISRUPTOR DESIGN: MODES AND FREQUENCIES
In a Disrupted-MDP disruptors affecting various stages of the enviroment interaction can operate in different modes. We typically consider four common modes found in the robust RL literature, each driven by specific real-world scenarios and robustness requirements. These modes allow the construction of tasks with varying levels of difficulty:
Random disturbance: for all disruptors. Stochastic noise is ubiquitous in sensors, me- chanical hardware, and random events, often modeled as random noise added to nominal
hhtps//wwww.miireeccmm///
5

---

adjusts parameters within Published as a conference paper at ICLR 2025
(a) Original
(b) Change
[c) Change leg
[d) Original task
(e) Change distance
(f)
Figure 3: Illustration of two disruption modes of the environment-disruptor:: internal dynamic shift and external disturbance.
components in the interaction process (Duan et al., 2016). The noise typically follows a dis- tribution such as Gaussian or uniform. This mode can be applied to all disruptors, affecting the agent's observed state, observed reward, action, and environment.
We offer Gaussian distribution -..))) (Zhang et al. 2018 and bounded uniform distri- bution u(:--) Zouitine et al.) 2024) default options. For instance, the environment- disruptor can introduce noise to robot dynamics (e.g., mass, torso length) or external fac- tors (e.g., gravity, wind), as shown in Fig. 4 The observation-disrupoo can add noise to (He,
the observed state and/or reward, namely, St 9t + (W σa) (Ju, and a, are the mean
and variance) 5; = 5; Ua,,, ba) (aa, are the min and max thresholds). The action- disruptor can also introduce noise to the action sent to the environment.
Adversarial disturbance: for all disruptors. In real-world applications, adversarial distur- bances occur when external forces deliberately attempt to degrade the agent's 's performance. This mode also relevant when prioritizing safety, ensuring the agent can perform well in worst-case scenarios within certain predefined sets. It can be applied to all three disruptors. This mode can be viewed as a two-player zero sum game between the agent and an ad- versarial player (Tanabe et al. 2022). Any algorithms can acts the adversarial player through this interface to adversarialyy attack the process. This mode is applicable to all dis- ruptors; for instance, the observation -iisruttt generates fake state that falls within the prescribed set around the true state, or the enviromment-dissue neighborhood of the nominal values;
Notably, in our benchmark, we implement and feature an algorithm leveraging LLM to determine the disturbance. In particular, the LLM told of the task and uses the current state and reward signal as the input. It directly outputs the disturbed results like a fake state for the agent. See more details the code in Appendix C.
Internal dynamic shift: for the environment-disse -iirrnttt-ddsuptorr This mode captures variations in the agent's 's internal model between training and testing, caused by factors such as the sim- to-real gap, measurement noise, or accidental malfunctions. The environment-disruppee introduces biases to dynamic parameters within a prescribed uncertainty set. For example, the torso length (Fig. 4 (c)) might shift from 0.3 to 0.5.
For tasks in control and robotics, the environment disruptor can alter the robot model, changing the system's 's internal dynamics (Zhang al. 2020) Zouitine al., 2024). Using Gymnasium-MuJoor MuJoooo as an example, Fig. (Jb)-(c) depict the consequences of such disrup- tion by altering the Ant robot's head and legs around its original model (Fig. (Fig.3aa) (a))
External disturbance: for the environment-disrupee workspace can result from variability in the physical world or human behavior, such as changes in wind, friction, or human intervention. The prrironmentddirrupee uses this mode modify the external task environment by altering properties and configurations within the robot's workspace
Nonstationarity in the external
by introducing abrupt external interventions (Luo et al., 2024 Pinto et al., 2017)] Ding et al. (023a).
For example, in robosuite. Fig. eff) illustrate disrupted tasks compared to the original reference in Fig. d). In these tasks, the environment disruptor changes the distance be- introduces an additional arm to actively interfere with the yellow-black robot's ability to accomplish its goal.
tween the table and the arm,
Timing of operations for disruptors. We support perturbations occurring any stage of the pro- cess and at different frequencies. Users can choose to apply perturbations at any time step or episode during the training process, exclusively during testing.

---

Published as a conference paper at ICLR 2025
Ceuntee Vake twatco Crach
Cra OR
Corsery
Gnverte
(a) Attack on robot wind
(b) Attack on robot gravity
(c) Attack on robot torso length
Figure 4: Adversary attack on robot environments, dynamics and shape with different distributions (We can also attack on robot state space, action space and reward signal, etc.).
3.3 CONSTRUCTING ROBUST RL TASKS
Robust- Gymbus--mmmmaiumm is modular benchmark that offers flexible methods for constructing robust RL tasks through three main steps. First, we select task base from the eleven options outlined in Sec. 3.1 Second, we choose a disruptor from the observation, action, and environment categories in- troduced in Sec. Sec.3.2). and specify its operation modes (random disturbance, adversarial disturbance, internal dynamic shift, and external disturbance, as detailed in Sec. f.2p. Finally, we determine the interaction process and frequencies between the disruptor, agent, and environment.
In addition these basic construction methods, our benchmark supports advanced modes: combi- nation of disruptors allows users to select multiple disruptors, such as an observation- an
and to simulate conditions where perception sensors have system noise and external disturbances from human occur; Varying operation frequencies enables disruptors to oper- ate intermittently during interactions, either at fixed intervals or random pattern to characterize accidental events and uncertainties.
4 EXPERIMENTS AND ANALYSIS
Robust- Gyybust-Gmmmsiumm offers variety of tasks for comprehensively evaluating the robustness of different RL paradigms. We demonstrate flexibility by constructing robust RI. tasks based on various task bases, incorporating disruptions with different types, modes, and frequencies, and eval- uating several SOTA algorithms on these tasks. In addition to benchmarking existing algorithms, also highlight adversarial disruption mode that leverages LLMs. Examples of robust RL tasks are shown Figure More details about the experiments can be found in Appendix
Benchmark RL algorithms. Specifically, benchmark several SOTA algorithms in their corre- sponding robust RL tasks: Standard RL: Proximal Policy Optimization (PPO) (Schulman et al. 2017), Soft Actor-Critic (SAC) (Haarojj al., 2018); Robust RL: Occupancy-Matching Pol- icy Optimization (OMPO) (Luo et al., 2024). Robust State-Confounded SAC (RSC) (Ding et al., 2023a), Alternating Training with Learned Adversaries (ATLA) (Zhang et al. 2021b). and Deep Bisimulation for Control (DBC) (Zhang et al., (021a): Safe RL: Projection Constrant-Recttfied Policy Optimization (PCRPO) (Gu et al., 2024b). Constraint-Rectifeedd Policy Optimization (CRPO) (Xu et al. 2021); Multi-Agent RL: Multi-Aentt PPO (MAPPO) Yuu et al. 2022). Independent PPO (IPPO) De Witt al. 2020).
Evaluation processes. We mainly focus on two evaluation settings: In-training: the disruptor si- multaneously affects the agent and environment during both training and testing at each time step. This process is typically used in robotics address sim-to-real gaps by introducing potential noise during training; 2) Post-training: the disruptor only impacts the agent and environment during test- ing, mimicking scenarios where learning algorithms
unaware of testing variability.
Robust metrics. In this work, we usually use the performance in the original (deployment) environ- ment as the robust metric for evaluations. While there many different formulations of the robust RL objective (robust metrics), such as risk-sensitive metrics (c.g., CVaR) (Chan et al. 2019). and the worst-case average performance when the environment shifts (Zouitine et al. 2024).

---

Published as a conference paper at ICLR 2025
4.1 EVALUATION OF STANDARD RL BASELINES
To begin, we evaluate two types of robust RL tasks: one with an observation disruptor (affecting the agent's s observed state) and the other with an action disruptor (affecting the action), both subjected to random disturbances at varying levels. We benchmark the performance of standard RL base- lines-PPO (Schulman et al., 2017) and SAC (Haarnoja et al. 2018) on robust RI. tasks based on the representative HalfCheetah-v4 task from Syymnasium-Mulode Here, S=0.1 indicates that the random disturbance over the state follows a Gaussian distribution with mean of 0 and standard deviation of 0.1 (resp.0.15). The same applies for A=0.1 I or 150.15. (i (a) and Figure33 present the results from two different evaluation processes- In- training and Post-training, respectively. The results show that the disturbance level increases, the performance of the baselines degrades quickly, particularly when the training process is unaware of potential disturbances (as seen in the Post-training results). More experiments, including those using disturbances over reward or the results for SAC, can be found in Appendix B.
as partially shown in Figure
4.2 EVALUATION OF ROBUST RL BASELINES
In this section, we evaluate robust RL tasks using environment disruptor under two representative modes: internal dynamic shift and external disturbance. The robust RL tasks are based on various task bases, including Ant-v5 and Hopper-v5 from Symmnasium-MuloCC as well as DoorCausal-vl and LiftCausal-vl from Robosuite, utilizing the In-training evaluation process.
Specifically, Figure a)-b) displays the performance of the robust RL baseline OMPO across two tasks with internal dynamic shifts: (a) Ant-v5 with varying gravity and wind strength, and (b) Hopper-v5 with changes to the robot model's shape, including torso and foot length. Experimental settings can be found in (4) and (6) in Appendix C.2, The results indicate that OMPO's performance significantly declines in non non-stationary environments compared to stationary conditions without dis- turbances.
Figures el--d) illustrate the performance of three robust RL baselines (RSC, ATLA, DBC) in two tasks from Robosuite involving disruptions on the environment with external semantic disturbances. In the DoorCausal task, the initial distance of the door from the robot and the height of the door han- dle are varied in a correlated manner. In the Causall .ift task, both the position and color of the object to be lifted are changed together according to specific patterns. RSC demonstrates greater robust- ness than ATLA and DBC, maintaining stable reward trajectories throughout the training process. However, RSC's training efficiency may need further improvement, as generates augmentation data during policy learning.
4.3 EVALUATION OF SAFE RL BASELINES
Two safe RL baselines, PCRPO Gu et al. 2024b and CRPO (Xu et al. 2021) are benchmarked on robust safety-critical tasks using the In-training evaluation process. Specifically, we assess two types of robust RL tasks based on Walker2d from Syymnaiium-MuJoCC
(a) an action-disruption attacks the agent's action with different levels; (b) the agent's 's observe immediate safety cost is disturbed in different levels. These attacks follow Gaussian distribution with mean of 0 and standard devia- tions of 0.15 5 or 0.3 for both the action and the observed cost. The outcomes and safety costs for these tasks are presented in Figures 12 a-b) and Figures 12 c-d), respectively. The performance of CRPO
#b-fCheetihh-ee vil State Atlak
Harlnneetannee A Actin Attckk
Haltheestah State Alback
HaltChestan Acckin Attack
1720
6000
5753
$751
3000
8000
4:00
4300
5272
1000
2000
2000
Fussin Stone
Taari State
-/0
120
Awhis
(a) In-training attack
(b) In-training attack
(c) Post-training attack
(d) Post-training attack
Figure 5: Adversary attack on state and action space in robust HalfCheetah- v4 tasks. attack state and A denotes attack on action.
denotes
8

---

Published as a conference paper at ICLR 2025
Happoras & Nagge BTock
DosrCoandrys Serannte Allace
LFIGouad #1 Seraat Allace
2500
2000
KIAA
AFLA
03
02
32
(a) Attack wind & gravity
(b) Attack shape
(c) Attack position
(d) Attack color
Figure 6: (a-b): Internal dynamic shift attacks on Ant-v5 and Hopper-v5 tasks, (c-d): External
semantic attacks on Robosuite tasks.
20:00
2514
1540
Bestoot
Lcco
Bastog
13.0
100
naniigg Stope
Tuiring Steps
Ta ring Steps
Training Stace
Attack safe action
Attack safe action
(c) Attack cost signal
(d) Attack signal
Figure 7: Robust safe RL rasks: Random disturbances over either the action or the agent's 's observed immediate cost feedback.
quickly degrades when disruptions occur, while PCRPO demonstrates greater robustness against disturbances in either action or observed cost. Notably, PCRPO's performance under disturbance surpasses its performance without disturbance, suggesting that introducing appropriate disturbances during training may enhance overall performance. Due to space limitations, additional results can be found in Appendix B.2
4.4 EVALUATION OF MULTI-AGENT RL BASELINES
We evaluate MARL baselines: Multi-Agen PPO (MAPPO) (Yu al. 2022) and Independent PPO (IPPO) (De Witt et al. 2020) on MA-HalfCheetthh-ve from MAMoJoCo under various dis- ruption settings affecting the agents' observed states, actions, and rewards. Using the In-training evaluation process, shown in Figure g we apply disruptions to all agents. The results indicate that the performance both MAPPO and IPPO degrades accordingly as the disruptions occur. Ad- ditionally, we conduct experiments involving partial disruptions subset of agents within the multi-agent system; further details can be found in Appendix B.3
MA-HallCheetah-wt State Allack
MA-HllCheelah--v A tion Allack
Mo-HallCheetahhwd Reward Allack
Aveerage Egisade Reward
MAPPO
Average Episodt Reward
MUPPO
Aieerage Eptodd Reward
MAFFO
25:00
HA.0O.5-0.05
2500
NUPPO-A00.
2500
MAPFO-802
2000
MAPPO-S-0.1
2000
MARPD-M-015
2000
MARPO-S--1
IPP
PP0
IRRO
15:00
3:50
PPC-N-02
1.00
IPPO-R0:15
1000
IPO---0.22
1000
PPO-A/03
1000
IPRO R-0.5
0.5
3.5
2.0
3.0
1.5
2.
3.0
0.0
1.5
2.0
3.0
Training Steps
Training Steps
Training Steps
(a) Attack all agents" state
(b) Attack all agents" action
Attack reward signal
Figure 8: Multi-Agent HalfCheetah-211 robustness: training attack on state, action, and reward for all the two agents. S denotes state, denotes action and denotes reward.
4.5 ADVERSARIAL DISTURBANCE THROUGH LLMS
In addition to benchmarking various existing RL algorithms, this section demonstrates the adver- sarial disturbance mode by leveraging featured approach with LLMs. As shown in Figure 9

---

Published as a conference paper at ICLR 2025
Ant-v4 Atlack State with LLMs
Ant-v4 Allack State with LLMs
Average Episode Reward
1200
Average Episode Reward
1200
PPO
PPO-S-10OF-CDO --1-1000--00222088--LLLMM,
1000
-PO-S-100F-C10.--008----8
100-5-100F-Uj0.2-00
PP0:S-50FF-ID2222.8)LLLM.
800
800
600
400
400
2.5
Training Steps
Training Steps
(a) LLMs v.s. uniform
(b) Varying frequencies attacking
Figure 9: LLM-based attacks with different settings.
evaluate the performance of PPO on Ant-v4 with adversarial disruptions to the agent's observed state. Different attack configurations employed, including comparisons to uniform noise and testing varying frequencies. Here, "C[0.2-0.8J" indicates that the noise level from the LLM is constrained within the [0.2, 0.8] range; "100F" (resp. "500F") signifies that the agent attacked every 100 (resp. 500) steps; and "U[0.2-0.8T] represents noise drawn from uniform distribution 240022,.. 0.8). The results show that LLM-based attacks lead to a more significant performance drop for PPO compared to that using uniform distribution (Figure Figure (b) examines how varying attack frequencies affect performance, revealing that higher-frequency attacks (PPO-S- 100F) result in greater performance degradation. Due space constraints, additional frequency experiments on other robust tasks based on Gymnasium-Muun using PPO are provided Appendix B.4
CONCLUSION
In this work, we introduce Babust-Gymmasium. a unified modular benchmark explicitly designed for robust RL. Unlike existing RL benchmarks, Robust-Gymnasium aims to evaluate the resilience of RL algorithms across a wide range of disruptions. These disruptions include perturbations at ev- ery stage of the entire agent-environment interaction process, affecting agent observations, actions, rewards, and environmental dynamics. Robust-Gymm Gymons-u provides a comprehensive platform for benchmarking RL algorithms, featuring over 60 diverse task environments across domains such as robotics, multi-agent systems, and safe RL. Additionally, we benchmark various SOTA RL algo- rithms, including PPO, MAPPO, OMPO, RSC, and IPPO, across a wide array robust RL tasks in Soouust-Gmmasiumm The results highlight the deficiencies of current algorithms and motivate the development of new ones. This work represents a significant step forward standardizing and advancing the field of robust RI,, promoting the creation of more reliable, generalizable, and robust learning algorithms.
ACKNOWLEDGMENTS
The work of S. Gu is supported by funds from Prof. Spanos' Andrew S. Grove Endowed Chair. The work of L. Shi is supported in part by the Resnick Institute and Computing, Data, and Society Postdoctoral Fellowship at California Institute of Technology. The work of Y. Chi is supported in part by the grant NSF CCF-2106778. The work E. Mazumdar supported in part from NSF- 2240110. The work of A. Wierman is supported in part from the NSF through CCF-2326609, CNS- 2146814, CPS-2136197, CNS-2 106403, UGGSDI2210560 M. Jin is supported from NSF ECCS-233177 IIS 2312744, the Amazon- Virginia Tech Initiative for Efficient and Robust Machine Learning. and the Commonwealth Cyber Initiative. We extend our sincere appreciation to Weirui Ye, Pieter Abbeel, Banghua Zhu, and Carmelo Sferrazza for their insightful and valuable discussions.
and the Resnick Institute. The work of
10

---

Published as a conference paper at ICLR 2025
REFERENCES
W Brian Arthur. Designing economic agents that act like human agents: A behavioral approach to bounded rationality. The American economic review, 81(2):353 359, 1991.
Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning, pp. 511-520. PMLR, 2021.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Dimitris Bertsimas, Melvyn Sim, and Meilin Zhang. Adaptive distributionally robust optimization. Management Science, 65(2))604-61 618, 2019.
Clément Bonnet, Daniel Luo, Donal John Byrne, Shikha Surana, Sasha Abramowitz, Paul Duck- worth, Vincent Coyette, Laurence Illing Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries Petrus Smit, Nathan Grinsztajn, Raphael Boige, Cem- lyn Neil Waters, Mohamed Ali Ali Mimouni, Ulrich Armel Mbou Sob, Ruan John de Koek, Siddarth Singh, Daniel Furelos- Blanco, Victor Le, Arnu Pretorius, and Alexandre Laterre. Ju- manji: a diverse suite of scalable reinforcement learning environments JAX. In The Twelfth International Conference Learning Representations, 2024. URL net/forus 2tt.roormmmmmdcACc0 d=CCCxxmp 9wc
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, arXiv preprint 16Xiv:1606.015: 2016.
Stephanic CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama. Measuring the reliability reinforcement learning algorithms. arXiv preprint arXiv:19/2.05663. 2019.
Xin Chen, Guannan Qu, Yujie Tang, Steven Low, and Na Li. Reinforcement learning for selective key applications in power systems: Recent advances and future challenges. IEEE Transactions on Smart Grid, 13(4):2935-299 2022.
Cédric Colas, Olivier Sigaud, and Pierre- Yves Oudeyer. How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint 1806.v880000888 2018
Murat Cubuktepe, Nils Jansen, Sebastian Junges, Ahmadreza Marandi, Marnix Suilen, and Ufuk Topcu. Robust finite-state controllers for uncertain pomdps. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 35, pp. 11992-11800, 2021.
Mark Davenport and Justin Romberg. An overview of low-rank matrix recovery from incomplete observations. IEEE Journal of Selected Topics in Signal Processing, 10(4):608-622, 2016.
Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 2020.
Esther Derman and Shie Mannor. Distributional robustness and regularization in reinforcement learning. arXiv preprint arXiv:2003.02894 2020.
Wenhao Ding, Laixi Shi, Yuejie Chi, and Ding Zhao. Seeing is not believing: Robust reinforce- ment learning against spurious correlation. In Thirty-seventh Conference on Newral Information Processing Systems, 2023a.
Wenhao Ding, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li. and Ding Zhao. A survey safety-critical driving scenario generation-a methodological perspective. IEEE Transactions on Intelligent Transportation Systems, 24(7):6971-698 2023b.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pp. 1329-1338. PMLR, 2016.
11

---

Published as a conference paper at ICLR 2025
Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, and Mar- ius Lindauer. Dacbench: A benchmark library for dynamic algorithm configuration. IJCAI, 2021.
Vineet Goyal and Julien -eemm-Ceeeent.. Robust markov decision processes: Beyond rectangularity. Mathematics of Operations Research, 2022.
Shangding Gu, Jakub Grudzien Kuba, Yuanpei Chen, Yali Du Long Yang, Alois Knoll, and Yaodong Yang. Safe multi-agent reinforcement learning for multi-robot control. Artificial In- felligence, 319:103905, 2023.
Shangding Gu, Puze Liu, Alap Kshirsagar, Guang Chen, Jan Peters, and Alois Knoll. Roscom: Robust safe reinforcement learning stochastic constraint manifolds. IEEE Transactions on Automation Science and Engineering, 2024a.
Shangding Gu, Bilgehan Sel. Yuhao Ding, Lu Wang. Qingwei Lin, Ming Jin. and Alois Knoll. Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp- 21099-21 11999-211066 106, 2024b.
Shangding Gu, Long Yang. Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories and applications. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024c.
Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang, Qingwei Lin, Alois Knoll, and Ming Jin. Safe and balanced: A framework for constrained multi-objective reinforcement learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long- long-horizon tasks via imitation and reinforcement learning. In Conference Robot Learming, pp. 1025-1037. PMLR, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Songyang Han, Sanbao Su, Sihong He, Shuo Han, Haizhao Yang, and Fei Miao. What is the solution for state adversarial multi-agent reinforcement learning? arXiv preprint arXiv:2212.02705, 2022.
Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. Transactions on Machine Learning Research, 2023.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast bellman updates for robust MDPs. In International Conference Machine Learning, pp. 1979-1988. PMLR, 2018.
Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Partial policy iteration for robust markov decision processes. Journal of Machine Learning Research, 22(275):1-46, 2021.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks neural network policies. arXiv preprint arXiv:1702.0223 2017.
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
David L Kaufman and Andrew Schaefer. Robust modified policy iteration. INFORMS Journal on Computing, 25(3):996-400, 2013.
Olga Klopp. Karim Lounici, and Alexandre B Tsybakov. Robust matrix completion. Probability Theory and Related Fields, (69(1-2)5533566 2017.
12

---

Published as a conference paper at ICLR 2025
Joseph Peter Kozak, Ruizhe Zhang. Matthew Porter, Qihao Song, Jingeun Liu, Bixuan Wang, Rudy Wang, Wataru Saito, and Yuhao Zhang. Stability, reliability, and robustness of gan power devices: A review. IEEE Transactions on Power Electronics, 38(7):8442-844 44444441111,, 2023.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684 2020.
Zhongyu Li, Xuxin Cheng, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath. Reinforcement learning for robust parameterized locomotion control of bipedal robots. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2811-281 1-177. IEEE, 2021.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
Qing Liu, Yulan Li, and Katherine Odem-Davis On robustness noninferiority clinical trial designs against bias, variability, and nonconstancy. Journal of biopharmaceutical statistics, 25(1):206~ 225, 2015.
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. arXiv preprint arXiv:2205.14691, 2022.
Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. Ompo: A unified framework for under policy and dynamics shifts. In Forty-first International Conference on Machine Learning, 2024.
A Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra. Benchmarking reinforcement learning algorithms real-world robots. In Conference robot learning, pp. 561-591. PMLR, 2018.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically-plausbbl perturbations. In 2017 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 9932-3939, IEEE, 2017.
Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, et al. Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:2012.0742) 2020.
Eric Mazumdar, Kishan Panaganti, and Laixi Shi. Tractable equilibrium computation in markov games through risk aversion. arXiv preprint 1rXiv:2406.14156, 2024.
Ishita Mediratta, Qingfei You, Minqi Jiang, and Roberta Raileanu. The generalization gap in offline reinforcement lcarning. arXiv preprint arXiv:2312.05742, 2023.
Andrew Melnik, Luca Lach, Matthias Plappert, Timo Korthals, Robert Haschke, and Helge Ritter. Using tactile sensing to improve the sample efficiency and performance of deep deterministic pol- icy gradients for simulated in-hand manipulation tasks. Frontiers in Robotics and AI, 8:538773, 2021.
Jorge A Mendez, Marcel Hussing, Meghna Gummadi, and Eric Eaton. Composuite: A composi- tional reinforcement learning benchmark. In Conference Lifelong Learning Agents, pp. 982- 1003. PMLR, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, and Georg Ostrovski. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-35 -333,993,333 2015.
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction, 4(1):276-315, 2022.
13

---

Published as a conference paper at ICLR 2025
Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang. Xuanlin Li, Stone Tao, Zhiao Huang, Zhi- wei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXlv:2107.14483, 2021.
OpenAl. Gpt-4 technical report. 2023.
Yuxin Pan, Yize Chen, and Fangzhen Lin. Adjustable robust reinforcement learning for online 3d bin packing, arXiv preprint arXiv:2310.04333, 2023.
Ian Parberry. Introduction to Game Physics with Box2D. CRC Press, 2017.
Beomsoo Park and Benjamin Van Roy. Adaptive execution: Exploration and learning price
impact. Operations Research, (3(5):1058-10766 2015.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary, Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632, 2017.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Alexre-le Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gra- dients. Advances in Neural Information Processing Systems, 01112208-122221 2021.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference on Machine Learning, pp. 2817-2826. PMLR, 2017.
Matthias Plappert, Marcin Andrychowicz. Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforce- ment learning: Challenging roboties environments and request for research. arXiv preprint arXiv: 1802.09464 2018.
Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung- Yen Yang, Ruslan Part- sey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakr- ishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars, and robots. The Twelfth International Conference on Learning Representations, 2024. https: URLhttipss//ppereeei... Set//Eorum.11=422nn3300
Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The colosseum: benehmark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.08191, 2024.
You Qiaoben, Xinning Zhou, Chengyang Ying, and Jun Zhu, Strategically-tinee state-observation attacks on deep reinforcement learning agents. In ICML 2021 Workshop on Adversarial Machine Learning, 2021.
Raghu Rajan, Jessica Lizeth Borja Diaz, Suresh Guttikonda, Fabio Ferreira, André Biedenkapp. Jan Ole von Hartz, and Frank Hutter. Mdp playground) An analysis and debug testbed for rein- forcement learning. Journal of Artificial Intelligence Research, 77:821-890 2023.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems XIV, 2018.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01700 7(1):2 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0633 2017.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-bodi body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024.
14

---

Published as a conference paper at ICLR 2025
Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05766 2022.
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, Matthieu Geist, and Yuejie Chi. The curious price of distributional robustness in reinforcement learning with a generative model. Advances in Neural Information Processing Systems, 36:79903-79917 2023.
Laixi Shi, Jingchu Gai, Eric Mazumdar, Yuejie Chi, and Adam Wierman. Breaking the curse of mul- tiagency robust multi-agent reinforcement learning. arXiv preprint 2xXiv:249.200067 2024a.
Laixi Shi, Eric Mazumdar, Yuejie Chi, and Adam Wierman. Sample-efficient robust multi-agent re- inforcement learning in the face of environmental uncertainty. arXiv preprint arXiv:2404.1809, 2024b.
Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcementt learning. arXiv preprint arXiv: 1199900007708 2019.
Ke Sun, Yi Liu, Yingnan Zhao, Hengshuai Yao, Shangling Jui, and Linglong Kong. Exploring the training robustness of distributional reinforcement learning against noisy state observations. arXiv preprint wrXiv:2109.08776, 2021.
Zhongchang Sun, Sihong He, Fei Miao, and Shaofeng Zou. Constrained reinforcement learning under model mismatch. arXiv preprint arXiv:2405.012. 2024.
Aviv Tamar, Shic Mannor, and Huan Xu. Scaling up robust MDPs using function approximation. In International conference on machine learning, pp. 181-189. PMLR, 2014.
Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, and Soumik Sarkar. Robustifying reinforce- ment learning agents via action space adversarial training. In 2020 American control conference (ACC), pp. 3959-3964. IEEE, 2020.
Takumi Tanabe. Rei Sato, Kazuto Fukuchi, Jun Sakuma, and Youhei Akimoto. Max-min off-policy actor-critic method focusing on worst-case robustness to model misspecification. Advances in Neural Information Processing Systems, 35:6967-699 -681,, 2022.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control. In International Conference on Machine Learning, pp. 6215-6224.
PMLR, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1109/ROS.22012.33866.
Mark Towers, Ariel Kwiatkowski, Jordan Terry. John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407. 17032, 2024.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020.
Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157): 1974.
Daniel Vial, Sanjay Shakkottai, and R Srikant. Robust multi-agent bandits over undirected graphs. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 6(3):1-57, 2022.
Oriol Vinyals, Igor Babuschkin, Wojeiech Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350-352 2019.
He Wang. Laixi Shi. and Yuejie Chi. Sample complexity of offline distributionally robust linear markov decision processes. arXiy preprint arXiv:2403. 12946, 2024.
15

---

Published as a conference paper at ICLR 2025
Maciej Wolczyk. Michal Zajac, Razvan Pascanu, Lukasz Kucinski, and Piotr Milos. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Informa- tion Processing Systems, 34:28496-2855 2021.
Eric M Wolff, Ufuk Topcu, and Richard M Murray. Robust control of uncertain Markov decision processes with temporal logic specifications. In 2012 IEEE 51st IEEE Conference Deciston and Control (CDC), pp. 3372-3379. IEFE, 2012.
Zikang Xiong, Joe Eappen, He Zhu, and Suresh Jagannathan. Defending observation attacks in deep reinforcement learning via detection and denoising, arXiv preprint arXiv:2206.077 0780882000071888, 2022.
Huan Xu and Shie Mannor. The robustness robustness-performmeee tradeoff in markov decision processes.
Advances in Neural Information Processing Systems, 19, 2006.
Huan Xu and Shie Mannor. Distributionally robust Markov decision processes. Mathematics of
Operations Research, 37(2):288-33 2012.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pp- 11480-11491. PMLR, 2021.
Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild- time: A benchmark of wn-the-wild distribution shift over time. Advances in Neural Information Processing Systems, 35:10309-10324, 2022.
Christopher Yeh, Victor Li, Rajeev Datta, Julio Arroyo, Nicolas Christianson, Chi Zhang, Yize Chen, Mohammad Mchdi Hosseini, Azarang Golmohammadi, Yuanyuan Shi, et al. Sustaingym: Reinforcement learning environments for sustainable energy systems. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games, Advances Neural Information Processing Systems, 35:24611 1-24624, 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng Hc, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094 1000. PMLR, 2020.
Zhaocong Yuan, Adam W Hall, Siqi Zhou, Lukas Brunke, Melissa Greeff, Jacopo Panerati, and Angela Schoellig. Safe-control-gym: A unified benchmark suite for safe learning-basee control and reinforcement learning in robotics. IEEE Robotics and Automation Letters, 7(4::11142- 11149, 2022.
Amy Zhang, Yuxin Wu, and Joelle Pineau. Natural environment benchmarks for reinforcement learning. arXiv preprint
2018.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Con- ference Learning Representations, 2021a. URL https:://pperreveew.. wwt.:/wwwwww.ee.eeee.eeeeeeeeeee forum?
id= 2FCDDKRREU
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Ad- vances in Neural Information Processing Systems, 33:21024-2103:
2020.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning state observations with leamned optimal adversary. In International Conference on Learning Representations, 2021b. /ttphhttss://pperreee
net/ wtt.re/wwwweeeeeeeooooooooaaaa/e
Zhengfei Zhang, Kishan Panaganti, Laixi Shi, Yanan Sui, Adam Wierman, and Yisong Yue. Dis- tributionally robust constrained reinforcement learning under strong duality. arXiv preprint arXiv:2406. 15788, 2024.
16

---

Published as a conference paper at ICLR 2025
Zhili Zhang, Yanchao Sun, Furong Huang, and Fei Miao. Safe and robust multi-agent reinforce- ment learning for connected autonomous vehicles under state perturbations. arXiv preprint arXiv:2509.11057, 2023.
Ziyuan Zhou and Guanjun Liu. Robustness testing for multi-agent reinforcement learning: State perturbations on critical agents. arXiv preprini arXiv:2306.061 36, 2023.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiri- any, and Yifeng Zhu. robosuite: modular simulation framework and benchmark for robot learning. arXiv preprint 1222:23009.1299, 2020.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint Xrx:
2019.
Adil Zouitine, David Bertoin, Pierre Clavier, Matthieu Geist, and Emmanuel Rachelson. Rrls: Ro- bust reinforcement learning suite. arXiv preprint arXiv:2406 08406, 2024.
17

---

Published as a conference paper at ICLR 2025
A
RELATED WORKS
Related RL benchmarks. To the best of our knowledge, Zouitine et al. (2024) is the only existing benchmark designed specifically for robustness evaluations, with the same goal of this work. introduced six continuous control tasks in Gymnasium-Muuke shifts. A clear lack of standardized benchmarks is present that offer a wide range of diverse tasks and account for uncertainty and disruptions over multiple stages throughout the interaction process, (not only the environment). Such a comprehensive evaluation platform is essential for the community to evaluate existing efforts and inspire new algorithms. Robust-Gyymnat evaluation of RL as a unified modular benchmark that supports over sixty diverse tasks in robotics and control for comprehensive evaluation, and accounting for different types of uncertainty and disruptions across multiple stages of the interaction process.
designed to address environmental
fills the gaps for robust
Moreover, enhancing robustness against environment shifts be seen as slight generalization to unseen tasks or environments. A non-exhaustive list of relevant benchmarks includes: a domain generalization benchmark in offline RL (Mediratla et al. 2023), KeradWorld for meta-RL (Yu et al. 2020). generalization benchmark for robot manipulation (Pumacay et al. 2024), SustainGyym
2024). continual robot learning (Wol- czyk et al. 2021), b. lifelong robot learning (Liu et al. 20244. skill manipulation robot learning (Mu cet [2221), ) safe RI. (Ray et al. 2019; Yuan et al. 2022| Gu et al., 2023), multi-task/oojjeciive RI. (Mendez et al. 2022 Gu et al. (2025). human-robot collaboration tasks (Puig et al. (022). dynamic algorithm configuration (Eimer et al. 202I). RL in JAX (Bonnet et al. 2024), procedurally gener- ated environments (Küttler et al. [2020]. DM control (Tunyasuvunakool et 20,220 arcade learning environments (Bellemare et al.] 2013). a MDP playground for evaluation (Rajan et al. 2023), and
generalization for sustainable energy systems (Yeh al.,
others (Marklund et al. 20200] Yao et al. 2022).
RL works involving tasks for robust evaluation. Although not primarily focusing on building a benchmark for robust RI,, there exists lot of prior works or benchmarks that involves tasks for robust evaluation. While they typically support a few robust evaluation tasks associated with only one disruption type, which is not sufficient for comprehensive evaluations for robustness in real- world applications.
Specifically, there exists lot of benchmarks for different RL problems, such as standard RL, safe RL, multi-agent RL, offline RL, and etc. These benchmarks either don't 't have robust evaluation tasks, or only have a narrow range of tasks for robust evaluation (since robust evaluation is not their primary goals), such as Duan et al. (2016) support 5 tasks with robust evaluations in control. Besides, there are many existing robust RI. works that involve tasks for robust evaluations, while they often evaluate one-off and a narrow range of tasks in specific domains, such as 8 tasks for robotics and control (Ding et al. 2023a). 9 robot and control tasks in StateAdvRL (Zhang et al. 2020). robust RL tasks in RARL Pinto et al. 2017), 3D bin-packing task (Pan et al. (023). Since their primary goal is to design robust RL a algorithms, but not a platform to evaluate the algorithms.
Robustness in single-agent RL. Robustness is key principle in designing RL algorithms, train- ing processes are often idcalized and limited in data and scenarios, while real-world environments changeable, unpredictable, and highly diverse. An emerging body of work focuses on devel- oping robust RL. algorithms that can withstand potential uncertainties, perturbations, and attacks during real-world execution. These efforts can largely be categorized under our unified robust RL framework (Sec. [2). which formulates uncertainty events affecting the agent-enviromment interac- tion behaviors of three types of disruptors. Our proposed Mobustt-ymmasite encompasses all types of robust RL tasks within this framework, providing a flexible and comprehensive platform for evaluating and developing robust RL algorithms.
Specifically, prior works typically involve one type of disruptors: Zhang et al. (2020) 20216); Han jet al. (2022); ; Qiaoben et al.) (202T): : [Sun et al.) (202T); Xiong et al. (2022) studied the uncertainty of agent's 's observed state, controlled by the observation-disruppe
who can add restricted noise or perform adversarial attack; Tessler et al. 2211)) : Tan et al. (2020) considered the robustness w.r.t. the uncertainty of the action, where the action is possibly distorted by the action-disruptor abruptly or smoothly before forwarding to the environment to be executed; A large amount of prior works focus on dealing with the perturbation/shift on the environmental controlled by the onvironment-disupp includes the reward function, the dynamics, or the task itself, ranging from theory (Iyengar) (2005]
Xu & Mannor 2012 Wolff et al., 2012 Kaufman & Schaefer 2013 Ho et al. 2018 Smirnova al.
18

---

Published as a conference paper at ICLR 2025
2019
Ho al. 2021: Goyal & -eanm-Clement 2022 Derman & Mannor 2020 Tamar al. 2014
Badrinath & Kalathil 2021 Shi & Chi 2022; Shi et al. 2023; Wang et al. (022)) to applications
(Pinto et al. | Pattanaik al. 2017 Tanabe et | Ding et al. 2023a). Besides them, only
a few works consider more complex scenarios that more than one disruptors are involved (aaveee(MMnnn (Mandlekar
al.
2017). See|Moos al. (2022) for a recent review.
Robustness in safe RL and multi-agent RL. Besides the class of standard single-agent RL, ro- bustness in RL algorithms are ubiquitously demanded and has emerges growing line of works for other RL problems such as partially observable Markov decision processes (POMDPs) (Cubuktepe)
al. 202I). safe RL (Liu al.
2022] Sun al., 2024| Zhang et al. 2024 Gu et al. 2024a c) and
mulli-a -agent RL (Vial et al., [2022] Han et al. 2022 He et al. [023 Zhou & Liu 2023; Zhang et al.
2023
bb2Tb Shi et al. 2024b a Mazumdar al. 2024). Additional challenges arise when combin-
ing robustness requirements with issues such as safety constraints and strategic interactions, which
are often understudied and lack standardized benchmarks for evaluation. Our Robust- Sabbust-Gymmsiium not only provides single-agent RL tasks but also encompasses broader range of RL paradigms, including safe RL and multi-agent RL. This enables faster and more comprehensive process for designing and evaluating robust RL algorithms across wider array of RL tasks.
B SUPPLEMENTARY EXPERIMENTS AND ANALYSIS
B.1 SUPPLEMENTARY FOR EVALUATION ROBUSTNESS OF STANDARD RI.
As shown in Figures 10 they demonstrates the robustness of PPO in the HalfCheetah-v4 environ- ment under various adversarial conditions. Fach graph presents the average episode reward across training steps, contrasting the performance of the standard PPO algorithm against its adaptations un- der diverse adversarial attack parameters. Specifically, the figure for in nn-Trainin Attack on Reward (Figure 10 (a)) investigates how modifications the rewards during training influence the learning performance, employing multiple levels of perturbation. Moreover, the graph for Post-Training At- tack on Reward (Figure I0(b), (b)) assesses how the trained policy withstands alterations the reward signals post-trainins training. The experimental results suggest that training an RL agent with disturbances and then testing it in ideal environments may lead to improved reward performance test scenarios. Similarly, we conducted experiment to evaluate the robustness of another popular RL baseline, SAC. As shown in Figure II] the performance of SAC degrades under a disturbance attack.
This experiment aids in understanding the stability and robustness of RL policies under adversarial conditions, which is pivotal for deploying these models in real-world scenarios where they may encounter unexpected or adversarial changes in input data.
B.2 SUPPLEMENTARY FOR EVALUATION ROBUSTNESS OF SAFE RL
As depicted in Figures 12 a) and (b), we implement PCRPO (Gu et al. 2024b) and CRPO (uu et al.
2021]. SOTA safe RL algorithms, in robust safety-critical tasks. We selected representative task from robust safe RL to assess the effectiveness of the safe RL algorithm. Specifically, we introduce a disruptor to attack the Walker2d robot's observations during training, as shown in Figures 12 (b)). Under these adversarial attacks, the reward performance of both PCRPO and CRPO degrades. The attacks follow a Gaussian distribution with mean of and standard deviation of 0.3. highlighting the importance of considering disturbance testing before deploying safe RL models in real-world applications.
B.3 SUPPLEMENTARY FOR EVALUATION ROBUSTNESS OF MULTT-AGENT RI.
As shown in Figures 13 (d), (e), and (f), we investigate partial state, action, and reward attacks on MAPPO, where only a subset of agents or aspects is attacked. These figures show a smaller drop in performance, indicating partial attacks are less harmful compared to full attacks (See Figure
B.4 FREQUENCY ATTACK
We offer interactive modes that support step-wise, variable interactions between disruptors, agents, and environments, allowing users apply perturbations at any point in time and in any manner they
19

---

Published as a conference paper at ICLR 2025
HalfCheetah-v4 Reward Attack
HalfCheetah-we -44 Reward Attack
Average Episode Reward
8000
7000
6812
Average Episode Reward
6351
5751
6000
5000
4000
4000
2000
PPO
PPO-R-0.1
2000
PPO-R-0.15
1000
1.5
2.0
2.5
3.0
Training Steps
PPO
R=0,1
R=0.15
(a) In- in-Training Attack on Reward
(b) Post-Training Attack Reward
Figure 10: HalfCheetah-v4 robustness: training attack,reward. Specifically, in experiment (a), train the PPO algorithm under conditions: without reward attack, and with a reward attack in- volving Gaussian noise with standard deviations of 0.1 and 0.5, respectively. In both reward attack scenarios, the noise has a mean of O, with attack noise standard deviations of 0.1 and 0.15, respec- tively. In experiment (b), test the trained PPO models that attacked during training with reward attacks, using standard deviations of 0.1 and 0.15. After the attack- training, the mod- els are evaluated in environments without any attacks.
HalfCheetah-v4 State Attack
Average Episode Reward
8000
SAC
SAC-S-0.01
6000
SAC-S-0.1
4000
2000
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Training Steps
Figure 11: Evaluation SAC robustness HalfCheetah-v4 tasks.
Safetyyalker2d
SafetyWalker2div4
Average Episode Reward
3500
Average Episode Cost
PaUrO-5-0.3
3000
CRPO
70
CRPO-5-0.3
2500
Lim 2-40
2000
1500
1000
PORPO
500
PCRPO-S-0.3
CRPO
CR
1
2
8 les
6
Training Steps
Training Steps
(a) Attack on safe state
(b) Attack safe state
Figure 12: Robust Safety RL Tasks.
choose. As shown in Figure 14 the frequency of attacks on tasks illustrated. Perturbations can occur various points during the training and testing phases, with different frequencies.
20

---

Published as a conference paper at ICLR 2025
MA-HailChherrah-- Portial State Attack
MH-HalltChatthh-v Partial Action Attack
MA-HallChaetah-44 Partial Raward Attack
Average Episade Reward
25:00
MAPPO 65
MAPPO
Aver Episode Rewwadd
14PPD
MAPPO
Average sgidade Reward
MAFFO PR00.1
MARPO
2000
P5-..2
2000
HAPPO-PA...
MAFPO P80.2
15:00
35500
1500
1000
2000
1000
520
550
500
0.5
2.0
0.0
055
1.5
2.0
0.0
15 2.0
3.0
Training Stops
Training Steps
Training Steps
(a) Attack partial state
(b) Attack partial action
(c) Attack reward signal
Figure 13: Multi-Agent HalfCheetah-244 robustness: training attack on state, action, and reward for all the two agents. S denotes state, A denotes action and R denotes reward, P denotes partial attacks. Some of agents are attacked with various attack factors.
adversary attack
attacker
environment
every step attack
agent
every episode attack
every epoch attack
every several-epoch attack
Figure 14: Different levels of robust RL's attack frequency.
As shown in Figure 15 we provide the results of robustness evaluations on the Ant-v4 task un- der frequency-based adversarial attacks. The figure consists of two subplots, each examining the performance of PPO-based algorithms under different attack levels and frequencies. In Figure (a), we explore the impact of varying attack intensities at a fixed attack frequency (every 50 steps) tar- geting the agent's actions. As shown, PPO without adversarial intervention achieves the highest episode rewards. However, as the attack intensity increases (PPO-F50-A-0.01. PPO-F50-A-0.05, PPO-F50-A-0.11 I), the performance declines progressively. The highest intensity attack (PPO-F50- A-0.1) results in the most significant reduction rewards, indicating a substantial performance drop under stronger attacks. In Figure (b), we examine the effect of varying attack frequencies while keeping the attack intensity constant. Here, PPO-F5O-S-O.1 and PPO-F100-S-O.1 repre- sent attacks occurring every 50 and 100 steps, respectively. The results indicate that more frequent attacks (PPO-F50-S-0.15 lead to a larger decline in episode rewards compared to less frequent at- tacks (PPO-F100-S-O11) 1-0.15).. This suggests that attack frequency plays critical role in determining the robustness of PPO algorithms. Overall, these findings demonstrate that both the intensity and frequency of attacks significantly affect the performance RL agents, with higher intensities and more frequent attacks causing greater degradation in task performance.
C OTHER SETTINGS OF THE FRAMEWORK
C. BENCHMARK FEATURES
The features of the benchmark follows:
High Modularity: It is designed for flexible adaptation to a variety of research needs, featuring high modularity support wide range of experiments.
Wide Coverage of It provides a comprehensive set of tasks to evaluate robustness across different RL scenarios. An overview of the task list is shown in Figure 17
High Compatibility: It can be seamless and compatible with a wide range existing environments. An example is shown in Listing 1 Moreover, this benchmark supports vectorized environments,
21

---

Published as a conference paper at ICLR 2025
Ant-v4 Frequency Attack on Action
Ant-v4
Average Episode Reward
5000
Average Episode Reward
5000
4000
3000
3000
2000
PPO
2000
PPO-F50-A-0.09
PPO
PPO F50 0.05
1000
PPO-F50-S 0.15
0
PPO-F50-A-0.1
PPO-F100-S-0.115
1.5
2.0
2.5
2.0
2.5
3.0
Training Epochs
Training Epochs
(a) Different level of attacks with same frequency (b) Different level frequency with same attack
Figure 15: Robust Ant Tasks with Frequency attacks.
which means can be useful enable parallel processing of multiple environments for efficient experimentation.
From robust robust__gymaasinle gymnasium. configs. cenffigg..bbu.....m....m sobust_settinn import get_config parse_args|
args get_config() parse args
action env.action space. sample
robust_input = ("action": action, "robust_config" args)
observation, reward, terminated, truncated, info env step (robust (nnuut
Listing An example of python interface
Support for New Gym API: It fully supports the latest standards in Gym API, facilitating easy integration and expansion.
Adversarial Attack with LLMs: We feature an approach that leverages LLMs adversary poli- cies. An example is shown in Listing
prompt "This is about robust reinforcement learning setting; we want you adversary policy. If the current reward exceeds the previous reward value, please input some observation noise to disturb the environment and improve the learning algorithm's s robustness." The noise should be in this area:" +str (aargg..r high)
args. the current reward:" stewaa))d) + ", the previ reward is" revise the current envi ronment state values:" + str (observation) just output the revised state with its original format" "do not output any other things."
(self
"please slightly
prompt_state state observation prompt_state
(prompt)
Listing 2: An example of LLMs for robust learning
C.2 ROBUST NON-STATIONARY TASKS:
Inspired by OMPO Luo et al. 2024p, we provide various task settings to evaluate policy robust- ness, as illustrated in Figure 16 During policy learning, we introduce adversarial attacks during walking or running tasks by altering robot dynamics and environmental conditions. For instance, we stochastically adjust the robot's gravity and the environment's wind speed, introducing uncer- tain disturbances during policy learning. Additionally, stochastically modify the robot's physical shape throughout the learning process to test and enhance policy robustness.
Specifically, in non-stationary Ant-v5 Tasks, during each step, introduce noise into the agent's dynamics by attacking factors like the Ant robot's gravity and the wind speed in the robot's en- vironment. As demonstrated in Equation (2) for attacks at initial and training steps, introduce deterministic perturbations to the Ant robot, such as variations gravity and environmental wind speed, the pseudo code is shown in Listing Furthermore, Equation 3 is for initial noise, and
22

---

Published as a conference paper at ICLR 2025
Running task
Flat road task
Uneven road without disturbance task
Walking
Uneven road task
Uneven road with disturbance task
(a)
(b)
(c)
Figure 16: Examples of robust non-stationary tasks (Luo al. 2024).
these Equarions to consider the incorporation of stochastic disturbances into the Ant robot model, again including factors like gravity fluctuations and wind speed variations, the pseudo code is shown in Listing [ Apart from wind and gravity disturbances, we also investigate the robot shape disturbances during policy learning, as shown in Equations -
Equation (4) is for noise during training
(). and example of pseudo code is shown in Listing
At the initial and training steps, if we choose non-stationary attack as deterministic noise,
Gravity 14.715,
Ant deterministic noise
Wind 1.0.
if we choose non-stationary attack stochastic noise,
Gravity ~ Uniform/ (9.81, 19.82),
Ant and Humanoid stochastic noise at initial steps
Wind Uniform(O.. (0.8, 1.2).
During training steps, if we choose non-stationary attack as stochastic noise, where ispisode denotes the training step number,
Gravity = 14.715 + 4.905 sin (0.5 depisode), (4)
Ant and Humanoid noise during training
Wind = 1.0 + 0.2 sin (0.5 lepisode).
Torso Length ~ Uniform 0.3),
Walker stochastic noise at initial steps
(5)
Foot Length Uniform Uniform(0.05, 0.15).
Torso Length = 0.2 + 0.1 sin 'episode) )
Walker Stochastic noise
(6)
Foot Length 0.1 0.05 sin repisade)
Torso Length ~ Uniform 0.5),
Hopper stochastic noise at initial steps
Foot Length Uniform Aniform(0.29,1 0.49).
Torso Length 0.4 0.1 sin iepisode )
Walker Stochastic noise
(8)
Foot Length 0.39 + 0.1 sin Rspisode).
if deter_noise: noise:
gravity = 11.
wind = 1.
else:
gravity np. random. uni form (9. 81, 19.82) wind np. re.dam.om uni form 8, 1.2)
Listing 3: An example of Non-stationary Ant python code for initial steps.
23

---

Published as a conference paper at ICLR 2025
if config. deter noise:
gravity 145 wind = 1.
else:
gravity 14.715 + 4.905 np.sin
wind 1. 0.2 np.sin (0.5 1 i_spisode)
Listing An example Non-stationary Ant python code for training steps.
if config.deter_nolle deter_noise: torso_les 0.2
foot_len = 0.1
else:
torso_len 0.2 + 0.1 np..sin (0.3 i_episode)
foot_len 0.1 0.05 np.sin (0.3
i_epi sode)
Listing 5: An example of Non-stationary Walker python code for training steps.
D REPRESENTATIVE EXAMPLES OF USING Robust-Gymnasiun
In this section, we present an overview of the task environments, as illustrated in Figure 17 tionally, show some bobustness-focussee tasks, detailed in Tables [
Addi-
Moreover, inspired by Yu et al., 2020), to illustrate the standardized usage of our benchmark, we propose the following framework for evaluation settings. These align with the principles of bench- marking, including standardized performance metrics and evaluation protocols:
Random attack (Easy) Adversarial attack (Hard). Random Attack (Fasy): Ran- dom noise, drawn from distributions such as Gaussian or uniform, is added to the nominal variables. This mode is applicable
sources of perturbation and allows for testing ro- bustness under stochastic disturbances, c.g., see Figure 5 (a) and (b). Adversarial Attack (Hard): An adversarial attacker selects perturbations to adversely degrade the agent's per- formance. This mode can be applied to observation or action perturbations and represents the most challenging scenario, e.g., see Figure (a) (b).
Low state-action dimensions (Easy) High state-action dimensions (Hard) As the state and action space dimensions increase, the tasks become significantly more challeng- ing. The difficulty level of tasks typically progresses from Box2D, Mujoco tasks, robot manipulation, and safe tasks to multi-agent and humanoid tasks. For instance, the Hu- manoid task, with 51-dimensional action space and 151-dimensional state space, is substantially more challenging than the Mujoco Hopper task, which has a 3-dimensional action space and 11 I-dimensional state space.
Table 1: A List of Examples for Robustness in MuJoCo Tasks
Tasks' Robust type
Robust State
Robust Action
Robust Reward Robust Dynamics
Ant-v2-v3-wl-v5
HallCheetaah-22433x4445
Hopper-vv-v3-vv-v55
Walker2d-vv2-y3-vv--55
Swinmeer-v..v3.v4..555
Humanoidd-v2-v3-vd..5
V
HumanoddSiandupp22233vvww
V
Pusher-x2-33-v4.-1
Reacher-v2-v3--vl-55
E EXPERIMENT SETTINGS
We deploy several SOTA baselines in our benchmark to evaluate their robustness across various challenging scenarios. The implementation parameters associated with these methods are provided in Tables 1
24

---

Published as a conference paper at ICLR 2025
Observed state/ Observed reward/ Action
Environment
Class of Tasks \
disturbance types
Adversarial (LLM)
Dynamics disturbance
random
Box2D
Control
Gymnasium-MuloCo
Robot Navigation
Dexterous Hand
Single-agent
Adroit Hand
Robot Manipulation (diverse robots and tasks)
Fetch Manipulation
Franka Kitchen
robosuite
Humanoid
Safety
Safety MuloCo
Multi-agent
MAMuloCo
Figure 17: An overview of task environments and supported disruptions in Robust-Gymnasium.
Table 2: A List of Examples for Robustness in Box2d Tasks
Tasks Robust Type
Robust State Robust Action
Robust Reward
CarRacing-v2
Lumarlaanerrommimouuss33
BipedalWallerr-3 Luner_annerr-v3 (Discrete Task)
Since RL performance can be significantly influenced by different random seeds (Henderson et al. Colas al. (0188), we aim balance computational costs and experimental rigor by typically using 5 seeds in our experiments. For single-agen settings, we use the same 3 seeds across all baselines ensure a fair comparison. In aueti-agent settings, where variance tends to be higher, we employ the same 5 seeds across all baselines to achieve more reliable evaluation. We recognize the importance of robust experimental evaluation and intend include additional seeds in future studies to further examine RI. robustness.
2018
Moreover, when selecting different robust disturbance parameters, the choice can significantly af- feet the evaluation of various RL algorithms. For instance. in standard RL, disturbances can be modeled as Gaussian distributions, such as (0, 0.1) or (0, 0.15), applied to the state or action space, which can notably influence the performance of algorithms like PPO. Alternatively, uniform disturbances within the range [0.2, 0.8] can be used to effectively assess the robustness of standard RL approaches. For robust RI, additional parameters bustness. For example, as for the evaluation robustness of MOPO method, wind speed may follow uniform distribution U (0.8, 1.2), while robot gravity may vary uniformly within (9.811, 19.82). Other factors include variations in the robot's physical dimensions, such as the torso length, which can be expressed as the original length plus 0.1 sin..... 2-iteration number), and the foot length, which follows similar perturbation. Our benchmark also incorporates robust parameters to evaluate the safety of RL algorithms. For example, Gaussian disturbances (0, 0.3) are particularly effective for assessing the robustness of safe RL algorithms such as PCRPO and CRPO. In the context of multi- agent RL, robustness can evaluated by selectively perturbing partial agents. Gaussian distur- bances, such as (00 0.1) or (0, 0.15), applied to the state or action space, provide significant insights into the robustness of algorithms like MAPPO and IPPO.
often employed to evaluate algorithm
25

---

Published as a conference paper at ICLR 2025
Table 3: A List of Examples for Robustness in Robosuite Tasks
Tasks
RRbbst Type
Robust State
Robust Action
Robust Reward
Lift
Door
NutAssembly
PickPlace
Stack
Wipe
ToolHang
TwoArml.it
TwoArmPeginHon
Two Armandoveerr
TwoAmmTanssoon
MultiDoor
Table 4: List of Examples for Robustness in Safety Tasks
Tasks Robust Type AatetyAnt-v4
Robust State Robust Action
Robust Reward
SafetyHalfCheetah-
SafetyHopper-v4
Safety Walker2y4422
SafeeySSwimmer-v4
SafetyHumanoid-vv
SafetyHumanoidStaddupp-- v4
SafetyPusher--v4
ReactyRea
Table 5: A List of Examples for Robustness in Adroit Hand Tasks
Tasks Robust Type
Robust State Robust Action Robust Reward
AdroiiHandDoor-vl
AdroitHandHammerrv
AdroitHandPen-vl
AdroitHandRelocateevi/
Table 6: A List of Examples for Robustness in Hand Manipulation Tasks
Tasks Robust Type
Robust State Robust Action
Robust Reward
HandMamipulatellig Boolean TouchSensons-vi
HandReach-v2
HandManipulaaeHlockk//
HandMamipulateFgg..
HandManipullteeenn.s
Table 7: A List of Examples for Robustness in Fetch Manipulation Tasks
Tasks Robust Type
Robust State Robust Action
Robust Reward
F
FetchReach-v3
FetchSlide-v3
FetchPickAnnPPace-v3
26

---

Published as a conference paper at ICLR 2025
Table 8: List of Examples for Robustness Multi-Agent Tasks
Tasks Robbust Type
Robust State
Robust Action
Robust Reward
MA-Ant-2x4, 2x4d, 4x2, 4x1
H4-ashiChho,,ah-233 6x1 MA-Hopper-3xl MA-Walker2d-2x3
MA-Swimmer-2x1
MA-Humanoid-99-55
MA Hu-FFuumanoidSSanduu-vv
MA-Pusher-3p
MA Reacherrr--
Many-Ma Sanny-MA-Siimnnn 10x2, 5x4, 6x1, 1x2
Many-Man MA-Ant-2x3, 3x1
CoupledHal/Cheetaa-ppp
Parameters
Value
Parameters
Value
buffer size
4096
hidden size
[64, 64]
3e-4
gamma
0.99
epoch
100
steps per epoch
30000
steps per collect
2048
repeat per collect
10
batch size
64
training num rew norm ent coef bound action clip
8
testing num
10
True
vf coef gae lambda
0.25
0.0
0.95
clip
Ir decay
True
max grad norm
0.5
eps clip
0.2
dual clip
None
value clip
norm adv
recompute adv
Table Parameter values used for PPO De Witt al. 2020) in experiments.
(Schulman et al.
2017), MAPPO Yu et al. 2022) and IPPO
Parameters
Value
Parameters
Value
buffer size
4096
hidden size
[64, 64]
actor Ir
le-3
critic Ir
le-3
gamma
0.99
tau
0.005
alpha
0.0.2
auto alpha
False
epoch
100
steps per epoch
30000
steps per collect
2048
update per step
start time step
10000
step training num
batch size
64
testing num
10
Table 10: Parameter values used for SAC (Haarnoja et al. 2018) in the experiment.
Parameters
Value
Parameters
Value
start steps
5000
num steps
300000
eval
True
eval episode
10
eval times
10
local reply size
1000
gamma Ir batch size
0.99
tau
0.005
3e-4
alpha
0.2
256
update per step
target update interval
hidden size
256
gail batch le-3
256
exponent
1.5
tomac alpha
reward max
Table 11: Parameter values used for OMPO (Luo et al. ments.
2024) in non-stationary MuJoCo experi-
27

---

Published as a conference paper at ICLR 2025
Parameters
Value
Parameters
Value
image obs
False
actor Ir
3e-4
critic Ir
le-3
gamma
0.99
tau auto alpha
5e-3
alpha
0.1
True
alpha Ir
3e-4
hidden size
[256,
256, 256]
n steps
buffer size
1e6
step per epoch
1e4
step per collect
20
batch size exploration noise camera width training num sigma augmented ratio
128
start time step
0
0
horizon
300
agentview
height encoder type
128
128
mlp
10
test num bound
10
0.01
0.01
0.5
vae sigma
control frequency
20
Table 12: Parameter values used for RSC (Ding et al. 2023a) in the causaldoor/causalliftt experi-
ments; for DBC (Zhang et al. (021a), based on above parameters, transition model type is proba- bilistic, encoder feature dim is 256, encoder 1e-4, decoder Ir is le-4, bisim coef is 0.5, log std min is 100, log std max is 2; for ATLA (Zhang al. 2021b), policy update max is 100, adv update max is 100, and adv eps is 0.01.
Parameters
Value
Parameters
Value
gamma
0.995
hidden layer dim
64
cost limit
0.04
slack bound
5e-3
exploration iteration
40
epoch
500
tau max kl batch size
0.97
12 reg
le-3
lc-2
damping
le-1
150000
gradient wr
0.4
gradient wc
0.6
Table 13: Parameter values used for PCRPO safety experiments.
(Gu al.
2024b) and CRPO
(Xu al. 2021 in the
28


Reviews:
[{"review_id": "RzbRK8xjTQ", "reviewer": "Reviewer_2wCe", "summary": "The paper introduces Robust-Gymnasium, a unified and modular benchmark designed for evaluating robust reinforcement learning (RL) algorithms. It addresses the lack of standardized benchmarks for robust RL by providing a platform that supports a wide variety of disruptions across key RL components, including agents' observed state and reward, agents' actions, and the environment. The benchmark includes over sixty diverse task environments spanning control, robotics, safe RL, and multi-agent RL. The paper also benchmarks existing standard and robust RL algorithms within this framework, revealing significant deficiencies in current algorithms and offering new insights. The code for Robust-Gymnasium is available online.", "questions": "- How does Robust-Gymnasium handle continuous action spaces and high-dimensional state spaces?\n- Can the benchmark be used to evaluate the robustness of RL algorithms in partially observable environments?\n- What are the limitations of the current implementation of Robust-Gymnasium, and how might these be addressed in future work?\n- How does the benchmark compare to other existing RL benchmarks in terms of robustness evaluation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Robust-Gymnasium offers a broad range of tasks for evaluating robust RL algorithms, covering various domains.\n- The benchmark is highly modular, allowing for flexible construction of diverse tasks and easy integration with existing environments.\n- It supports different types of disruptions, including random disturbances, adversarial attacks, internal dynamic shifts, and external disturbances.\n- The benchmark is designed to be user-friendly, with clear documentation and examples.", "weaknesses": "- The variety of disruptions and the modular nature might make the benchmark complex to understand and use for some users.\n- The effectiveness of some robust RL algorithms might rely on the quality and quantity of offline demonstration data.\n- The performance of algorithms on the benchmark could be sensitive to hyperparameter tuning, which might not be straightforward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V80qz0SmXW", "reviewer": "Reviewer_SF5o", "summary": "The work proposes a new benchmark for robust reinforcement learning termed Robust-Gymnasium. The manuscript introduces a framework for MDPs under disturbances and models its benchmark after it. There are three types of disturbances: observation, action and environment disruptions. The paper outlines 60 standard tasks that can be used in the benchmark with these disturbances and provides an experimental validation using baselines from standard, robust, safe, and multi-agent RL demonstrating the utility of the benchmark.", "questions": "Q1: In section 2.1, can you elaborate why maximization of the reward is over disturbed actions but not disturbed states?  \n\nQ2: L213 “Not all task bases support every type of disruption.” Could you elaborate why not? What is the limitation? This answer should likely be added to the text.  \n\nQ3: For Safety Gym, how do disturbances interact with the constraints?   \n\nQ4: I am confused about the adversarial disturbance mode. The text states “Any algorithm can be applied through this interface to adversarially attack the process.” L301. Does that mean that there are no standard disruptors implemented and the user has to implement them themselves?  \n\nQ5: Does the LLM for the adversarial disturbance mode require the user to run a local LLM?  \n\nQ6: Are there any tasks that you believe become significantly harder by introducing the perturbations, so much so that they might be unsolvable now?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Clarity  \na) The text uses clear language and is easy to follow.  \nb) Figure 1 is very  useful as it nicely summarizes environments, agents and disruptions and Figure 2 is a nice addition to describe the environment flow.  \n\n2. Problem Motivation  \na) I think the motivation for this problem is solid and we do need benchmarks that test real world robustness. Even if this benchmark is not perfect for that as it creates artificial disturbances, this might be the closest we can get with general solutions. I do think the benchmark solves a good problem the community is facing.  \n\n3. Novelty  \na) I am not aware of any benchmarks for robust RL that are very extensive lending credibility to the novelty of this benchmark.  \n\n4. Experiments  \na) While I am not familiar with some of the baselines, it seems that the evaluation is somewhat extensive. At least I believe it is sufficient to demonstrate that current algorithms fail on this benchmark which allows for new research to be done.  \nb) I do appreciate the two setting evaluations of training and testing. I think it is crucial to demonstrate what happens when training works fine but disturbances occur during testing. This experiment highlights the importance of this work.", "weaknesses": "1. Clarity   \na) Overall, several sections are very wordy and or redundant, repeating lots of information but missing useful information early on. Some examples:\n* Section 2.1 and 2.2 could be more concise, it feel like they are repeating the same thing multiple times when describing the disruptors. To remedy this it might be good to consolidate the functionality and highlight specific disruptors in section 2.2. For instance, it is not clear to me what random noise on an environment disruptor means. I also don’t quite understand what “The environment-disruptor uses this mode to alter the external conditions of the environment.” entails.\n* The same goes for sections 3.2 and 2.2. Both sections address the design of disruptors and essentially repeat a lot of information. It seems easy to simply combine these two sections which will also avoid confusion about how disruptors work.  I understand that there is supposed to be a differentiation between the universal framework and the implementation but nonetheless there would be lots of text that can be cut for clarity.   \nb) I find that section 3.2 is missing crucial information. The section can likely be improved by adding additional information about the state-action space and how the different disruptors affect them for each environment. The space for this can likely be obtained by condensing sections 2.1 and 2.2. If action spaces are similar, it might be possible to cluster environments and add information about the action spaces per cluster such as “these environments all use joint control with action spaces according to the number of degrees and an additional grasp action”.  \n\n2. Related Work   \na) In L 73, the text states “While numerous RL benchmarks exist, including a recent one focused on robustness to environment shifts (Zouitine et al., 2024), none are specifically designed for comprehensively evaluating robust RL algorithms.” I only skimmed the referenced work but it seems that the citation aims to do exactly that. However, they might have a less comprehensive benchmark. We can likely count them as X work but I believe a more thorough differentiation from this paper would benefit the presented manuscript.  \nb) I appreciate the additional section on robust benchmarks in Appendix A. In general for benchmark papers, I find it beneficial to demonstrate the novelty of the benchmark but providing citations to benchmarks that are related to demonstrate that there is a gap in the existing literature. Here is a non-exhaustive list of possibly relevant recent benchmarks that might be of use as a starting point [1-11]. There are older benchmarks too such as ALE and DM Control for which I recommend standard citations. Such a differentiation does obviously not have to happen in the main text.  \n\n3. Benchmark Feedback  \na) “Notably, in our benchmark, we implement and feature an algorithm leveraging LLM to determine the disturbance. In particular, the LLM is told of the task and uses the current state and reward signal as the input” L302 - It seems quite wasteful to have to run a full LLM at every environment step and it might be good to have simpler adversarial features that don’t limit usage to labs with lots of money for compute. The LLM feels a lot like using an LLM for the sake of the LLM. It is unclear to me why this choice was made rather than a simpler adversarial attacker.  \nb) What I am missing is metrics other than cost and reward that are useful to determine whether one is making progress on this benchmark. Given two algorithms with the same performance, what let’s us determine whether either of them is more robust? I think providing useful metrics of interest would be good to make this benchmark stand out. For instance, reliability metrics such as those in [12] might be useful to measure.  \nc) The second thing I am missing is guidelines on how to choose parameters for the disturbances. I think elaborating on what values are valid in section 3.2 as I mentioned before and providing suggestions would be useful for standardized usage of the benchmark. For instance, it is unclear in section 4.3, why the attacks follow a Gaussian distribution and not a Uniform distribution. Is this more realistic? Maybe it is arbitrary but then it should at least be stated earlier that this is recommended by the work.  \n\n4. Experiments  \na) It is unclear over how many seeds the experiments were conducted. Given the high variance in RL results in general [13], and the need for many experiments even without disturbances [14], we should conclude that more robust experimental evaluation is needed in Disturbed MDPs. For instance, 5 random seeds would definitely not be enough to draw meaningful conclusions from many of the provided graphs.  \nb) It is unclear to me how the tasks were picked and why the evaluations are not incorporating all tasks for all baselines. Running all tasks with all baselines would definitely strengthen the argument for the necessity of the benchmark and avoid uncertainty about how to choose tasks. At least, there should be one experiment that runs one algorithm on all tasks to verify that all tasks are in fact still learnable. I understand that that is computationally costly but I believe it is needed to verify the utility of the benchmark.  \n\nMinor suggestions  \n* In L156, L180, In Disrupted MDP -> In a Disrupted MDP\n* L192 and L197: for environment disruptor -> for the environment disruptor\n* L201 Disrupted-MDP allows disruptors to operate flexibly over time during the interaction process.\n\nOverall, I do think this work might constitute a good contribution. However, I think there need to be various adjustments for clarity. These are mostly things that require rewriting and not running any experiments. This includes consolidating text and providing insights into how to choose tasks, metrics and disturbance parameters. The latter is especially important if the benchmark ought to provide a standardized basis. If these changes are made I am willing to recommend acceptance. To make this a very strong publication, I think more extensive experiments to validate that all tasks are learnable are needed, and experiments would have to be run over a large number of trials to ensure statistical significance.\n\n[1] Continual World: A Robotic Benchmark For Continual Reinforcement Learning. Maciej Wolczyk, Michał Zając, Razvan Pascanu, Łukasz Kuciński, Piotr Miłoś. NeurIPS 2021.  \n[2] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, Peter Stone. NeurIPS 2023.  \n[3] Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill bench-mark with large-scale demonstrations. NeurIPS D&B 2024.  \n[4] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019.\n[5] Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wuthrich, Yoshua Bengio, Bernhard Schölkopf, and Stefan Bauer. CausalWorld: A robotic manipulation benchmark for causal structure and transfer learning. ICLR 2021.  \n[6] Jorge A. Mendez, Marcel Hussing, Meghna Gummadi, and Eric Eaton. CompoSuite: A compositional reinforcement learning benchmark. CoLLAs 2022.  \n[7] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakr ishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars, and robots. ICLR 2024.  \n[8] DACBench: A Benchmark Library for Dynamic Algorithm Configuration. Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, Marius Lindauer. ICJAI 2021.  \n[9] Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I. Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries P. Smit, Nathan Grinsztajn, Raphael Boige, Cemlyn N. Waters, Mohamed A. Mimouni, Ulrich A. Mbou Sob, Ruan de Kock, Siddarth Singh, Daniel Furelos Blanco, Victor Le, Arnu Pretorius, and Alexandre Laterre. Jumanji: a diverse suite of scalable reinforcement learning environments in jax, 2024.  \n[10] Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. NeuRIPS 2020.  \n[11] Zhaocong Yuan, Adam W. Hall, Siqi Zhou, Lukas Brunke, Melissa Greeff, Jacopo Panerati, and Angela P. Schoellig. Safe-control-gym: A unified benchmark suite for safe learning-based control and reinforcement learning in robotics. IEEE Robotics and Automation 2022.  \n[12] Measuring the Reliability of Reinforcement Learning Algorithms. Stephanie C.Y. Chan, Samuel Fishman, John Canny, Anoop Korattikara, Sergio Guadarrama. ICLR 2020.  \n[13] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep\nreinforcement learning that matters. AAAI 2018.  \n[14] How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer. 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ODqiVA1uIB", "reviewer": "Reviewer_RCMy", "summary": "The paper proposes a robust reinforcement learning benchmark, designed for facilitating fast and flexible constructions of tasks to evaluate robust RL.  This benchmark provides various robust RL tasks by adding various perturbations to standard tasks from multiple RL benchmarks.", "questions": ".", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The provided overview in Figure 1 is good. \n- Sixty robust RL tasks are offered in this benchmark.", "weaknesses": "This paper made an effort in transforming diverse RL tasks into robust RL tasks where environmental perturbations are considered. However, it might be of limited significance, since there are some existing benchmarks ([1], [2], [3], [4]) that allow to add disturbances to RL tasks to test the robustness of RL algorithms. Besides, it offers a limited technical contribution, as the main technical work is to add a wrapper to the existing RL benchmarks that implements disturbances.  Therefore, I recommend rejection.\n\nI have some other concerns about the current version.   \n- The author stated that this is the first unified benchmark specifically designed for robust RL in the introduction. It is a bit overstated, as RRLS focuses on the evaluations for robust RL and some other benchmarks allow for evaluating the robustness of RL algorithms.\n- In Section 3.2, the authors present several disruptors that are used in previous works. Providing citations to them is suggested. \n- The discussion about the limitation of the benchmark is missing. \n\n\n[1] https://github.com/utiasDSL/safe-control-gym  \n[2] RRLS: Robust Reinforcement Learning Suite  \n[3] Datasets and benchmarks for offline safe reinforcement learning  \n[4] Natural Environment Benchmarks for Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zKmpq6fZ0p", "reviewer": "Reviewer_AwG1", "summary": "The authors introduce a robust reinforcement learning benchmark that addresses multiple types of robustness. These include robustness concerning the transition kernel, observation noise, action noise, and reward noise. The framework considers both random noise and adversarially selected worst-case noise. To generalize robustness, the concept of a \"disrupted MDP\" is introduced. The environments proposed are diverse, primarily involving robotics and continuous control tasks, covering both single and multi-agent settings.\n\nAgents are evaluated on this benchmark across multiple tasks, using various baselines such as SAC and PPO for standard RL approaches. For Robust RL with a nominal transition kernel, baselines like RSC are used. The paper also includes evaluations for robust learning under dynamic shifts (OMPO), state adversarial attacks (ALTA), visual distractions (DBC), safe RL (PCRPO and CRPO), and multi-agent RL (IPPO).", "questions": "Remarks: \n- Emphasize the introduction of the \"disrupted MDP\" by bolding its first mention.\n- There is a minor formatting issue on line 132 with a space before \"environment-disruptor.\"\n- Providing examples in the appendix on how to modify external parameters like wind would enhance usability.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written\n- The benchmark is an important contribution to the robust reinforcement learning community, offering a unified framework that fills a significant gap. It is comprehensive, covering a broad spectrum of robustness types, making it a valuable tool for evaluating and designing Robust RL algorithms.", "weaknesses": "- M2TD3, a state-of-the-art baseline for robustness under model misspecification, is not cited. Its inclusion would strengthen the paper’s coverage of relevant baselines.\n- The explanation of adversarial disturbance via LLMs is interesting but could be more general. Instead of focusing on LLMs, the paper should emphasize the adversarial setup and consider an adversary such as two player Markov games with potential LLM integration as an example.\n- While the benchmark is nearly exhaustive, baselines like RARL and M2TD3 are missing. It is unclear how uncertainty sets can be built with the benchmark. Including examples in the appendix on constructing such sets, as proposed in the M2TD3 paper, would be beneficial.\n- The environments are primarily robotics-based, except for Gymnasium Box2D. Including use cases like autonomous driving or drone simulations would diversify the benchmark and offer more relevant challenges to the community, fostering the development of more general RRL algorithms.\n\nM2TD3 Reference:  \nTanabe, T., Sato, R., Fukuchi, K., Sakuma, J., & Akimoto, Y. (2022). Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification. *Advances in Neural Information Processing Systems*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)