Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning

Authors: Shangding Gu, Laixi Shi, Muning Wen, Ming Jin, Eric Mazumdar, Yuejie Chi, Adam Wierman, Costas Spanos
Primary area: datasets and benchmarks
Keywords: Robust reinforcement learning, benchmark, reinforcement learning, multi-agent reinforcement learning

Abstract:
Driven by inherent uncertainty and the sim-to-real gap, robust reinforcement learning (RL) seeks to improve resilience against the complexity and variability in agent-environment sequential interactions. Despite the existence of a large number of RL benchmarks, there is a lack of standardized benchmarks for robust RL. Current robust RL policies often focus on a specific type of uncertainty and are evaluated in distinct, one-off environments. In this work, we introduce Robust-Gymnasium, a unified modular benchmark designed for robust RL that supports a wide variety of disruptions across all key RL components—agents' observed state and reward, agents' actions, and the environment. Offering over sixty diverse task environments spanning control and robotics, safe RL, and multi-agent RL, it provides an open-source and user-friendly tool for the community to assess current methods and foster the development of robust RL algorithms. 
In addition, we benchmark existing standard and robust RL algorithms within this framework, uncovering significant deficiencies in each and offering new insights.

Decision: accept (poster)
Reviewer scores: 8, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1729265001636

thread_type:
official_review

<thread_object>
creator: reviewer_awg1

<content>
summary:
The authors introduce a robust reinforcement learning benchmark that addresses multiple types of robustness. These include robustness concerning the transition kernel, observation noise, action noise, and reward noise. The framework considers both random noise and adversarially selected worst-case noise. To generalize robustness, the concept of a "disrupted MDP" is introduced. The environments proposed are diverse, primarily involving robotics and continuous control tasks, covering both single and multi-agent settings.

Agents are evaluated on this benchmark across multiple tasks, using various baselines such as SAC and PPO for standard RL approaches. For Robust RL with a nominal transition kernel, baselines like RSC are used. The paper also includes evaluations for robust learning under dynamic shifts (OMPO), state adversarial attacks (ALTA), visual distractions (DBC), safe RL (PCRPO and CRPO), and multi-agent RL (IPPO).

strengths:
- The paper is well written
- The benchmark is an important contribution to the robust reinforcement learning community, offering a unified framework that fills a significant gap. It is comprehensive, covering a broad spectrum of robustness types, making it a valuable tool for evaluating and designing Robust RL algorithms.

weaknesses:
- M2TD3, a state-of-the-art baseline for robustness under model misspecification, is not cited. Its inclusion would strengthen the paper’s coverage of relevant baselines.
- The explanation of adversarial disturbance via LLMs is interesting but could be more general. Instead of focusing on LLMs, the paper should emphasize the adversarial setup and consider an adversary such as two player Markov games with potential LLM integration as an example.
- While the benchmark is nearly exhaustive, baselines like RARL and M2TD3 are missing. It is unclear how uncertainty sets can be built with the benchmark. Including examples in the appendix on constructing such sets, as proposed in the M2TD3 paper, would be beneficial.
- The environments are primarily robotics-based, except for Gymnasium Box2D. Including use cases like autonomous driving or drone simulations would diversify the benchmark and offer more relevant challenges to the community, fostering the development of more general RRL algorithms.

M2TD3 Reference:  
Tanabe, T., Sato, R., Fukuchi, K., Sakuma, J., & Akimoto, Y. (2022). Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification. *Advances in Neural Information Processing Systems*.

questions:
Remarks: 
- Emphasize the introduction of the "disrupted MDP" by bolding its first mention.
- There is a minor formatting issue on line 132 with a space before "environment-disruptor."
- Providing examples in the appendix on how to modify external parameters like wind would enhance usability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730611464713

thread_type:
official_review

<thread_object>
creator: reviewer_rcmy

<content>
summary:
The paper proposes a robust reinforcement learning benchmark, designed for facilitating fast and flexible constructions of tasks to evaluate robust RL.  This benchmark provides various robust RL tasks by adding various perturbations to standard tasks from multiple RL benchmarks.

strengths:
- The provided overview in Figure 1 is good. 
- Sixty robust RL tasks are offered in this benchmark.

weaknesses:
This paper made an effort in transforming diverse RL tasks into robust RL tasks where environmental perturbations are considered. However, it might be of limited significance, since there are some existing benchmarks ([1], [2], [3], [4]) that allow to add disturbances to RL tasks to test the robustness of RL algorithms. Besides, it offers a limited technical contribution, as the main technical work is to add a wrapper to the existing RL benchmarks that implements disturbances.  Therefore, I recommend rejection.

I have some other concerns about the current version.   
- The author stated that this is the first unified benchmark specifically designed for robust RL in the introduction. It is a bit overstated, as RRLS focuses on the evaluations for robust RL and some other benchmarks allow for evaluating the robustness of RL algorithms.
- In Section 3.2, the authors present several disruptors that are used in previous works. Providing citations to them is suggested. 
- The discussion about the limitation of the benchmark is missing. 


[1] https://github.com/utiasDSL/safe-control-gym  
[2] RRLS: Robust Reinforcement Learning Suite  
[3] Datasets and benchmarks for offline safe reinforcement learning  
[4] Natural Environment Benchmarks for Reinforcement Learning

questions:
.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649703541

thread_type:
official_review

<thread_object>
creator: reviewer_sf5o

<content>
summary:
The work proposes a new benchmark for robust reinforcement learning termed Robust-Gymnasium. The manuscript introduces a framework for MDPs under disturbances and models its benchmark after it. There are three types of disturbances: observation, action and environment disruptions. The paper outlines 60 standard tasks that can be used in the benchmark with these disturbances and provides an experimental validation using baselines from standard, robust, safe, and multi-agent RL demonstrating the utility of the benchmark.

strengths:
1. Clarity  
a) The text uses clear language and is easy to follow.  
b) Figure 1 is very  useful as it nicely summarizes environments, agents and disruptions and Figure 2 is a nice addition to describe the environment flow.  

2. Problem Motivation  
a) I think the motivation for this problem is solid and we do need benchmarks that test real world robustness. Even if this benchmark is not perfect for that as it creates artificial disturbances, this might be the closest we can get with general solutions. I do think the benchmark solves a good problem the community is facing.  

3. Novelty  
a) I am not aware of any benchmarks for robust RL that are very extensive lending credibility to the novelty of this benchmark.  

4. Experiments  
a) While I am not familiar with some of the baselines, it seems that the evaluation is somewhat extensive. At least I believe it is sufficient to demonstrate that current algorithms fail on this benchmark which allows for new research to be done.  
b) I do appreciate the two setting evaluations of training and testing. I think it is crucial to demonstrate what happens when training works fine but disturbances occur during testing. This experiment highlights the importance of this work.

weaknesses:
1. Clarity   
a) Overall, several sections are very wordy and or redundant, repeating lots of information but missing useful information early on. Some examples:
* Section 2.1 and 2.2 could be more concise, it feel like they are repeating the same thing multiple times when describing the disruptors. To remedy this it might be good to consolidate the functionality and highlight specific disruptors in section 2.2. For instance, it is not clear to me what random noise on an environment disruptor means. I also don’t quite understand what “The environment-disruptor uses this mode to alter the external conditions of the environment.” entails.
* The same goes for sections 3.2 and 2.2. Both sections address the design of disruptors and essentially repeat a lot of information. It seems easy to simply combine these two sections which will also avoid confusion about how disruptors work.  I understand that there is supposed to be a differentiation between the universal framework and the implementation but nonetheless there would be lots of text that can be cut for clarity.   
b) I find that section 3.2 is missing crucial information. The section can likely be improved by adding additional information about the state-action space and how the different disruptors affect them for each environment. The space for this can likely be obtained by condensing sections 2.1 and 2.2. If action spaces are similar, it might be possible to cluster environments and add information about the action spaces per cluster such as “these environments all use joint control with action spaces according to the number of degrees and an additional grasp action”.  

2. Related Work   
a) In L 73, the text states “While numerous RL benchmarks exist, including a recent one focused on robustness to environment shifts (Zouitine et al., 2024), none are specifically designed for comprehensively evaluating robust RL algorithms.” I only skimmed the referenced work but it seems that the citation aims to do exactly that. However, they might have a less comprehensive benchmark. We can likely count them as X work but I believe a more thorough differentiation from this paper would benefit the presented manuscript.  
b) I appreciate the additional section on robust benchmarks in Appendix A. In general for benchmark papers, I find it beneficial to demonstrate the novelty of the benchmark but providing citations to benchmarks that are related to demonstrate that there is a gap in the existing literature. Here is a non-exhaustive list of possibly relevant recent benchmarks that might be of use as a starting point [1-11]. There are older benchmarks too such as ALE and DM Control for which I recommend standard citations. Such a differentiation does obviously not have to happen in the main text.  

3. Benchmark Feedback  
a) “Notably, in our benchmark, we implement and feature an algorithm leveraging LLM to determine the disturbance. In particular, the LLM is told of the task and uses the current state and reward signal as the input” L302 - It seems quite wasteful to have to run a full LLM at every environment step and it might be good to have simpler adversarial features that don’t limit usage to labs with lots of money for compute. The LLM feels a lot like using an LLM for the sake of the LLM. It is unclear to me why this choice was made rather than a simpler adversarial attacker.  
b) What I am missing is metrics other than cost and reward that are useful to determine whether one is making progress on this benchmark. Given two algorithms with the same performance, what let’s us determine whether either of them is more robust? I think providing useful metrics of interest would be good to make this benchmark stand out. For instance, reliability metrics such as those in [12] might be useful to measure.  
c) The second thing I am missing is guidelines on how to choose parameters for the disturbances. I think elaborating on what values are valid in section 3.2 as I mentioned before and providing suggestions would be useful for standardized usage of the benchmark. For instance, it is unclear in section 4.3, why the attacks follow a Gaussian distribution and not a Uniform distribution. Is this more realistic? Maybe it is arbitrary but then it should at least be stated earlier that this is recommended by the work.  

4. Experiments  
a) It is unclear over how many seeds the experiments were conducted. Given the high variance in RL results in general [13], and the need for many experiments even without disturbances [14], we should conclude that more robust experimental evaluation is needed in Disturbed MDPs. For instance, 5 random seeds would definitely not be enough to draw meaningful conclusions from many of the provided graphs.  
b) It is unclear to me how the tasks were picked and why the evaluations are not incorporating all tasks for all baselines. Running all tasks with all baselines would definitely strengthen the argument for the necessity of the benchmark and avoid uncertainty about how to choose tasks. At least, there should be one experiment that runs one algorithm on all tasks to verify that all tasks are in fact still learnable. I understand that that is computationally costly but I believe it is needed to verify the utility of the benchmark.  

Minor suggestions  
* In L156, L180, In Disrupted MDP -> In a Disrupted MDP
* L192 and L197: for environment disruptor -> for the environment disruptor
* L201 Disrupted-MDP allows disruptors to operate flexibly over time during the interaction process.

Overall, I do think this work might constitute a good contribution. However, I think there need to be various adjustments for clarity. These are mostly things that require rewriting and not running any experiments. This includes consolidating text and providing insights into how to choose tasks, metrics and disturbance parameters. The latter is especially important if the benchmark ought to provide a standardized basis. If these changes are made I am willing to recommend acceptance. To make this a very strong publication, I think more extensive experiments to validate that all tasks are learnable are needed, and experiments would have to be run over a large number of trials to ensure statistical significance.

[1] Continual World: A Robotic Benchmark For Continual Reinforcement Learning. Maciej Wolczyk, Michał Zając, Razvan Pascanu, Łukasz Kuciński, Piotr Miłoś. NeurIPS 2021.  
[2] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, Peter Stone. NeurIPS 2023.  
[3] Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill bench-mark with large-scale demonstrations. NeurIPS D&B 2024.  
[4] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019.
[5] Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wuthrich, Yoshua Bengio, Bernhard Schölkopf, and Stefan Bauer. CausalWorld: A robotic manipulation benchmark for causal structure and transfer learning. ICLR 2021.  
[6] Jorge A. Mendez, Marcel Hussing, Meghna Gummadi, and Eric Eaton. CompoSuite: A compositional reinforcement learning benchmark. CoLLAs 2022.  
[7] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakr ishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars, and robots. ICLR 2024.  
[8] DACBench: A Benchmark Library for Dynamic Algorithm Configuration. Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, Marius Lindauer. ICJAI 2021.  
[9] Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I. Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries P. Smit, Nathan Grinsztajn, Raphael Boige, Cemlyn N. Waters, Mohamed A. Mimouni, Ulrich A. Mbou Sob, Ruan de Kock, Siddarth Singh, Daniel Furelos Blanco, Victor Le, Arnu Pretorius, and Alexandre Laterre. Jumanji: a diverse suite of scalable reinforcement learning environments in jax, 2024.  
[10] Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. NeuRIPS 2020.  
[11] Zhaocong Yuan, Adam W. Hall, Siqi Zhou, Lukas Brunke, Melissa Greeff, Jacopo Panerati, and Angela P. Schoellig. Safe-control-gym: A unified benchmark suite for safe learning-based control and reinforcement learning in robotics. IEEE Robotics and Automation 2022.  
[12] Measuring the Reliability of Reinforcement Learning Algorithms. Stephanie C.Y. Chan, Samuel Fishman, John Canny, Anoop Korattikara, Sergio Guadarrama. ICLR 2020.  
[13] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep
reinforcement learning that matters. AAAI 2018.  
[14] How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer. 2018.

questions:
Q1: In section 2.1, can you elaborate why maximization of the reward is over disturbed actions but not disturbed states?  

Q2: L213 “Not all task bases support every type of disruption.” Could you elaborate why not? What is the limitation? This answer should likely be added to the text.  

Q3: For Safety Gym, how do disturbances interact with the constraints?   

Q4: I am confused about the adversarial disturbance mode. The text states “Any algorithm can be applied through this interface to adversarially attack the process.” L301. Does that mean that there are no standard disruptors implemented and the user has to implement them themselves?  

Q5: Does the LLM for the adversarial disturbance mode require the user to run a local LLM?  

Q6: Are there any tasks that you believe become significantly harder by introducing the perturbations, so much so that they might be unsolvable now?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685320576

thread_type:
official_review

<thread_object>
creator: reviewer_2wce

<content>
summary:
The paper introduces Robust-Gymnasium, a unified and modular benchmark designed for evaluating robust reinforcement learning (RL) algorithms. It addresses the lack of standardized benchmarks for robust RL by providing a platform that supports a wide variety of disruptions across key RL components, including agents' observed state and reward, agents' actions, and the environment. The benchmark includes over sixty diverse task environments spanning control, robotics, safe RL, and multi-agent RL. The paper also benchmarks existing standard and robust RL algorithms within this framework, revealing significant deficiencies in current algorithms and offering new insights. The code for Robust-Gymnasium is available online.

strengths:
- Robust-Gymnasium offers a broad range of tasks for evaluating robust RL algorithms, covering various domains.
- The benchmark is highly modular, allowing for flexible construction of diverse tasks and easy integration with existing environments.
- It supports different types of disruptions, including random disturbances, adversarial attacks, internal dynamic shifts, and external disturbances.
- The benchmark is designed to be user-friendly, with clear documentation and examples.

weaknesses:
- The variety of disruptions and the modular nature might make the benchmark complex to understand and use for some users.
- The effectiveness of some robust RL algorithms might rely on the quality and quantity of offline demonstration data.
- The performance of algorithms on the benchmark could be sensitive to hyperparameter tuning, which might not be straightforward.

questions:
- How does Robust-Gymnasium handle continuous action spaces and high-dimensional state spaces?
- Can the benchmark be used to evaluate the robustness of RL algorithms in partially observable environments?
- What are the limitations of the current implementation of Robust-Gymnasium, and how might these be addressed in future work?
- How does the benchmark compare to other existing RL benchmarks in terms of robustness evaluation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732095237285

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# General response
We thank the reviewers for their careful reading of the paper and their insightful and valuable feedback. Below, we provide a summary of the main changes and highlight our contributions compared with prior art.


## **Clarification: Public comment was on the related work RRLS, not this work**
We want to clarify that the public comments were actually a response to "reviewer RCMy" **about prior work RRLS**, but not this work.

## **A new tutorial:**
We make **new tutorials:** https://robust-gymnasium-rl.readthedocs.io to ensure that Robust Gymnasium with the variety of disruptions and the modular nature, leads to flexible and friendly usability rather than hindering it.

## **Comparisons to existing works with robust evaluations:**

A brief answer for the differentiation: This work (Robust Gymnasium) fills the gaps for comprehensive robust evaluation of RL: 1) support **a large number of tasks (over 60)** for robust evaluation; 2) **support various disruption types that hinder robustness** --- support potential uncertainty over various stages of the agent-environment interaction, over the observed state, observed reward, action, and the environment (transition kernel).
* **Comparison to RL benchmarks designed for robustness evaluations.** To the best of our knowledge, [RRLS](https://github.com/SuReLI/RRLS) is the only existing benchmark designed specifically for robustness evaluations. We illustrate the comparisons in the following table.
* **Comparisons to other works involving robust evaluation RL tasks.**
    We added **a new section in the related works (Appendix A)** for the differentiation: RL works involving tasks for robust evaluation. Despite recent advancements, prior works involving robust evaluations of RL typically support a few robust evaluation tasks associated with only one disruption type. 

    Specifically, there exists a lot of benchmarks for different RL problems, such as benchmarks for standard RL [1,5], safe RL, multi-agent RL, offline RL, etc. These benchmarks either don't have robust evaluation tasks, or only have a narrow range of tasks for robust evaluation (since robust evaluation is not their primary goal), such as [1] support 5 tasks with robust evaluations in control. Besides, there are many existing robust RL works that involve tasks for robust evaluations, while they often evaluate only a few tasks in specific domains, such as 8 tasks for robotics and control [6], 9 robust RL tasks in StateAdvRL [2], 5 robust RL tasks in RARL [3], a 3D bin-packing task [7],  etc. Since their primary goal is to design robust RL algorithms, but not a platform to evaluate the algorithms.


| Robust RL Platform  | Task Coverage | Disruption Type | Disruption Mode  | Benchmark Feature  |
| :---------: |  :---------: | :---------: | :---------: | :---------: |
| [Robust Gymnasium](https://robust-rl.online/) (ours)   | **over 60 tasks** (&#9312; single agent RL, &#9313; Multi-agent, &#9314; safe RL)   | &#9312; Observation (state+reward); &#9313; Action; &#9314; Environments; | &#9312; Random; &#9313; Adversarial disturbance; &#9314; Internal disturbance; &#9315; External disturbance  | &#9312; High Modularity; &#9313; High Compatibility; &#9314; Vectorized Environments;|
| [RRLS](https://github.com/SuReLI/RRLS)           | **6 tasks** (&#9312; Single agent RL)     |       &#9314; Environments        | &#9312; Random disturbance  | / |
||

> [1] Duan, Yan, et al. "Benchmarking deep reinforcement learning for continuous control." International conference on machine learning. PMLR, 2016.  
> [2] Zhang, Huan, et al. "Robust deep reinforcement learning against adversarial perturbations on state observations." Advances in Neural Information Processing Systems 33 (2020): 21024-21037.  
> [3] Pinto, Lerrel, et al. "Robust adversarial reinforcement learning." International conference on machine learning. PMLR, 2017.  
> [4] Zouitine, A., Bertoin, D., Clavier, P., Geist, M., & Rachelson, E. (2024). RRLS: Robust Reinforcement Learning Suite. arXiv preprint arXiv:2406.08406.  
> [5] Towers, Mark, et al. "Gymnasium: A standard interface for reinforcement learning environments." arXiv preprint arXiv:2407.17032 (2024).  
> [6] Ding, Wenhao, et al. "Seeing is not believing: Robust reinforcement learning against spurious correlation." Advances in Neural Information Processing Systems 36 (2024).  
> [7] Pan, Yuxin, Yize Chen, and Fangzhen Lin. "Adjustable robust reinforcement learning for online 3d bin packing." Advances in Neural Information Processing Systems 36 (2023): 51926-51954.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission introduces a benchmark, Robust Gymnasium, for evaluating RL methods' robustness to variations in an environment, a major factor that determines the usefulness of an RL approach. A major part of this work's contribution is a wrapper for "diversifying" existing RL environments.

The work's strengths are:

- Comprehensiveness of the benchmark. Robust Gymnasium includes 60 tasks, a wrapper to onboard more, and multiple sources of environment variation.
- Comprehensiveness of the evaluation. The paper presents an extensive empirical study of existing RL algorithms on Robust Gymnasium.
- The overall clarity of the documentation.
- Potential for impact. This work provides a tool for evaluating an important aspect of RL algorithms, for which few comparably comprehensive tools are available.


The weaknesses are:

- It's not clear how to determine whether a given variation of an environment is solvable.
- The lack of clarity around the standard use of this benchmark, as summarized in this post by reviewer SF5o: https://openreview.net/forum?id=2uQBSa2X4R&noteId=R0pN5LYxsE 

The metareviewer recommends this paper for acceptance. The benchmark it introduces is likely to be useful to the community even in its current state, despite its shortcomings. This is not to say that the shortcomings are negligible: the metareviewer agrees with reviewer SF5o's comments about the need for clearer standard use guidelines. While this issue is not a show-stopper, addressing it can significantly increase the benchmark's impact.