Memory Gym: Partially Observable Challenges to Memory-Based Agents
Authors: Marco Pleines
Matthias Pallasch
Frank Zimmer
Mike Preuss
Venue: ICLR 2023 poster (2023)
Keywords: Deep Reinforcement Learning
Memory
Benchmark
Proximal Policy Optimization
Gated Recurrent Unit
HELM

Abstract: Memory Gym is a novel benchmark for challenging Deep Reinforcement Learning agents to memorize events across long sequences, be robust to noise, and generalize. It consists of the partially observable 2D and discrete control environments Mortar Mayhem, Mystery Path, and Searing Spotlights. These environments are believed to be unsolvable by memory-less agents because they feature strong dependencies on memory and frequent agent-memory interactions. Empirical results based on Proximal Policy Optimization (PPO) and Gated Recurrent Unit (GRU) underline the strong memory dependency of the contributed environments. The hardness of these environments can be smoothly scaled, while different levels of difficulty (some of them unsolved yet) emerge for Mortar Mayhem and Mystery Path. Surprisingly, Searing Spotlights poses a tremendous challenge to GRU-PPO, which remains an open puzzle. Even though the
randomly moving spotlights reveal parts of the environment’s ground truth, environmental ablations hint that these pose a severe perturbation to agents that leverage recurrent model architectures as their memory. 
Source Code: https://github.com/MarcoMeter/drl-memory-gym/

Published as a conference paper at ICLR 2023
MEMORY GYM: PARTIALLY OBSERVABLE CHALLENGES TO MEMORY-BASED AGENTS
Marco Pleines'
Matthias Pallasch'
Frank Zimmer?
Mike Preuss
'TU Dortmund University Shine-WWal University of Applied Sciences
'LICSS Universiteit Leiden
marco. parco.plelle wwwpleneettuddrreetteee
ABSTRACT
Memory Gym is a novel benchmark for challenging Deep Reinforcement Learn- ing agents to memorize events across long sequences, be robust noise, and generalize. It consists of the partially observable 2D and discrete control envi- ronments Mortar Mayhem, Mystery Path, and Searing Spotlights. These environ- ments are believed to be unsolvable by memory-less agents because they feature strong dependencies on memory and frequent agent-memory interactions. Empir- ical results based on Proximal Policy Optimization (PPO) and Gated Recurrent Unit (GRU) underline the strong memory dependency of the contributed envi- ronments. The hardness of these environments can be smoothly scaled, while different levels of difficulty (some of them unsolved yet) emerge for Mortar Mayhem and Mystery Path. Surprisingly, Searing Spotlights poses tremen- dous challenge to GRU-PPO, which remains an open puzzle. Even though the randomly moving spotlights reveal parts of the environment's ground truth, vironmental ablations hint that these pose a severe perturbation to agents that leverage recurrent model architectures as their memory. Source Code: https: memory gym/
// wtw.tttb..co///aaottttee///////e
INTRODUCTION
Memory vital mechanism of intelligent living beings to make favorable decisions sequentially under imperfect information and uncertainty. One's immediate sensory perception may not suffice information from past events cannot recalled. Reasoning, imagination, planning, and learning are skills that may become unattainable. When developing autonomously learning decision-making agents, the agent's memory mechanism is required to maintain representation of former obser- vations to ground its next decision. Adding memory mechanisms a recurrent neural network (Werbos) 1990) or a transformer (Vaswani et al. 2017) led to successfully learned policies both virtual and real-world tasks. For instance, Deep Reinforeement Learning (DRL) methods master complex video games such as StarCraft II (Vinyals et al., 2019). and DotA (Berner et al.] (2019). Examples of successes in real-world problems are dexterous in-hand manipulation (Andrychow- icz et al. 2020) and controlling tokamak plasmas (Degrave et al. (002).. In addition to leveraging memory, these tasks require vast amounts of computation resources and additional methods (e.g. domain randomization, incorporating domain knowledge, ect.), which make them undesirable for solely benchmarking an agent's ability to interact with its memory meaningfully.
We propose Memory Gym as a novel and open source benchmark consisting of three unique envi- ronments: Mortar Mayhem, Mystery Path, and Searing Spotlights. These environments challenge memory-based agents to memorize events across long sequences, generalize, be robust to noise, and be sample efficient. By accomplishing the desiderata that we define in this work, we believe that Memory Gym has the potential to complement existing benchmarks and therefore accelerate the de- velopment of DRL agents leveraging memory. All three environments feature visual observations, discrete action spaces, and are notably not solvable without memory. This allows users to assure early on whether their developed memory mechanism is working or not. To fit the problem of se- quential decision-maker by Memory Gym. Several related environments ask the agent only to memorize initial cues, which require infrequent agent-memory interactions. Our environments smoothly configurable. This
agents have to frequently leverage their memory to solve the posed tasks

---

Published as a conference paper at ICLR 2023
is useful to adapt the environment's difficulty the available resources, while easier difficulties can be used as a proof of concept. Competent methods can show off themselves in new challenges or identify their limits in profound way. All environments agent's is ability to generalize to unseen levels (or seeds). Due to the aforementioned configurable dif- ficulties, the trained agent be evaluated on out-of-disributiion levels. Memory Gym's significant dependencies are gym (Brockman et al. 2016) and PyGame" This allows Memory Gym to be eas- ily set up and executed on Linux, macOS, and Windows. Multiple thousands of agent-environment interactions per second are simulated by all environments.
procedurally generated to evaluate the
This paper proceeds as follows. We first define the memory benchmarks' criteria and portray the related benchmarks' landscape. Next, Mortar Mayhem, Mystery Path, and Searing Spotlights are detailed. Afterward, we show that memory is crucial in our environments by conducting empirical experiments using recurrent implementation (GRU-PPO) of Proximal Policy Optimization (PPO) (Schulman et al. 2017) and HELM (Paischer et al.) 2022). When scaling the hardness in Mortar Mayhem and Mystery Path, full range of difficulty levels emerge. Searing Spotlights remains unsolved because the recurrent agent is volatile to perturbations of the environment's core mechanic: the randomly wandering spotlights. This observation is also apparent when training single Procgen BossFight (Cobbe et al., (020)) level under the same spotlight perturbations as in Searing Spotlights. At last, this work concludes and enumerates future work.
2
COMPARISON OF RELATED MEMORY BENCHMARKS
2.1 DESIDERATA OF MEMORY BENCIMMRRSS
Before detailing related benchmarks, we define the aforementioned desiderata that we believe are essential memory benchmarks and benchmarks general.
Accessibility refers to the competence to easily set up and execute the environment. Benchmarks shall be publicly available, acceptably documented, and open source while running on the commonly used operating systems Linux, macOS, and Windows. Linux, in general, is important because many high-performance computing (HPC) facilities employ this operating system. As HPC facilities might not support virtualization, benchmarks should not be solely deployed as docker image similar. At last, benchmarks shall run headless because otherwise, these potentially rely on dependencies like xvfb EGL, which HPC facilities may not support well. Suppose relevant benchmark details, such as environment dynamics, are missing. In that case, it can be desirable to support humanly playable environments so that these can be explored in advance.
Fast simulation speeds, which achieve thousands of steps per second (i.e. FPS), allow training runs to be more wall time efficient, enabling to upscale experiments and their repetitions or faster development iterations. The benchmark's speed also depends on the time takes to reset the envi- ronment to set up a new episode for the agent to play. Towards maxing out FPS on a single machine, benchmarks shall be able to run multiple instances of their environments concurrently.
High diversity attributes environments that offer a large distribution of procedurally generated levels to reasonably challenge an agent's 's ability to generalize (Cobbe et al., 2020). Also, it is desirable to implement smoothly configurable environments to evaluate the agent at out-of-distribution levels.
Scalable difficulty is a property that shall make environments controllable such that their current hardness can be increased or decreased. Easier environments can have benefits: a proof-of-concep state is sooner reachable while developing novel methods, and research groups can fit the difficulty to their available resources. Moreover, increasing the difficulty ensures that already competent methods may prove themselves in new challenges to demonstrate their abilities or limits.
Strong dependency on memory refers to tasks that are only solvable if the agent can recall past information (i.e. successfully leveraging its memory). Section 2.4 describes partially observable environments that can be solved to some extent without memory. While memory-based agents might more efficiently solve these tasks, these do not guarantee that the agent's memory is working, To ensure that the utilized memory mechanism is working and does not suffer from bugs, this criterion cannot be omitted by benchmarks targeting specifically the agent's memory.
https:// http///wwwppp....e/////////// org
2

---

Published as a conference paper at ICLR 2023
Table I: An overview of the simulation speed and meta desiderata of the considered environments. The meta desiderata cover the criteria that are applicable to any benchmark in DRL. The mean FPS are measured across 100 episodes using constant actions. Procgen is averaged across its memory distribution environments, and so is Memory Gym. The other benchmarks are measured using single environment. refers true and O false. Appendix providees further details.
Benchmark
Mean HPS
Mcta Desiderata
Publicly Avaslable
High Diversity Scalable Difficulty
Open Source
Concurrent
Windows
Headless
Playable
Linus
macOS
Docker
Memory Gym (ours)
10123
Procgen Memory Distribution (Cobbe et al.
2020
18530
DM Ballet I1 ampinen et al. 2021
6631
MiniGrid Memory (Chevalier-baee Hoisver et al.
2018
5185
VizDoom (Wydmuch et al. 2018
549
DM Memory Task Suite (Fortunato et al.
2019
442
DM Lab 30 (Beattic al., 2016
433
DM
(Parisotio (Parsoooa al. 2020
DM Memory Maze (Parisotto al. 2020
DM Object Permanence (Lampinen al. 2021
Strong dependency on frequent memory interactions is a property of tasks that forces the agent to recall information from and add information to its memory frequently. We believe this is more problems because some related environments can be turned
suitable for sequential decision-maker into supervised learning problems (Section 2.5) and therefore only assess the memory's capacity and potentially its robustness to noise.
2.2 CONSIDERED ENVIRONMENTS
A diverse set of environments were used in the past to challenge memory-based agents. Some of them are originally fully observable but are turned into partially observable Markov Decision Pro- cesses (POMDP) by adding noise masking out information from the agent's 's observation space. For instance, this was done for the Arcade Learning Environment (Bellemare et al., 2013) by using flickering frames (Hausknecht & Stone, 2015) and common control tasks by removing the velocity from the agent's observation (Heess et al. al,] 2015) Meng et al. [2021] Shang et al. 2021). These envi- ronments do not require the agent memorize long sequences and can already be approached using frame stacking. Control tasks also touch on the context of Meta Reinforcement Learning (Meta RL), where memory mechanisms are prominent (Wang et al. 2021; Melo 2022, Ni et al. 2022). The same applies to Multi-Agent Reinforcement Learning (Berner et al. 2019; Baker et al. 2020 Vinyals et al. 2019). As we solely focus on benchmarking the agent's memory and its ability gen- eralize, we do not compare Memory Gym to environments of more complex contexts such as DM Alchemy (Wang al. 202T), Crafter (Hafner (02T)) or Obstacle Tower TowerJuliani Juliani al. (2019). Those might need additional components to the agent's 's architecture and its training paradigm. Within this section, we consider DRL benchmarks (Table [T that were used by recently contributed memory approaches MRA (Fortunato al.] 2019). GTrXL (Parisotto et al., 2020), HCAM (Lampinen al. 2222), HELM (Paischer et al., 2022), ) and A2C-Transformer (Sopov & Makarov 2022). By exam- ining these works, it becomes apparent that all of them different environments to evaluate their methods, making their results harder to compare.
2.3 META DESIDERATA
Table I provides information on the meta desiderata of the considered environments, Some environ- ments are inaccessible to some extent, as some cannot be run headless. DM Memory Maze (Parisotto et al. 2020). DM Numpad (Humplik et al. 2019), and DM Object Permanence (Lampinen al.) 2021 are not publicly available and can, therefore, not be used to reproduce and compare results in adjacent research. DM Memory Task Suite (Fortunato al. 2019) is not open source and can
3

---

Published as a conference paper at ICLR 2023
only be accessed via docker. It may be for these reasons that DM Memory Task Suite, Object Per- manence, Memory Maze, and Numpad are not widely used yet. DM Lab 30 (Beattie et al., 2016) has more uses, but the available environments, especially those used in the context of memory, superficially documented. [Parisotto et al. 2020 divide these environments into reactive and mem- ory tasks without further reasoning. Potential users have to explore these environments by playing them, which is possible on machines running Linux only, Procgen (Cobbe et al. 2020). DM Ballet Seevvalie--Boiveee et al.) 2018), and Memory Gym (ours) are the only environments that achieve throughput of thousands of steps per second. 3D environments usually achieve a smaller throughput, like VizDoom (Wydmuch et al. 2018) or DM Memory Task Suite (Fortunato et al. 2019).
(Lampinen et al., 2021), MiniGrid
2.4 STRONG DEPENDENCY ON MEMORY
As stated before, if an environment is solvable to some extent using an agent without memory, not easy to differentiate whether the memory mechanism working. This impression can be retrieved by examining the results of (Sopov& & Makarov 2022 the VizDoom environments where a policy without memory makes progress or achieves comparable performances. The six environments of Procgen's 's memory distribution leave with controversial thoughts. Paischer et al. 2022] show results in these environments that indicate that a memory-less approach can perform similarly to memory one. We further take closer look Miner. The agent can momentarily perceive some cues related to past steps of the episode: its last action is retrievable from its rotation. At the same time, the density of vanished tiles shows the agent whether this region was explored or not. Exploiting these cues potentially helps memory-less agents to find successful strategies and make us believe that this environment does not strongly depend on memory, while doubtlessly. a memory-based agent should be more efficient concerning the number of steps needed to solve the entire task. Another problem is that the mean cumulative reward usually reported, which averages the rewards achieved by collecting diamonds and using the exit. This data does not precisely tell how good an agent is at completing the entire task. Especially recalling the exit's position seems crucial to succeeding as soon possible. Reporting the success rate and the episode length should provide more meaningful insights. The environments Heist and Maze encompass fully observable levels, which raises concerns on strong memory dependence.
2.5 STRONG DEPENDENCY ON FREQUENT MEMORY INTERACTIONS
Concerning frequent memory interactions, we believe that MiniGrid Memory (Chevalier-Be Boisverr et al. 2018), Spot the Difference (Fortunato et al., 2019). and DM Ballet (Lampinen et al. [202]) are not well suited the need for memory in sequential decision making. These environments demand the agent's memory to solely memorize the to-be-observed goal cues during the very beginning of the episode. Once this cue is memorized, there is no need to manipulate the agent's memory further. Simply maintaining it is enough to solve the task. It can be hypothesized that the extracted features from observing the goal cues are sufficient to solve the ballet environment. To show this, the ballet environment can be made fully observable (markovization) by feeding the entire sequence of cues to a recurrent encoder that extracts features that are utilized by the agent's policy. Consequently, the policy does not need to maintain its memory in this case. The task at hand gets much easier because the agent does not need to make obsolete decisions while observing the sequence of cues as its position is frozen. These tasks would, therefore, only challenge the capacity and robustness to noise of the agent's memory. We believe this can be done more efficiently by other benchmarks that do not belong to the context of DRL, like Long Range Arena (Tay al. 2021).
MEMORY GYM ENVIRONMENTS
Mini-games of the commercial video game Pummel Party inspired Memory Gym's environments. The agent perceives all environments using 84 x 84 RGB pixels, while its action space is multi- discrete, featuring two dimensions of size three as shown in appendix C.I One dimension allows the agent to move horizontally (no movement, move left, move right), and the other concems the agent's 's vertical locomotion (no movement, move up, move down). Mortar Mayhem and Mystery
httt:://rebullttgees..comm
4

---

Published as a conference paper at ICLR 2023
Agent
Next Target
Command
(a) Annotated Ground Truth
(h) Agent Observation
(c) Visual Feedback
Figure 1: Mortar Mayhem's relevant entities are presented by the annotated ground truth of the environment (a). The green circle, which solely part of the ground truth, indicates the very next tile to move to. Once the agent has solved this command, the green circle moves to the next target tile. At the start of an episode, the commands are rendered sequentially onto the agent's observation (b) while the agent cannot move. Once all commands are shown, the agent has move to the target tile certain amount of time. As soon as the time for executing the current command over, the agent's s success is verified, After a delay of few steps, the agent can approach the following command if the episode did not terminate yet due to failure completing the entire command sequence.
in figure (c). This visual feedback is perceivable by the agent.
Path also feature variants based on grid-like locomotion (Figures I0bb to Io(d).. This way, the action space is discrete and allows the agent to move at all, rotate left, rotate right, move forward. Subsequently, we further detail the environments' dynamics, their peculiarities towards memory, and how these can be smoothly scaled support varying difficulty levels. Appendix C.2 quantifies the episode lengths. All parameters used for scaling the environments' hardness and their default values are found in appendix Appendices C.4 to C.6 visualize played episodes.
3.1 MORTAR MAYHEM
Mortar Mayhem (MM) (Figure takes place inside grid-like arena and consists of two tasks that depend on memory. At first, while unable to move, the agent has to memorize sequence of five commands (Clue Task), and afterward, it has to execute each command in the observed order (Act Task). One command orders the agent to move to one adjacent floor tile or to stay at the current one. If the agent fails, the episode terminates while receiving no reward. Upon successfully executing one command, the agent receives a reward of +0.1. Episode lengths in MM are dependent on the agent's current ability. The better the agent, the longer the episode lasts until an upper bound (i.e. max episode length) is reached. MM can be reduced to provide only the Act Task (MMAct). In this case, the command sequence fully observable as a one-hot encoded feature vector. The Act Task requires an agent to leverage its memory frequently because otherwise, the agent does not know which commands were already executed to fulfill the next one, while there are nine tiles maximum to consider. To solve this problem, the agent could learn to track time (e.g. count steps) where short-term memory should suffice. The hardness MM can be further simplified by equipping the agent with grid-like movement (MMGrid. MMActGrid). The agent is capable of moving tile at a time. To ensure a meaningful task, the agent must execute ten commands, not five. Further examples to raise MM's difficulty are extend sample the number of commands or the delay between command execution. compared to DM Ballet, MM's Act Task requires many correct actions in sequence, while DM Ballet asks the agent only identify the requested dancer.
3.2 MYSTERY PATH
Mystery Path (MP) (Figure challenges the agent to traverse an invisible path in grid-like level of dimension x 7, while only the path's origin is visible to the agent. If the agent moves off the path (i.e. falls down the pit), the agent is relocated to its origin. The episode terminates if the agent reaches the goal or runs out of time (512 steps). Upon reaching the goal, the environment signals reward of +1. To overcome uncertainty in this environment, the agent has to memorize several locations: its steps on the path and the locations where it fell off. The invisible path is procedurally generated using the path-finding algorithm A (Hart et al. 1968). At first, the path's
5

---

Published as a conference paper at ICLR 2023
Goal
Path
A Wall
Pit
Agent
Origin
(a) Annotated Ground Truth
(b) Agent Observation
(c) Visual Feedback
Figure 2:
Mystery Path's relevant entities are described by the environment's annotated ground
truth (a). The walkable parts of the path are colored blue (origin), green (goal), and white (path). Every other part of the environment is considered pit (black and red) where the agent falls off and then relocated the path's 's origin during its next step. The initially randomly chosen red tiles (A* Walls) not considered during path generation. The agent observes itself and the path s origin (b). the agent is off the path, a red cross provides visual feedback (c), which also observed.
Health Bar
Last Action
Exit
Agent
Spotlight
Coin
Annotated Ground Truth
Agent Observation
(c) Visual Feedback
Figure All relevant entities can be identified in the annotated ground truth of Searing Spot- lights (a). The top rows of pixels feature the agent's remaining health points and its last action that two chunks and three colors encode. Yellow circles correspond coins, while the somewhat rounded gray shape resembles the exit. If the exit is open (i.e. no coins are left), it turns green. The floor of the environment
chessboard colored blue and white. As seen in the agent's observa- tion (b), the spotlights hide or reveal the other entities. As additional visual feedback (c), the blue floor tiles turn red a spotlight spots the agent.
origin is sampled from the grid's border. Then, the goal is placed in random position the opposing side. Usually, A seeks to find the shortest path. A cost function is sampled to ensure that a wide variety of paths is procedurally generated. The path generation randomizes some cells to untraceable (A* walls). Note that the walls are considered pits and not physical obstacles. MP can also be simplified to grid variant (MPGrid), featuring the single discrete action space of four actions, while the agent's 's time limit is reduced from 512 steps to 128. In contrast to MM, the episode terminates sooner as the agent improves.
3.3 SEARING SPOTLIGHTS
Searing Spotlights (Figure 3 is perceived as pitch-black surrounding by the agent where the only information is unveiled by roaming and threatening spotlights. While initially starting with limited number of health points (e.g. 100), the agent loses one health point per step if hit by spotlight. The episode terminates if the agent has no remaining health points runs out of time (512 steps). Because of the posed threats, the agent has to hide in the dark. To successfully run away from closing in spotlights, the agent must memorize its past actions and at least one past position of itself to infer its current location. That also requires the agent to carefully use its health point budget to figure out its location ideally once briefly. To avoid numb policies, two additional tasks are added to

---

Published as a conference paper at ICLR 2023
IQM Commands Executed
10
1.0
1.0
GRU-FPO (eeest
MM Success Rate
0.8
J0M Success Rate
0.8
prahing
MD Inced scats)
traiiing seeder
0.6
0.6
MEMM frood scudsl
4bckkk (traiinng seadxt
0.4
4.ackk Trovel medal
2
0.2
Sandom
0.0
0.0
0
50
100
150
50
100
150
0
50
100
150
Steps (in millions)
Steps (in millions)
Steps (in millions)
(a) Mortar Mayhem Act Grid
(b) Mystery Path Grid
(c) Searing Spotlights
Figure 4: Training and generalization performances show strong dependence on memory.
the enviromment that requires the agent to traverse the environment. The first one randomly places an exit in the environment, which the agent can use to receive a reward of +1 and terminate the episode. Secondly, a coin collection task is added. Before using the environment's exit, the agent has to collect a certain number of randomly positioned coins. Collecting one coin grants a reward of +0.25 to the agent. If the agent successfully uses its memory, it can infer its current location, and recall the locations of the exit and the coins. A simplified grid version of Searing Spotlights is not available. As default simplification, the episode starts with perfect information, and after few steps, the global light is dimmed until off. Just like in MP, successful episodes terminate sooner. Appendix enumerates multiple parameters that organize the measures, behavior, and spawning frequency of the spotlights. Other parameters concern the reward function, the agent's properties, and the visual representation of Scaring Spotlights.
BASELINE EXPERIMENTS
We run empirical baseline experiments using the DRL algorithm Proximal Policy Optimiza- tion (PPO) (Schulman et al. 2017). To support memory, our implementation adds gated recurrent unit (GRU) (Coo et al. 2014) to the actor-critic model architecture of PPO (GRU-PPO). Related memory-based approaches such MRA (Fortunato et al., 2019), GTrXL (Parisotto et al. 2020). HCAM (Lampinen et al. 2021). and ALD (Parisotto & Salakhutdinov 2021) are not considered, because there is applicable code available (see appendix G). while these methods are expensive to reproduce. That does not account for HELM (Paischer et al. (022), which we successfully added to our training framework. However, HELM has poor wall-time efficiency (see figs. 17 and T8 in appendix H}. Due to the vast transformer component consisting of 18 blocks, the policy takes much more time to produce actions during inference. One GRU-PPO training run takes about ten hours in MMActGrid with a sequence length of 79, while HELM needs at least six times longer. This effect worsens for longer sequences. The last two baselines consider frame stacking. One stacks 4 RGB frames (4-Stack), while the other one stacks 16 grayscale frames (16-Stack). We repeat all experiments times. All training runs utilize 100,000 environment seeds. Generalization is assessed on 30 novel seeds, which are repeated 5 times. Hence, each data point aggregates 750 episodes. The subsequent sample efficiency curves show the interquartile mean (IQM) and a confidence interval of report
0.95 as recommended by Agarwal et al. (202I). For Mystery Path and Searing Spollights, the success rate, which indicates whether the agent succeeded at the entire task or not. Results on Mortar Mayhem show the number of commands that were properly executed during one episode. Appendix Ampendix[]dettalle detaiis all baselines, the model architecture, and the used hyperparameters.
4.1 DEPENDENCY ON MEMORY
To support our claim that Memory Gym's environments strongly depend on memory, the agents are trained with PPO (memory-less), GRU-PPO, frame stacking, and HELM MMActGrid, MPGrid (hidden goal), and Searing Spotlights. Figure 4(a) shows the results on MMActGrid where GRU- PPO successfully executes ten commands after 50 million training steps. while memory-less PPO is ineffective by solving only one to two commands. Even though the task requires short-term memory only, HELMM& and the frame stacking baselines are notably inferior to GRU-PPO. Concerning MPGrid
7

---

Published as a conference paper at ICLR 2023
IQM Commands Executed
10
1.0
1.0
8
Rate
0.8
OOM Success Rate
0.8
VM
0.6
0.6
Wrond
0.4
0.4
002
100
160 Steps (in millions)
150
Steps (in millions)
Steps (in millions)
(a) Mortar Mayhem Novel Seeds
(b) Mystery Path Training Seeds
(c) Mystery Path Novel Seeds
By scaling the hardness of Mortar Mayhem and Mystery Path, unsolved and diverse levels of difficulties emerge. The agents are trained using GRU-PPO. MMAct and MM ask for five commands to be executed and not ten, Only the generalization performance is shown because, as in MPGrid, the goal and origin visible, while the other two experiments hide the goal (IG) or both (!GO). MP Dense provides denser reward signal. Whenever the agent touches tile of the path for the first time, it is rewarded by 0.03.
Figure 5:
figure 4[a) both performances are nearly identical.
(Figure 4(b). GRU-PPO needs the entire training time to succeed and generalize. HELM reaches success rate of about 0.84 during generalization. The frame stacking agents perform worse, while PPO's performance is close random. Results of Searing Spotlights are illustrated in Figure [ All baselines are barely able to reliably complete this environment, although HELM is the best baseline achieving a success rate of about 0.3. To further investigate this outcome, subsection 4.3 examines environment ablations and hints at why randomly moving spotlights may be significant issue to memory based on recurrence.
4.2 LEVELS OF DIFFICULTY IN MORTAR MAYHEM AND MYSTERY PATH
Levels of difficulty, ranging from easy to unsolved, emerge when scaling Mortar Mayhem and Mys- tery Path and if trained using GRU-PPO. As seen in figure S(a) the agent in MMActGrid only needs 50 million steps to accomplish the whole task. In MMAct, it needs twice as long to solve only five commands. Note that the available commands in the grid variants comprise only five command choices and not the full range of nine ones. Adding the Clue Task notably increases the agent's challenge because the agent trained MMGrid needs the entire training time succeed. At the same time, the complete task of MM remains unsolved. The performance on training and novel seeds nearly identical with nearly no variance, which can be explained by the high uncertainty of this environment. A single observation leaves the agent clueless about which of the nine tiles to move next. Therefore the agent has no other choice but to obtain the ability to memorize and recall past events.
Concerning MPGrid, if the goal and origin are part of the agent's observation, it takes about 65 million steps for the agent to generalize to novel seeds, shown in figure 5(c) If only the goal is hidden, the agent needs the entire training time to reach a success rate of 1.0. Hiding the goal and origin degrades the agent's performance MP or MP Dense do not evaluate well on novel seeds. The agent MP Dense makes little progress due to the help of denser reward function that signals reward of 0.03 for visiting tile of the path for only the first time. Consequently, we also experimented with a negative reward -0.001 for each step taken, severely hurting the training performance.
success rate of 0.2 novel seeds. Agents trained on
4.3 RECURRENCE IS VULNERABLE TO SPOTLIGHT PERTURBATIONS
Searing Spotlights remains puzzle. It is difficult to tell whether the agent's trained recurrent policy leamned anything meaningful. Collecting the coin seems worse than chance. The agent's is remaining health points drop to zero in nearly all evaluated episodes because avoiding the spotlights does not work well. It seems that the agent struggles to determine what it controls. Even though the spotlights unveil information on the ground truth of the environment or leave room for exploitation, the recur- rent agent seems severely hurt by the perturbations inflicted by the randomly moving spotlights,
8

---

Published as a conference paper at ICLR 2023
12
PPO
10
Spotlights
1QM Return
GRU-PPO Spotiighs
Steps (in millions)
(a) Fully Observable
(b) Spotlight Perturbation
(c) BossFight Performance on Training Seeds
Figure 6: Under perfect information, the noise inflicted by spotlight perturbations harms the recur- rent agents' performance in Searing Spotlights (a) and Procgen's BossFight (b). (c) denotes the training performances on training seeds when trained with and without perturbations using PPO and GRU-PPO. The IQM return denotes the summed rewards that the agent achieved during one episode.
which can be considered noise. Regardless of any executed measure (see appendix (Foor details), for instance, environmental ablations, model architectures, and hyperparametens, progress in learning a reasonable behavior is lacking. When trained with GRU-PPO, the only solvable scenario is to remove the spotlights. In this case, if the global light is initially turned on (perfect information) and dimmed until off during the first few steps, the agent rapidly collects the coin and uses the exit. In contradiction, if the global light is slightly dimmed while spotlights are present (Figure (oa)))o(....))))))_))) GRU- PPO fails. On the other hand, memory-less PPO quickly succeeds because this fully observable task trivial.
Those results are also apparent in training a different environment. We choose Procgen's 's BossFight because rather needs fewer steps to train as seen the results Cobbe et al. 2020, BossFight set to utilize only one level causing the boss always to behave the same. We extrapolate this level by 100,000 spotlight perturbation seeds under perfect information. Its background rendered white to match Searing Spotlight's visual appearance more closely. The now-established BossFight task (Figure 6(b) invites overfitting. Figure 6(c) shows results with and without spotlight perturbations on PPO and GRU-PPO. Notably, GRU-PPO, perturbed by spotlights, struggles.
CONCLUSION AND FUTURE WORK
By accomplishing the desiderata that have defined and examined in this work, and by evalu- ating the baseline experiments, we believe that Memory Gym has proven potential to complement the landscape of Deep Reinforcement Learning benchmarks specializing agents leveraging mem- ory. Memory Gym's unique environments, Mortar Mayhem, Mystery Path, and Searing Spotlights, strongly depend on memory and frequent egent-memory interactions. As the environments' diffi- culties are smoothly scalable, current and future approaches to memory may find their right fit for examination. To our surprise, Searing Spotlights poses yet unsolved challenge to agents leveraging GRU or LSTM.
In future work, it will be intriguing to see whether adjacent and novel memory approaches suffer from the noisy perturbations inflicted by the spotlights. HELM's performance on Searing Spot- lights may point toward a transformer-basee approach (e.g. HCAM (Lampinen et al. (02T). GTrXI. (Parisotto et al., 2020),, while approaches that have not been used yet in the context of Deep Rein- forcement Learning (e.g. Structural State-Space models (Gu et al. (022)). ), are also considerable for Memory Gym in general. Consecutive work may extrapolate the concepts of Memory Gym's envi- ronments. Mortar Mayhem could be advanced to an open-ended environment. For instance, the Clue Task and the Act Task could be stacked infinitely. Every Clue Task could only show command at a time, while the Act Task requires the agent to execute all commands that were shown so far. Hence, memory approaches could be compared by achieved episode lengths, which with the amount of information that the agent must be able to recall. Such goal in mind may inspire Mystery Path as well. The to-be-traversed path could be endless. However, a larger environment may not fit into the agent's 's observation that an egocentric observation could compensate.
correlated

---

Published as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning the edge of the statistical precipice. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glemn Powell, Alex Ray. Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in- hand manipulation. https://dol http://66b..rrr...11170///111310
Robotics Res., 39(1), 2020. doi: 10.117700288364119887777 URL
Andrea Baisero and Sammie Katt. gym-gridverse: Gridworld domains for fully and partially ob- servable reinforcement learning. https: com abba sero/ /eer//ymmeeeeeeeeeeeeeeeee.eee gym-ggmmggg- gr dverse 2021.
Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In 8th International Con- ference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htt.httts///peeeeeeeeee
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq. Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Ander- son, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen, Deepmind lab. CoRR, abs/1612.03801, 2016. URL org/ abs
/aarg/abs/aag 1612.61110 1at.s/6811000011111
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. J. Artif. Intell. Res., 47:253-279 279,, 2013. doi: 10.1613/jair.3912. URL
doi wtt.://..0......11........ 3912 3912
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung. Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowiez, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. CoRR, abbs/1912.0668. 2019. URL http: arxiv. org abs 19121 1b2.//2...11108 06680
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016. /ttL:a/ttaaaxxi///
0as/1606.00555
Maxime Chevalier- Chevalier-Boisven, Boisvent, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https:
com comm/mxximmen wwwp////wmemmm///mmmmmmmm.mmmmma
2018.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1724 1734. ACL, 2014. doi: 10.3115/v1/d14-1177 URL https://don http//di.oggg/...// http:/110.11111111111111
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural genera- tion to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 2048-20000 2056. PMLR, 2020. htthttp: htttt::/pprccediing/ mlr. w.a.ppres///e cttres//v11//obbeelecc
Matthew Crosby, Benjamin Beyret, Murray Shanahan, José Hernández-Oralle Lucy Cheke, and Marta Halina. The animal-ai testbed and competition. In Hugo Jair Escalante and Raia Hadsell (eds.), Proceedings of the NeurIPS 2019 Competition and Demonstration Track, volume 123 of Proceedings of Machine Learning Research, pp. 164-176. PMLR, 08-14 Dec 2020.
10

---

Published as a conference paper at ICLR 2023
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki. Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean-Mare Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic control of toka- mak plasmas through deep reinforcement learning. Nature, 602(7897::4144119 Feb 2022. 1038/
ISSN 1476-4687. doi: 10. (1188//11586-021-44433 241586-021-04300
URL
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: scalable distributed deep-rl with importance weighted actor-leamer architectures. In Jen- nifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüsxann Stockholm, Sweden, July 10-15, 2018, vol- 80 of Proceedings of Machine Learning Research, pp. 1406-1415. PMLR, 2018. URL
http::/proceedings mlr wtt..ppees///////ppphlll.1...1.. html
Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomênech Badia, Gavin Buttimore, Charles Deck, Joel Z Leibo, and Charles Blundell. Generalization of reinforcement learners with working and episodic memory. In Advances in Neural Information Processing Systems, pp. 22448-12557 2019.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
net./
forum? Borum?1d=uYY.Fooo
AC
Çaglar Gülçehre, Tom Le Paine, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil C. Rabinowitz, Duncan Williams, Gabriel Barth- Maron, Ziyu Wang. Nando de Freitas, and Worlds Team. Making efficient use of demonstra- tions solve hard exploration problems. In 8th International Conference Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: https://openreviee net forum? Siw.ummmddSyyyyeeillee
Danijar Hafner. Benchmarking the spectrum of agent capabilities. CoRR, abs/2109.06780, 2021.
URL
org. /ttps///19.om 06780
Peter F. Hart, Nils J. Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE Trans. Syst. Sci. Cybern., 100-1000100 1968. doi: 10.1109/TSSC. 1968.300136. URL
otg 1ww1111999SSCC 1968. 300136
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposia, Arlington, Virginia, USA, November 12-14, 2015, pp. 29-37. AAAI Press. 2015, /16733
URL http:// /tt////wwww www. aaai org/ www.///////.//o/o.ooooooo.oee.// indeiiidxx.... haameexwphh//ss//SSl///aeaeeee
Nicolas Heess, Jonathan Hunt, Timothy Lillicrap, and David Silver. Memory-based control with recurrent neural networks. CoRR, abs/1512.00455, 2015. URL /att:://atxxi///
hbs/1512.044555
Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A. Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning task inference. CoRR, abs//1955064424 2019.
URL
abs 195.. 06424]
Arthur Juliani, Vincent-Piene Pieeree Berges, Esh Vekay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. CoRR, htss/8899.0227,, 2018.
URL http:://axxiiii org/ abs / 8098 02627
Arthur Juliani, Ahmed Khalifa, Vincent-Pieee Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: generalization challenge in vision, control, and planning. In Sarit Kraus (ed.), Proceedings of the TwentyFFggth International
11

---

Published as a conference paper at ICLR 2023
Joint Conference Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp- 2684 2691. orcai.org, 2019. doi: 10.24965/ijeai.2111//77. URL httph::t/////... http://oi.oo.oo /ttp:///..oozz...... 24963/ ptt..aaaa...........
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent ex- perience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2019. URL [tKLLptta///pperreeeeee
net/ wtt.e/rrrmmmmini1imeaae
Andrew Kyle Lampinen. Stephanie C. Y. Chan, Andrea Banino, and Felix Hill. Towards mental time travel: a hierarchical memory for reinforcement learning agents. CoRR, abs/2105.14039,
2021. URL httttps:/arxxv..ott wttp:/www.org/bb///05.... htttppes//axxi...cr////////..... 14039
Luckeciano C. Melo. Transformers are meta-reinforeemme learners. In Kamalika Chaudhuri. Ste- fanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Con- ference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, vol- 162 of Proceedings of Machine Learning Research, pp. 15340-15359. PMLR, 2022. URL ml www.mr....sr.p...mmmmmmooml. vi 62 me 1o222 1o222.htmlll
Lingheng Meng, Rob Gorbet, and Dana Kulic. Memory-based deep reinforcement learning for pomdps. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021, Prague, Czech Republic, September 27 Oct. 1, 2021, pp. 5619-5626. IEEE, 2021. doi: URL https: /tt:://doo..oo http:///ooo.rr/////////o 1wt.00.11099/RSST11
10. N0.1109/ROS5111 26.21.9119.2))))333111 2021 9636140
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 518(7540):529-533 2015. I038/nature14233
doi: 10.1038/nature 14236.
URL huttps://doii http:///oo...g///////
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many pomdps. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, (CML2022) 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learming Research, pp. 16691-16723, PMLR, 2022. URL sstpss:/prrccedinggs
press/v162/ni3
html
Fabian Paischer, Thomas Adler, Vihang P. Patil, Angela Bitto-Nemling, Markus Holzleitner, Se- bastian Lehner, Hamid Eghbal-Zadeh, and Sepp Hochreiter. History compression via language models in reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 17156 17185. PMLR, 2022. URL https: pattps://proceedingg sanggs.m11 .mrr. wwtpe///aaacceeaaaaeee press/vl 62 pai www.rrcher2aaaccmm html
Emilio Parisotto and Ruslan Salakhutdinov. Efficient transformers in reinforcement learning ing actor-learner distillation. In 9th International Conference on Learning Representations,
ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:
//openreview. wwweenrview.net/ffooooe
_xF
Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Çaglar Gülçchre, Siddhant M.
Jayakumar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 7487- 7498. PMLR, 2020. URL htl.http:/// htttp://proceet/ ngs ml wtr.prees/vill wttee///i11//pprsstttoo/ html
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference Learning Representations, ICLR 2016,
12

---

Published as a conference paper at ICLR 2023
San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:
//arxiv.orq/ htsr:/bbaa//00 02438
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. wiXiv:177.006444 2017.
Wenling Shang, Xiaofei Wang, Aravind Srinivas, Aravind Rajeswaran, Yang Gao, Pieter Abbeel, and Michael Laskin. Reinforcement learning with latent flow. In Marc' Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 22171-22188 2021. URL https: http::///occeeeddeeeooooe www.sse.eriippcom wttp:/ppp.cpppppe/22//22 Abstract..................a..... .htrl
Vitalii Sopoy and Ilya Makarov. Transformer-based deep reinforcement learning in vizdoom. In Evgeny Burnaev, Dmitry I. Ignatov, Sergei Ivanov, Michael Khachay, Olessia Koltsova, Andrei Kutuzov, Sergei O. Kuznetsov, Natalia Loukachevitch, Amedeo Napoli, Alexander Panchenko, Panos M. Pardalos, Jari Saramäki, Andrey V. Savchenko, Evgenii Tsymbalov, and Elena Tu- tubalina (eds.), Recent Trends in Analysis of Images, Social Networks and Texts, pp. 96-110, Cham, 2022. Springer International Publishing. ISBN 978-3-031-15168-2.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena A benchmark for efficient transformers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httttpss://peeeeeiiiitt net./ GrC2k
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 9998-6008 2017. URL http/tts///ppcccdnnng....o. nerips httpppsccccpppppeeeee pccppappe//11///aa..t. 33FFa2255476e11110000caaaaaaaaa.
html
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czamecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforce- ment learning. Nat. 575(7782):350-354, 2019.
doi: 10. 10.103/641188--099 ---2------11 URL 019 1724-z
Jane Wang, Michael King, Nicolas Porcel, Zeb Kurth-Nelson, Tina Zhu, Charlie Deck, Peter Choy, Mary Cassin, Malcolm Reynolds, Francis Song, Gavin Buttimore, David Reichert, Neil Rabi- nowitz, Loie Matthey, Demis Hassabis, Alex Lerchner, and Matthew Botvinick. Alchemy: A structured task distribution for meta-reinforremmen learning, arXiv preprint arXiv:2102.02926. 2021.
abs htw.bb////222...2
Paul Werbos. Backpropagation through time: what does and how to do it. Proceedings of the IEEE, 15500))1500-150, 1990.
Marek Wydmuch, Michat Kempka, and Wojeiech Jaskowski. Vizdoom competitions: Playing doom from pixels. FEEE Transactions on Games, 2018
13

---

Published as a conference paper at ICLR 2023
A INSTALLATION AND EXECUTION OF MEMORY GYM
Create Anaconda environment
$ conda create -n memory-gym python=3.7 --yes
$ conda activate memory-gym
# Install Memory Gym
$ pip install memory-gym
# Play Mortar Mayhem
$ mortar_mayhem
Figure 7: Install and play Memory Gym's environments.
Memory Gym is available as PyPi package which installs the required dependencies gym and PyGame. We recommend to utilize Anaconda" to allow for console scripts to be executed. This way, the environments be played using the following commands:
mortar_mayhen
$ mortar_mayhem_b
$ mortar_mayhem_grid
$ mortar_mayhen.b_grid
$ mystery_path
$ mystery_path_grie
$ searing_spotlights
Figure 8: Console scripts to play Memory Gym's environments
import gym
import memory_gym
Configure desired reset parameters
options ["allowed_comandde"" 5, "command_count", [5])
env gym.mak ("MortarMayhem-vO"))
obs rev.reset (options)
print (obs www.iiialloeeeeaaaaeaiaaaaa .sapee # (84, 84, 3) print (ewv.action_saa space)
MultiDiscrete [3 3])
done False
while not done:
action = anv.actio icciion satiiiinnsseeeeeeee 1aaale () paatiionnsppp
obs, reward, done, info env. step (action)
Figure 9: Code interactions with Memory Gym environments
DDar reviewers, the package will he available upon acceptance due to anonymity. https: wwwp///wwww///////////////////w/ wttp://wwaannn...cc/oooocccc.oee
14

---

Published as a conference paper at ICLR 2023
B EXTENDED RELATED BENCHMARK OVERVIEW
Table 2: Comparison of the simulation speed our benchmark to related ones. A constant action executed to measure the speed. Procgen encompasses the mean steps per second for the envi- ronments CaveFlyer, Jumper, Heist, Dodgeball, Maze, and Miner. The reset duration is included in steps per second. Some benchmarks are only measured by single environment. The measurement
was done on a AMD Ryzen 2700X.
FPS
Reset Duration
Benchmark
Mean
Std
Mean (seconds)
Procgen Memory Distribution (Cobbe et al. (2020)
18530
5453
4e-6
Mystery Path (ours)
12187
330
5c-4
Mortar Mayhem (ours)
11692
179
2e-4
Atari Breakout
7117
19
6e-3
[elllemare et al. 2013)
DM Ballet
6631
795
6e-4
(Lampinen et al. (022
Searing Spotlights
5490
69
5e-4
(ours)
MiniGrid Memory
5185
141
le-3
Shevalier-Boisverl et al. (2018))
MiniWorld M'Mzze
1162
65
6c-4
Chevalieer et 2018)
VriVeerse Memory Nine
938
89
le-3
Baisero & Katt (021))
ML-Agents Hal way
702
42
le-3
Juliani et al. (018
VizDoom My Way Home
549
22
3e-3
Wydmuch et al. (2018)
Crafter
482
33
1e-1
(Hafner (2021)
DM Spot the Difference
442
20
le-1
(Fortunato et al. (2019))
DM Lab 30 rooms watermaze
433
53
5e-1
Beattie et al. (0016)
DM Alchemy
308
31
2c-1
Wang et al.
2021
DM Hard Eight ball room navigatioo nuuee cubes
160
2e-1
Gülçehre al.
2020)
AnimalA1 aa c32
190
59
le-2
Crosby (0020)
Obstacle Tower
43
2
16e-1
(Juliani et al. (0019)
15

---

Published as a conference paper at ICLR 2023
C
ENVIRONMENT DETAILS
C.1 ACTION SPACES
A
E
90p
90%
(a)
(b) Move Forward
(c) Rotate Left
(d) Rotate Right
Figure 10: Memory Gym's 's environments feature a multi-discrete action space as seen in (a). One dimension (green arrows) denotes the agent's vertical velocity. The other one (red arrows) refers to the agent's 's horizontal velocity. Both dimensions feature a no-op action. Therefore, the agent can utilize both dimensions move into eight distinct cardinal directions (all arrows). The speed of the agent is fixed at 2.83 pixels per step. Mortar Mayhem and Mystery Path also provide grid-like locomotion based on discrete action space four actions. The agent can move forward (b) one tile at time. Its forward speed in Mortar Mayhem is 14 pixels second. Its forward speed in Mystery Path 12 pixels per second. Two other actions allow the agent to rotate left (c) or right (d) at 90 degrees. The last action is no-op.
C.2 EPISODE LENGTHS
Table Lengths of successful (best case) and failed (worst case) episodes in Mortar Mayhem. The values are calculated by equations (1-6).
MM
MMAct
MMGrid
MMActGrid
Min (Lower Bound)
38
19
46
Max (Upper Bound)
135
115
119
79
ClueTask =
(ShowDurations + ShowDeloy) CommandCount
(1)
Actaask =
IeseccttinDDDaatiin + Iitecutionnelag)) CommandCount
(2)
Act' = ActI Execution Delay +
(3)
M Moxtppisodeeength ClueT + ActI ask
(4)
Act.uue Farecutionn +
(5)
Clue ask + Act Failure.
(6)
Table Lengths of successful episodes in Mystery Path, Mystery Path Grid, and Searing Spotlights. The values for MP and MPGrid are retrieved from one agent trained on the ground truth. The values for Searing Spotlights are based on an optimal policy. The optimal policy traverses the shortest path to the coin and then to the exit.
MP
MPGrid
Searing Spotlights
Min (Lower Bound)
23
Mean
39.13
13.17
35.33
Std
7.36
3.22
8.01
Max
73
Upper Bound
512
128
512
Policy
Agent
Agent 100k
Optimal 100k
Episode Samples
100k
16

---

Published as a conference paper at ICLR 2023
C.3 RESET PARAMETERS TO CONFIGURE THE ENVIRONMENTS
Table 5: These are the default parameters that we used throughout this paper as default. Parameters with indicate that these uniformly sampled. Values in square brackets are discrete choices, while values in parentheses consider range to sample from. Mortar Mayhem Grid features the same parameters as its parent, but only the modified ones are presented.
Mortar Mayhem
Searing Spotlights
Parameter
Default
Parameter
Default
Agent Scale
0.25
Max Episode Length
512
Agent Speed
2.5
Agent Scale
0.25
Atena Size
5
Agent Speed
2.5
Number of Available Commands
Agent Visible
False
Number of Commands*
[5]
Agent Health
100
Command Show Duration
[3]
Sample Agent Position
True
Command Show Delay*
[1]
Use Exit
True
Execution Delay*
[6]
Exit Scale
0.5
Execution Duration*
[18]
Exit Visible
False
Hide Visual Feedback
False
Number of Coins*
[1]
Reward Command Failure
Number of Initial Spotligtt Spawns
Reward Command Success
0.1
Number of Spotlight Spawns
30
Reward Episode Success
0
Spotlight Spawn Interval
30
Mortar Mayhem Grid
Spotlight Spawn Decay
0.95
Parameter
Default
Spotlight Spawn Threshold
10
Number of Available Commands
Spotlight Radius*
(7.5-13.75)
Number of Commands*
[10]
Spotlight Speed*
(0.0025-000075)
Execution Delay*
[2]
Spotlight Damage
Execution Duration* *
[6]
Light Dim Duration
Mystery Path
Light Threshold
255
Parameter
Default
Hide Visual Feedback
False
Max Episode Length
512
Render Background Black
False
Agent Scale
0.25
Reward Inside Spotlight
Agent Speed
2.5
Reward Outside Spotlights
Cardinal Origin Choice
10, 1, 2, 31
Reward Death
0
Hide Origin
False
Reward Exit
Hide Goal
True False
Reward Coin
0.25
Hide Visual Feedback
Reward Max Steps
0
Reward Goal
Reward Fall Off
Reward Path Progress
0
Reward Step
0
17

---

Published as a conference paper at ICLR 2023
C.4 MORTAR MAYHEM GRID TRAJECTORY
8
Figure 11: A successfully completed episode of Mortar Mayhem Grid showing the ground truth. The zip archive of the supplementary material provides an episode as a video.
18

---

Published as a conference paper at ICLR 2023
C.5 MYSTERY PATH GRID TRAJECTORY
Figure 12: Various randomly generated levels in the Mystery Path environment.
Figure 13: A successfully completed episode of Mystery Path Grid showing the ground truth. The zip archive of the supplementary material provides an episode as a video.
19

---

Published as a conference paper at ICLR 2023
C.6 SEARING SPOTLIGHTS TRAJECTORY
Figure 14: An excerpt of Searing Spotlights showing the ground truth and using a frame skip of
four. The zip archive of the supplementary material provides an episode as a video.
20

---

Published as a conference paper at ICLR 2023
(512)
Reshape sequences
Value
Reshape to batch
Corvv Size: Stride 4
Conv2 Size: Stride: 2
Conv3
(512)
Visual Observation
32
Filtera:
Faatten
FC (512
Size: 3
Stride:
Policy Branch
FC (512)
(128)
Concatenate
Policy (3)
Vector
Observalion
Figure 15: GRU-PPO utilizes actor-critic feed-forward convolutional recurrent neural network. Visual and vector observations are encoded as an entire batch by either convolutional or fully con- nected layers. The encoded features concatenated and then reshaped to sequences before feeding them to the recurrent layer. Its output has to be reshaped into the original batch shape. Further, the forward pass is divided into two streams relating to the value function and the policy, The number of policy heads is equal to the number of action dimensions given by multi-discrete action space.
spiit
Wg: to
Wg. fT
'ET
seqg, ln
seqp, RAT
padat+1 max(et)
:
w...
:
epi... t...
:
seq t...
:
padat+1 max(eT)
www. kg
www. tr
ople; to
opie: lET
sequ, kg
padat+1 mex(eT)
Sampled Trajectories
Spilt Episodes
Spltt Sequences
Padding
split
zero pad
Figure 16: The data preprocessing starts out by sampling trajectories across W workers for T steps. Next, E episodes of varying length ET are extracted from the trajectories. Those can be further split into # sequences of varying length #T' At last, zero padding is used to retrieve sequences of fixed length mTx(#T).
D BASELINES
D.1 GRU-PPO
Our GRU-PPO baseline utilizes the clipped surrogate objective (Schulman al., (017). Due the usage of the Gated Recurrent Unit as the recurrent layer, the to be selected action a, of the policy and hidden state h; of the recurrent layer. A, denotes advantage estimates based on generalized advantage estimation (GAE) (Schulman et al. 2016). the parameters of neural net and the clip range.
Ty depends on the current observation
L (0) = [m(min(qr
To(arloe, he) Tesla(a oz, ht)
with ratio (:())
The value function is optimized using the squared-error loss Lý (0). Ho.T)(o;)) denotes an entropy bonus encouraging exploration (Schulman et al., 2017). Both are weighted using the coefficients c₁ and C2 and are added to L to complete the loss:
1 CV (.) o.LY (0) + (or,
(8)
with (.)(0) (Voloe, he) vtaroz2
21

---

Published as a conference paper at ICLR 2023
IMPLEMENTATION DETAILS
Three major components have to be implemented: the forward pass of the model (Figure 15 the processing of the sampled training data (Figure 16], and the loss function (Equation 9p.
The training data is sampled by fixed number workers for a fixed amount of steps and is stored as tensor. Each collected trajectory may contain multiple episodes that might have been truncated. After sampling, the data has to be split into episodes. Optionally, these can be further split into smaller sequences of fixed length. Otherwise, the actual sequence length is equal the longest episode. Episodes or sequences that are shorter than the desired length are padded using zeros. As the data is structured into fragments of episodes, the hidden states of the recurrent layer have to be selected correctly. The output hidden state of the previous sequence is the input hidden state of its consecutive one. This approach also known as truncated backpropagation through time (truncated bptt). Finally, minibatches sample multiple sequences from the just processed data.
the length of
Concerning the forward pass of the model, is more efficient to feed every non-recurrent layer the entire batch of the data (i.e. batch size workers steps) and not each sequence by one. Whenever the batch is going to be fed to a recurrent layer during optimization, the batch has to be reshaped to the dimensions: number of sequences and sequence length. After passing the sequences to the recurrent layer, the data has to be reshaped again to the overall batch size. Note that the forward pass for sampling trajectories operates keeps its shape throughout the entire forward pass.
sequence length of one. In this case, the data
Once the loss function is being computed, the padded values of the sequences have be masked so that these do not affect the gradients. mask is the average over all losses not affected by the paddings.
[masks x LLVH (0)
mask (0)
(9)
E [maske]
where padding is used
with maski
where no padding is used
D.2 MEMORY-LESS PPO
In the of memory-less PPO, Frame Stacking, and HELM, the recurrent layer is removed. The training data not split into sequences. Loss masking is not used.
D.3 FRAME STACKING
n past frames are stacked the agents current visual observation. In the case of stacking frames, RGB frames are used. Grayscale frames are utilized in the case of stacking 16 frames to reduce the model's 's input dimension.
D.4 HELM
If HELM is used, the model receives another encoding branch that receives the current grayscale visual observation. The observation is then propagated by frozen hopfield and pre-trained trans- former as described by Paischer et al. [022 The resulting features are concatenated to the ones from the CNN and the vector observation encoder.
D.5 CHOICE OF HYPERPARAMETERS
Table @ememerates the utilized hyperparameters. The agent Searing Spotlights uses a frame skip of 2, which halves the length of an episode easing the agent's exploration. The quite large batch size is chosen to max out the wall-time efficiency the utilized resources. One training based on GRU-PPO utilizes almost 32GB of VRAM. Most of the hyperparameters related to memory were found by running grid search on an early version of Mortar Mayhem Grid. Besides that we made coarse hyperparameter sweeps especially for Searing Spotlights that did not yield any benefit. We
22

---

Published as a conference paper at ICLR 2023
Table 6: These are the hyperparameters that used for all training The sequence length is dynamically set by the longest episode inside the batch of the gathered training data. The experi- ments Searing Spotlights and Mystery Path utilized fixed sequence length of 128. The learning rate and the entropy coefficient decay linearly.
Hyperparameter
Value
Training Seeds
100000
Sequence Length
Max
Number of Workers
32
Worker Steps
512
Batch Size
16384
Number of Mini Batches
8
Mini Batch Size
2048
PPO Updates
10000
Training Steps
163840000
Discount Factor Gamma
0.99
GAE Lamda
0.95
Epochs
3
Value Loss Coefficient
0.25
Max Gradient Norm
0.5
Clip Range Epsilon
0.2
Initial Learning Rate
3c-4
Final Learning Rate
le-4
Initial Entropy Coefficient
le-4
Final Entropy Coefficient
le-5
HELM Beta
1000
Optimizer
Adam W
tried the values 50, 100, 200, and 1000 for HELM's beta hyperparameter on MMActGrid. All values ended up with nearly identical performance. We ultimately chose 1000 because there is not much variance in individual observations in Mortar Mayhem and Mystery Path. Overall, we rather allocated our resources scale the difficulty of Mortar Mayhem and Mystery Path, while investigating the issues of Searing Spotlights.
23

---

Published as a conference paper at ICLR 2023
E
WALL-TIME EFFICIENCY OF HELM
IQM Commands Executed
10
GRU-PPO
HELM Random
8
6
4
0
10 20
30 Hours
40
50
60
Figure 17: Generalization performances on novel seeds in Mortar Mayhem Act Grid showing the wall-time efficiency of PPO, GRU-PPO, and HELM. These experiments are run NVIDIA and an AMD EPYC 7542 CPU (32 cores).
A100
1.0
GRU-PPO
IQM Success Rate
0.8
Random
0.6
0.4
0.2
0.0
10
20
30
40
50
60
70
80
Hours
Figure 18: Generalization performances on novel seeds in Mystery Path Grid (hidden goal) showing the wall-time efficiency of PPO, GRU-PPO, and HELM. These experiments are run on an NVIDIA 100 Tensor-Core-GPU and an AMD EPYC 7542 CPU (32 cores).
24

---

Published as a conference paper at ICLR 2023
F MEASURES TAKEN ON SEARING SPOTLIGHTS
(a) Default
(b) Fully Observable
(c) No Spotlights
(d) No Chessboard
Black Floor
Figure 19: Varying visual representations of Searing Spotlights examined. (a) poses the original task. (b) provides perfect information, because the global light is just dimmed. (c) disables the spotlight dynamics while dimming the global light until off. (d) removes the tiled background. (e) renders the floor black that requires to render the spotlights' circumference white.
Multiple experiments were conducted to ablate the environment's difficulty and to vary the model architecture. Figure [9] shows few variations of the visual representation of Searing Spotlights. Whenever spotlights that we tried with limited success to help the agent learn a more meaningful policy. Increasing the scale of all entities but the spotlights reducing the number of spotlights may have undesirable consequences to the task's quality. Random policies may be more successful under these measures.
present, the recurrent agent fails. Subsequently, we enumerate the measures
Reduce episode length to ease explo- ration
Add health points bar
Render the agent's 's last action onto to the observation instead of feeding
Leverage frame skipping
Raise agent speed
one-hot encoded feature vector
Scale up all entities except spotlights
Feed the agent its exact position as normalized scalars
Vary spotlights
Add negative reward of -0.01 for the agent being inside the spotlights Stationary spotlights
Slightly dim the light (perfect infor- mation)
Render the environment's floor black while drawing the spotlights' circumference white
Fewer spotlights
Constant spotlight size and speed
Vary task
Vary model architecture
Use only one coin and no exit
Add residual connection around the GRU cell
Use the exit but no coin
Agent always starts at the center of the level
Use LSTM instead of GRU
Increase agent health points
Reduce sequence length from 128 to
Add negative death reward of -0.1
64
Vary agent observation
Use Impala CNN Espeholt al. (2018) instead of Nature CNN Mnih al. (2015)
Environment is fully observable for few steps
Make the agent, coin, exit, or all of them permanently visible
Add fully connected layer between
CNN and GRU
25

---

Published as a conference paper at ICLR 2023
G SOURCE CODE AVAILABILITY OF MAANSFORMER-BASER APPROACHES
During the rebuttal, few complex frameworks were brought to our attention that claim to support DRL algorithm leveraging Gated TransformerXL (Parisotto al., 2020) just TransformerXL, namely Rllib, DI-engine, and Brain Agent We tried to use Memory Gym with RLib but observed questionable outputs like negative rewards (even though the environments did not use any negative rewards). The framework is too complex to analyze and eventually to verify. The same accounts for Brain Agent, which is densely commented nor documented. DI-Engine implements an R2D2 Kapturowski et al. 2019) variant based on Gated TransformerXI. We tried it and learned that it has poor sample throughput by a magnitude of 10 compared GRU-PPO. We do not claim that those frameworks dysfunctional, because we just report our limited experience by exploring those.
By the time of acceptance, we finally implemented lightweight and easy-to-follow TransformerXL + PPO baseline, which is now used our momentary work on Memory Gym. Source Code: https: http://www.....aaaaae//////aaaaa
memooyyppoo
26


Reviews:
[{"review_id": "Vu_lbPfq3lR", "reviewer": "Reviewer_aYs5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a benchmark to solely focus on benchmarking Deep Reinforcement Learning agents' memory and ability to generalise. In addition to open source accessibility, simulation step speed and flexibility (level hardness, noise etc) the authors claim their benchmark is better suited to distinctly evaluate the role of memory in DRL performance. They show the need for RNN implementations of PPO taking learning from sequences to successfully complete (2/3) of the games where memory-less implementations fail.", "summary_of_the_review": "The work is of high relevance for benchmarking a very important aspect of learning which is the capability of networks and in this case reinforcement learning agents to achieve tasks requiring long sequence memorisation. The manuscript itself is generally well written, but lacks some clarity and rigour in manipulating core ideas to the case it is making. With additional elucidation of links between the proposed tasks and the quantitative increase in memorisation required for solving them, this work could prove to be suitable for benchmarking models not only today, but into the future as models improve. This potential is still vague in the current state of the work.", "strength_and_weaknesses": "Strengths:\nThe work tackles a very important problem that is of great interest to the Recurrent Neural Network (RNN) community, which is long term dependency learning, and large sequence memorisation, in the context of memory requiring Deep Reinforcement Learning (DRL) tasks.\nThe authors put notable effort into ensuring their environments require sequence memorisation capabilities to be successfully completed. They also clearly state a case for the need for such a benchmark among the plethora of available environments to specifically tackle this issue. Indeed:\n- Code is well written and simple enough to use.\n- The desiderata is met, especially in terms of enforcing a dependency between success and long term memory decision making.\n- Difficulty scaling for the games enables benchmarking memory capabilities of simplistic models as well as larger architectures.\n\nWeaknesses:\nThe paper presenting a new benchmark, it is naturally void of any technical novelty. However, it is unfortunate that it also lacks technical precision for many of the concepts manipulated:\n- The word memory itself, which gives the benchmark its whole purpose, is used 123 times throughout the paper without being clearly defined, or more specifically, put into context. It is evident that the authors refer to maintaining a representation of information from long sequence of past grids and its use for future decisions, however one has to refer to Fig 14 in appendix D on page 20 of the manuscript for the technical representation of the recurrent neural network block in the DRL architechture.\n- Thus, a clearer presentation of memory mechanisms in RNNs and their relationship to DRL algorithms would be much welcome to elucidate what exactly is being evaluated by the benchmark. With a plethora of  advances on RNN memory capabilities for sequence modelling (see works such as \"Efficiently Modeling Long Sequences with Structured State Spaces\" by Gu et al, and \"Liquid Structural State-Space Models\" by Hasani, Lechner et al), any performance evaluation should be based on a quantitative measuring of the dimension and sequence lengths of information required to achieve the task against the memorisation capabilities of the RNN network used in the PPO architecture (or any other DRL method relying on RNNs for memory).\n- With memory capabilities of state-of-the-art RNNs allowing successful learning of Long Range Arena tasks such as Path-X (16384 sequence length), it would be interesting to have a more extensive quantitative discussion about how increasing difficulty levels of the games affects the dimensions of the sequences that need to be memorised. This would justify why this benchmark can be useful in the future as we develop increasingly memory capable architectures.\n- Though not at the core of the work, the authors only test GRU and LSTM architectures. Future directions evoke implementing more elaborate models. However, it would have been interesting to see better adapted architectures used especially for the Searing Spotlights environment.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NHFwMEECY0k", "reviewer": "Reviewer_guu9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper outlines some desiderata for memory tasks for RL, and proposes/releases a set of three varied benchmark tasks that meet these desiderata. The paper runs a variety of baselines on the tasks and shows that they have nice properties such as strong memory dependence and scalable difficulty. One task, searing spotlights, leads to results that are particularly interesting from a scientific perspective.", "summary_of_the_review": "I think this paper offers an interesting new benchmark for memory tasks, and some independently interesting results. The authors do a decent job of motivating their work relative to prior work. There could always be more baselines, etc. but overall I find the paper to be a useful contribution.", "strength_and_weaknesses": "Strengths:\n* Presentation is very clear.\n* Tasks are interesting and deliver distinct  memory challenges.\n* Scalable difficulty is a very nice attribute for future research.\n* Results with spotlights + recurrence are fascinating. \n\nWeaknesses:\n* If possible, it would be nice to include more baselines from different algorithmic families, such as recurrent IMPALA or VMPO (used in the Fortunato et al. and Parisotto et al. papers, IIRC—of course the latter uses transformers, but VMPO does not require them). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5b01q-xTgi", "reviewer": "Reviewer_Arip", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As a standard benchmark for POMDP tasks is desired due to rising research interests, the paper proposed 3 new Gym tasks which are memory-dependent POMDPs. The key features of the environments are (1) \"strongly depend on memory\".  (2) different levels of difficulty. (3) good meta properties (Table 1). \n\nThe authors conducted baseline experiments using PPO as RL algorithm and GRU for history extraction and compared with PPO + feedforward NN and HELM. It can be seen that the tasks cannot be solved without memory. The results also show various difficulty levels of the tasks even using PPO+GRU. Therefore, the authors conclude that the proposed tasks yield a new benchmark for memory-dependent POMDP research.\n\n\n\n\n\n\n", "summary_of_the_review": "The paper proposed a new task set for memory-dependent POMDP studies. The tasks have many desired properties, and the experiments proved that memory is necessary to solve them. Meanwhile, the task scope is limited, and the experiments could be more comprehensive and convincible by testing more algorithms and models. In sum, my recommendation is a borderline reject.\n\n-------- Updated after rebuttal -------\n\nSee my comments below", "strength_and_weaknesses": "**Strength**\n- The being addressed problem is important, that is, the lack of a standard benchmark for POMDP\n- The proposed environments have good properties (Table 1)\n- The proposed tasks are interpretable and can be customized according to the research demand\n- The proposed tasks can test model's capacity of generalization\n- The authors conducted experiments to support their claims\n\n\n**Limitations**\n- The task scope is limited to 3 Atari-like games with 2D vision as input. There is no environment with continuous action space. However, due to the large diversity of POMDP research, e.g., robot control and multi-modality observations, the proposed framework still has not solved the lack of a standard POMDP benchmark.\n- The experiments can be improved. Some of possibly useful trials are as follows: (1) Testing feedforward NN with $N$-steps observations as input. The result of how performance changes with $N$ can provide clues of how much long-term memory (vs. short-term memory) is required. (2) Testing some off-policy and probably more powerful algorithm of memory-dependent RL such as R2D2 (https://openreview.net/forum?id=r1lyTjAqYX) and ACER (https://arxiv.org/abs/1611.01224v2). \n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "4thu-0YXzj", "reviewer": "Reviewer_e7qG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors point out drawbacks of current benchmarks that claim to evaluate memory in RL.\nThey propose a novel benchmark suite for evaluating the memory component of agents trained in partially observable environments.\nThe benchmark consists of three novel environments that vary in their difficulty and can be scaled to much harder difficulty.\nBaseline results indicate that two of the three environments are solvable by a recurrent agent, while the third is not solvable.\nThe authors identify the reason for that as vulnerability of recurrence to random perturbations that are part of the environment.", "summary_of_the_review": "The paper addresses an important issue in the field of partial observability in RL, namely to introduce a benchmark for evaluating memory capabilities of novel and existing methods.\nDespite a well-structured desiderata, the authors fail to clarify the benefit of their proposed benchmark over existing ones.\nFurther, the design choices of the different environments in the benchmark are very confusing and not always justified or even contradictive.\nThe provided baseline results are difficult to interpret and not very convincing, making it difficult for new methods to reference these results.\nThey also indicate that some environments are exploitable by a markovian policy to a certain degree.\nFinally, both environments and baseline results are presented poorly and would require a major revision in order to make the proposed benchmark usable by the RL community.", "strength_and_weaknesses": "### Strengths:\n\n- Potential relevance to the field:\n  As the authors correctly point out, there is a variety of methods available, but no go-to benchmark to properly compare memory-based agents.\n  Moreover, the memory dependence of some existing memory benchmarks is questionable.\n- Reproducibility:\n  Codebase is made publicly available.\n\n### Weaknesses:\n\n- Novelty:\n  Mortar Mayhem is almost equivalent to the publicly available DM Ballet environment [1].\n  Other existing benchmarks fulfill the same requirements of their desiderata as Memory-Gym, sometimes even with higher FPS, but no justification is given on why Memory-Gym should be the preferred choice.\n  The memory dependency of some environments is questionable, i.e. PPO can exploit visual artifacts in Mortar Mayhem and Mystery Path, while being the best method on Searing Spotlights.\n- Relevance of proposed benchmark:\n  There are other environments that fulfill every point on the proposed desiderata, but it is not clear what advantage Memory-Gym has over them.\n  Two of the three proposed environments are solved already within a moderate amount of training steps with a recurrent agent.\n  The memory dependence of Mystery Path is questionable, since it can be solved in ~50% of the cases by a markovian policy as shown in Fig4b).\n  The difficulty of Searing Spotlights stems from visual artifacts (random spotlights) rather than required memory, rendering it unsuitable for evaluating memory capabilities of an agent as proposed.\n  Furthermore, difficulty scaling is achieved either through increasing required memory capacity (longer sequence of commands in MM, scaling grid in MP, etc.), or by increasing the complexity of the action space (continuous instead of grid-like movement).\n  However, the authors state explicitly that evaluating the memory capacity can be done more effectively by other disciplines such as supervised learning, which contradicts their scaling approaches.\n  Varying the time intervals between commands, in MM, however, seems to be a valid approach to scale the hardness with respect to what kind of information to store, instead of probing memory capacity.\n- Presentation:\n  The presentation of the environments is somewhat confusing and often lacks some context.\n  It is not clear why some environment variants have different action spaces, diagonal movements would still be possible with the multi-discrete action space.\n  Fig. 1 is very confusing: Why is the \"Next Target\" in Fig 1a) to the left while the command points to the right? Shouldn't the command indicate the direction towards the next target?\n  It is not clear how the number of steps in Mortar Mayhem are computed.\n  It is not clear how the movement of the agent is defined in the non-grid versions. \n  Some unintuitive design choices were made without justification, e.g. negative feedback is only given visually instead of negative reward.\n  From visual feedback the agent could never learn that e.g. Spotlights are actually bad as there is no negative reward for losing health.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster