Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors point out drawbacks of current benchmarks that claim to evaluate memory in RL.
They propose a novel benchmark suite for evaluating the memory component of agents trained in partially observable environments.
The benchmark consists of three novel environments that vary in their difficulty and can be scaled to much harder difficulty.
Baseline results indicate that two of the three environments are solvable by a recurrent agent, while the third is not solvable.
The authors identify the reason for that as vulnerability of recurrence to random perturbations that are part of the environment.
strength_and_weaknesses: ### Strengths:

- Potential relevance to the field:
  As the authors correctly point out, there is a variety of methods available, but no go-to benchmark to properly compare memory-based agents.
  Moreover, the memory dependence of some existing memory benchmarks is questionable.
- Reproducibility:
  Codebase is made publicly available.

### Weaknesses:

- Novelty:
  Mortar Mayhem is almost equivalent to the publicly available DM Ballet environment [1].
  Other existing benchmarks fulfill the same requirements of their desiderata as Memory-Gym, sometimes even with higher FPS, but no justification is given on why Memory-Gym should be the preferred choice.
  The memory dependency of some environments is questionable, i.e. PPO can exploit visual artifacts in Mortar Mayhem and Mystery Path, while being the best method on Searing Spotlights.
- Relevance of proposed benchmark:
  There are other environments that fulfill every point on the proposed desiderata, but it is not clear what advantage Memory-Gym has over them.
  Two of the three proposed environments are solved already within a moderate amount of training steps with a recurrent agent.
  The memory dependence of Mystery Path is questionable, since it can be solved in ~50% of the cases by a markovian policy as shown in Fig4b).
  The difficulty of Searing Spotlights stems from visual artifacts (random spotlights) rather than required memory, rendering it unsuitable for evaluating memory capabilities of an agent as proposed.
  Furthermore, difficulty scaling is achieved either through increasing required memory capacity (longer sequence of commands in MM, scaling grid in MP, etc.), or by increasing the complexity of the action space (continuous instead of grid-like movement).
  However, the authors state explicitly that evaluating the memory capacity can be done more effectively by other disciplines such as supervised learning, which contradicts their scaling approaches.
  Varying the time intervals between commands, in MM, however, seems to be a valid approach to scale the hardness with respect to what kind of information to store, instead of probing memory capacity.
- Presentation:
  The presentation of the environments is somewhat confusing and often lacks some context.
  It is not clear why some environment variants have different action spaces, diagonal movements would still be possible with the multi-discrete action space.
  Fig. 1 is very confusing: Why is the "Next Target" in Fig 1a) to the left while the command points to the right? Shouldn't the command indicate the direction towards the next target?
  It is not clear how the number of steps in Mortar Mayhem are computed.
  It is not clear how the movement of the agent is defined in the non-grid versions. 
  Some unintuitive design choices were made without justification, e.g. negative feedback is only given visually instead of negative reward.
  From visual feedback the agent could never learn that e.g. Spotlights are actually bad as there is no negative reward for losing health.
clarity,_quality,_novelty_and_reproducibility: ### Major Points

- Proper baseline comparison:
  Framestacking is a simple baseline that should be included to highlight long-term memory dependence of the environments. 
  This ensures no other short-term aspects of the environment can be exploited and strengthens the claim for long-term dependencies.
  Codebases of other state-of-the-art approaches are actually publicly available:
 
  Several open-source implementations of GTrXL are available:
  https://github.com/opendilab/DI-engine
  https://github.com/dhruvramani/Transformers-RL
 
  Codebase for HCAM:
  https://github.com/deepmind/deepmind-research/tree/master/hierarchical_transformer_memory

- Unclear presentation of results:
  The presentation of results conflates generalization and memory capability, making it difficult to interpret results.
  It would be much easier to follow if those two effects would be disentangled, i.e. evaluate memory capabilities on the entire level distribution and generalization on a train/test split of seeds.
  Further, the number of training seeds are not mentioned in the main paper, however this information is crucial to interpreting generalization results and the apparent gap.
  A helpful analysis here would be to show the generalization gap over different number of training seeds as in [2].
  Also, why are only 10 novel seeds chosen for evaluation, but repeated 5 times?
  It would be much more intuitive to evaluate on a higher number of seeds to get a better performance estimate.
  The variance of the Random baseline should be included in all plots.
  Is Figure 4c actually correct?
  How can it be that PPO reaches performance on-par with a random policy, but generalizes on novel seeds?
  If the plot is actually correct it raises the question whether this environment is really dependent on memory, since PPO is the best performing method.
  The authors mention performance of GRU-PPO and PPO for their environment ablation in Fig 6a, but do not show any learning curves.
  These learning curves would be much more important for interpreting results than the performance on Fig 6b.

- Interpretation of results:
  There is a huge generalization gap for PPO on MPGrid, while it solves the task in 50% of the cases during training.
  Since the goal is always placed on the opposing side of the origin and the solution is mostly the shortest path then this gap from training to test seeds is difficult to explain.
  Why does the performance of GRU-PPO stagnate when going from a per-tile to continuous movement of the agent?
  A possible reason for that is limited exploration, and could be alleviated by running a proper hyperparameter search for each method on each environment.
  Adding additional rewards, as in MPDense, alters the optimization problem and the optimal policy may change.
  If the immediate reward overshadows the reward of reaching the goal tile, the agent may learn to exploit the immediate reward only and will not perform well on novel seeds.
  Since GRU-PPO has shown performance lower than random on Searing Spotlights, it did not learn anything meaningful.
  Not only is GRU-PPO incapable of solving Searing Spotlights, but all baseline methods are.
  The authors identify the reason for that as vulnerability of GRU-PPO to spotlight perturbations, but another reason could simply be suboptimal hyperparameters.
  If recurrence is the sole reason, HELM should capable of solving the environment, right?

- Computational resources were allocated to scale the hardness of the environments:
  Baselines should be tuned on the respective environments before scaling the hardness, otherwise it is not clear whether bad performance is caused by inappropriate hyperparameters.

### Minor Points:

- The authors claim that "if an environment is solvable to some extent using an agent without memory, it is not easy to differentiate whether the memory mechanism is working"
  This is not true, it shows that a Markovian policy can exploit structure in the environment without relying on memory.
  Whether a memory mechanism is working is ill-defined here, how is it defined that a memory mechanism "works"?
  If a recurrent agent significantly outperforms a markovian one, it indicates that the addition of memory alters the optimization problem such that the task is easier to solve.

- The authors claim that DM Alchemy, Crafter, or ObstacleTower require non-trivial additional components.
  What are those?
  Why can't GRU-PPO simply be applied to those?
  
- Why is open-sourceness explicitly stated as requirement?
  Is it not sufficient to access a benchmark via an API without access to the source code?
  Obviously open-sourceness should be a requirement for published algorithms and novel methods, but is it required for benchmarks as well?
  Further, why is headless a requirement?
  Why would HPC facilities not support dependencies such as xvfb, or EGL?

- How can an environment be turned into a supervised learning problem?

- What does it mean that an "agent is left in uncertainty"?

- The authors write "A single observation leaves the agent clueless about which of the nine tiles to move next" in MM.
  This is not true, in fact, the last action can be inferred by the direction the agent is facing (the eyes in Fig 1), also the performance of PPO in Fig 4a indicates otherwise.
  PPO appears to be capable of exploiting this information to reach a score > 2 during training.
  
- What are differences between pits and walls in Mystery Path?

- The authors claim that mean cumulative reward is not an appropriate measure for task completion, why?
  A task is defined by its reward function and maximum reward corresponds to completion of a task.
  In the miner example the task is defined by collecting all diamonds and navigating to the exit afterwards, thus, if maximum reward is not achieved the level is not completed.
  Another claim: "Heist and Maze can be of strong dependence on memory, but only if tiny levels, which memory-less agents trivially solve, are excluded"
  Why does the appearance of easier levels in the level distribution alleviate the need for memory on harder ones?

- What input information do the different baselines receive? only observations or actions as well? what is the vector observation for the different environments?

- Has HELM been properly tuned? The appendix only mentions tuning of the beta parameter with no influence on performance. What about other hyperparameters such as learning rate?

- Phrases such as "MMAct needs twice as long to solve only five commands" should be rephrased to "GRU-PPO needs twice as long...", environments do neither converge nor solve a task.

- How is the "uncertainty of the environment" defined? Is it measured somewhere?

- Why are training and test seeds for MM shown in one plot in Fig 5a, but in separate plots for MP in Fig 5b and 5c?

- The authors claim that MiniGrid-Memory [3], Spot the Difference [4], or DM Ballet [1] are not well suited for evaluating memory.
  In the next sentence, they confirm that all of these environments require an agent to memorize cues in the very beginning of an episode, thus they can be used to evaluate the memory capability of an agent.
 The authors follow up on that by "Once this cue is memorized there is no need to manipulate the agent's memory further." and that it is sufficient to match extracted features to goal cues.
 In fact, most of the proposed environments in Memory-Gym don't require an agent to "manipulate" its memory, but rather match the features of the current observation with the features it has extracted during observing the sequence of actions in Mortar Mayhem, or the position of coin and exit in Searing Spotlights.
 It is not even clearly defined what "manipulating" an agent's memory means.

 
 [1] Andrew Kyle Lampinen, Stephanie C. Y. Chan, Andrea Banino, and Felix Hill. Towards mental time travel: a hierarchical memory for reinforcement learning agents, NeurIPS 2021.
 [2] Karl Cobbe, Christopher Hesse, Jacob Hilton, John Schulman, Leveraging Procedural Generation to Benchmark Reinforcement Learning, ICML 2020.
 [3] Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym, Github 2018.
 [4] Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adria Puigdomenech Badia, Gavin Buttimore, Charles Deck, Joel Z Leibo, and Charles Blundell. Generalization of reinforcemnt learners with working and episodic memory, NeurIPS 2019.
summary_of_the_review: The paper addresses an important issue in the field of partial observability in RL, namely to introduce a benchmark for evaluating memory capabilities of novel and existing methods.
Despite a well-structured desiderata, the authors fail to clarify the benefit of their proposed benchmark over existing ones.
Further, the design choices of the different environments in the benchmark are very confusing and not always justified or even contradictive.
The provided baseline results are difficult to interpret and not very convincing, making it difficult for new methods to reference these results.
They also indicate that some environments are exploitable by a markovian policy to a certain degree.
Finally, both environments and baseline results are presented poorly and would require a major revision in order to make the proposed benchmark usable by the RL community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: As a standard benchmark for POMDP tasks is desired due to rising research interests, the paper proposed 3 new Gym tasks which are memory-dependent POMDPs. The key features of the environments are (1) "strongly depend on memory".  (2) different levels of difficulty. (3) good meta properties (Table 1). 

The authors conducted baseline experiments using PPO as RL algorithm and GRU for history extraction and compared with PPO + feedforward NN and HELM. It can be seen that the tasks cannot be solved without memory. The results also show various difficulty levels of the tasks even using PPO+GRU. Therefore, the authors conclude that the proposed tasks yield a new benchmark for memory-dependent POMDP research.







strength_and_weaknesses: **Strength**
- The being addressed problem is important, that is, the lack of a standard benchmark for POMDP
- The proposed environments have good properties (Table 1)
- The proposed tasks are interpretable and can be customized according to the research demand
- The proposed tasks can test model's capacity of generalization
- The authors conducted experiments to support their claims


**Limitations**
- The task scope is limited to 3 Atari-like games with 2D vision as input. There is no environment with continuous action space. However, due to the large diversity of POMDP research, e.g., robot control and multi-modality observations, the proposed framework still has not solved the lack of a standard POMDP benchmark.
- The experiments can be improved. Some of possibly useful trials are as follows: (1) Testing feedforward NN with $N$-steps observations as input. The result of how performance changes with $N$ can provide clues of how much long-term memory (vs. short-term memory) is required. (2) Testing some off-policy and probably more powerful algorithm of memory-dependent RL such as R2D2 (https://openreview.net/forum?id=r1lyTjAqYX) and ACER (https://arxiv.org/abs/1611.01224v2). 






clarity,_quality,_novelty_and_reproducibility: The paper is basically written well, and the ideas are clearly addressed. There is no new algorithm/model proposed, but a new task set for POMDP studies. The code will be open source.


summary_of_the_review: The paper proposed a new task set for memory-dependent POMDP studies. The tasks have many desired properties, and the experiments proved that memory is necessary to solve them. Meanwhile, the task scope is limited, and the experiments could be more comprehensive and convincible by testing more algorithms and models. In sum, my recommendation is a borderline reject.

-------- Updated after rebuttal -------

See my comments below
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper outlines some desiderata for memory tasks for RL, and proposes/releases a set of three varied benchmark tasks that meet these desiderata. The paper runs a variety of baselines on the tasks and shows that they have nice properties such as strong memory dependence and scalable difficulty. One task, searing spotlights, leads to results that are particularly interesting from a scientific perspective.
strength_and_weaknesses: Strengths:
* Presentation is very clear.
* Tasks are interesting and deliver distinct  memory challenges.
* Scalable difficulty is a very nice attribute for future research.
* Results with spotlights + recurrence are fascinating. 

Weaknesses:
* If possible, it would be nice to include more baselines from different algorithmic families, such as recurrent IMPALA or VMPO (used in the Fortunato et al. and Parisotto et al. papers, IIRC—of course the latter uses transformers, but VMPO does not require them). 

clarity,_quality,_novelty_and_reproducibility: The writing and presentation is extremely clear, kudos. 

The quality seems overall high.

The novelty of the benchmark is moderate given the existing work, but I think the authors do a decent job of describing the unique value of their benchmark relative to existing tasks. And the results with the searing spotlights are interesting scientifically, which boost the value a bit.

summary_of_the_review: I think this paper offers an interesting new benchmark for memory tasks, and some independently interesting results. The authors do a decent job of motivating their work relative to prior work. There could always be more baselines, etc. but overall I find the paper to be a useful contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a benchmark to solely focus on benchmarking Deep Reinforcement Learning agents' memory and ability to generalise. In addition to open source accessibility, simulation step speed and flexibility (level hardness, noise etc) the authors claim their benchmark is better suited to distinctly evaluate the role of memory in DRL performance. They show the need for RNN implementations of PPO taking learning from sequences to successfully complete (2/3) of the games where memory-less implementations fail.
strength_and_weaknesses: Strengths:
The work tackles a very important problem that is of great interest to the Recurrent Neural Network (RNN) community, which is long term dependency learning, and large sequence memorisation, in the context of memory requiring Deep Reinforcement Learning (DRL) tasks.
The authors put notable effort into ensuring their environments require sequence memorisation capabilities to be successfully completed. They also clearly state a case for the need for such a benchmark among the plethora of available environments to specifically tackle this issue. Indeed:
- Code is well written and simple enough to use.
- The desiderata is met, especially in terms of enforcing a dependency between success and long term memory decision making.
- Difficulty scaling for the games enables benchmarking memory capabilities of simplistic models as well as larger architectures.

Weaknesses:
The paper presenting a new benchmark, it is naturally void of any technical novelty. However, it is unfortunate that it also lacks technical precision for many of the concepts manipulated:
- The word memory itself, which gives the benchmark its whole purpose, is used 123 times throughout the paper without being clearly defined, or more specifically, put into context. It is evident that the authors refer to maintaining a representation of information from long sequence of past grids and its use for future decisions, however one has to refer to Fig 14 in appendix D on page 20 of the manuscript for the technical representation of the recurrent neural network block in the DRL architechture.
- Thus, a clearer presentation of memory mechanisms in RNNs and their relationship to DRL algorithms would be much welcome to elucidate what exactly is being evaluated by the benchmark. With a plethora of  advances on RNN memory capabilities for sequence modelling (see works such as "Efficiently Modeling Long Sequences with Structured State Spaces" by Gu et al, and "Liquid Structural State-Space Models" by Hasani, Lechner et al), any performance evaluation should be based on a quantitative measuring of the dimension and sequence lengths of information required to achieve the task against the memorisation capabilities of the RNN network used in the PPO architecture (or any other DRL method relying on RNNs for memory).
- With memory capabilities of state-of-the-art RNNs allowing successful learning of Long Range Arena tasks such as Path-X (16384 sequence length), it would be interesting to have a more extensive quantitative discussion about how increasing difficulty levels of the games affects the dimensions of the sequences that need to be memorised. This would justify why this benchmark can be useful in the future as we develop increasingly memory capable architectures.
- Though not at the core of the work, the authors only test GRU and LSTM architectures. Future directions evoke implementing more elaborate models. However, it would have been interesting to see better adapted architectures used especially for the Searing Spotlights environment.
clarity,_quality,_novelty_and_reproducibility: - The quality of the work is good (especially the code implementation). The manuscript is clear in making a case for this benchmark

- The work being a benchmark, it does not offer technical novelty. Its usefulness for the community is unclear or at least not clearly demonstrated given the current state of the work (see discussion on memory, sequence lengths and RNNs above).

- The manuscript lacks clarity in presentation and depth in defining concepts that are being manipulated. For example, all performance metrics presented (fig 4,5,6c,15,16) are IQM (commands, success rates, returns) which are used without any definition (the acronym itself appears without any prior presentation).

- The videos, provided in batch form without any explanation, do not help the reviewer to appreciate the intricacies of the work or acquire a clearer understanding of what is of interest in what is being shown.


summary_of_the_review: The work is of high relevance for benchmarking a very important aspect of learning which is the capability of networks and in this case reinforcement learning agents to achieve tasks requiring long sequence memorisation. The manuscript itself is generally well written, but lacks some clarity and rigour in manipulating core ideas to the case it is making. With additional elucidation of links between the proposed tasks and the quantitative increase in memorisation required for solving them, this work could prove to be suitable for benchmarking models not only today, but into the future as models improve. This potential is still vague in the current state of the work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a novel benchmark called Memory GYM to evaluate memory-based agents. 

This paper has raised a significant amount of discussions among the authors, reviewers, and the AC. While there are mixed opinions about the usefulness of the proposed benchmark. After the discussion between the reviewers and AC, I vote for the acceptance of the paper conditional to the below points to be addressed. I tend to accept the paper mainly due to its benefit for academic labs with limited compute capacity. The benchmark enables practitioners to test various models in a contained set of tasks which could enable new ideas from small labs.

Major Concerns to be addressed:

- Addressing detailed comments of Reviewer e7qG: Reviewer e7qG provided great insights and advice on matters that must be addressed in your potential camera-ready version. In particular, a summary of the discussions you had during the discussion period must be reflected in the conclusions, discussions, scope, and limitation sections of your paper.
- Addressing the remaining concerns of Reviewer Arip: In particular, authors must tone down the claims throughout the paper and precisely reflect exactly what challenges (discrete spaces + sequential decision-making) have been explored. This must be reflected in the Abstract, intro, and the rest of the paper. 
- The related works suggested by Reviewer aYs5, on candidate models that could be used, should be included in the revised version.
justification_for_why_not_higher_score: There are some reviewers that are on the fence about the acceptance of the paper. Therefore, I could not give a better score.
justification_for_why_not_lower_score: The work has many benefits for researchers from small labs.
Decision: Accept: poster