Evaluating Long-Term Memory in 3D Mazes
Authors: Jurgis Pašukonis
Timothy P Lillicrap
Danijar Hafner
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement Learning
Memory
Benchmark
Dataset
Representation Learning

Abstract: Intelligent agents need to remember salient information to reason in partially-observed environments. For example, agents with a first-person view should remember the positions of relevant objects even if they go out of view. Similarly, to effectively navigate through rooms agents need to remember the floor plan of how rooms are connected. However, most benchmark tasks in reinforcement learning do not test long-term memory in agents, slowing down progress in this important research direction. In this paper, we introduce the Memory Maze, a 3D domain of randomized mazes specifically designed for evaluating long-term memory in agents. Unlike existing benchmarks, Memory Maze measures long-term memory separate from confounding agent abilities and requires the agent to localize itself by integrating information over time. With Memory Maze, we propose an online reinforcement learning benchmark, a diverse offline dataset, and an offline probing evaluation. Recording a human player establishes a strong baseline and verifies the need to build up and retain memories, which is reflected in their gradually increasing rewards within each episode. We find that current algorithms benefit from training with truncated backpropagation through time and succeed on small mazes, but fall short of human performance on the large mazes, leaving room for future algorithmic designs to be evaluated on the Memory Maze.

Published as a conference paper at ICLR 2023
EVALUATING LONG-TERM MEMORY IN 3D MAZES
Jurgis Pasukonis
Timothy Lillicrap
Danijar Hafner
DeepMind
DeepMind
DeepMind
Verses Research Lab
University College London
University of Toronto
ABSTRACT
Intelligent agents need to remember salient information reason in partially- observed environments. For example, agents with a first-person view should remember the positions of relevant objects even if they go out of view. Similarly, effectively navigate through rooms agents need to remember the floor plan how rooms connected. However, most benchmark tasks in reinforcement learning do not test long-term memory in agents, slowing down progress in this important research direction. In this paper, introduce the Memory Maze, 3D domain of randomized mazes specifically designed for evaluating long term memory in agents. Unlike existing benchmarks. Memory Maze measures long-ter serg-termmmm memory separate from confounding agent abilities and requires the agent to localize itself by integrating information over time. With Memory Maze, we propose an online reinforcement learning benchmark, a diverse offline dataset, and an offline probing evaluation. Recording human player establishes a strong baseline and verifies the need to build up and retain memories, which is reflected in their gradually increasing rewards within each episode. We find that current algorithms benefit from training with truncated backpropagation through time and succeed small mazes, but fall short of human performance on the large mazes, leaving room for future algorithmic designs to be evaluated the Memory Maze. Videos available the website: https:/ / / /g thub. com/ jurgi sp htt////mmomrmmmme
INTRODUCTION
Deep reinforcement learning (RL) has made tremendous progress in recent years, outperforming humans on Atari games (Mnih et al., 2015; Badia al., 2020), board games (Silver et al., 2016; Schrittwieser et al., 2019), and advances in robot learning (Akkaya et al., 2019; Wu et al., 2022). Much of this progress has been driven by the availability of challenging benchmarks that are easy to use and allow for standardized comparison (Bellemare al., 2013; Tassa et al., 2018; Cobbe et al., 2020). What is more, the RL algorithms developed on these benchmarks are often general enough
Agent Inputs
Underlying Trajectory
1=0
30
60
90
120
150
Figure 1: The first 150 time steps of an episode in the Memory Maze 9x9 environment. The bottom row shows the top-down view of randomly generated maze with colored objects. The agent only observes the first-person view (top row) which includes a prompt for the next object to find border of the corresponding color. The agent receives +1 reward when it reaches the object of the prompted color. During the episode, the agent has to visit the same objects multiple times, testing its ability to memorize their positions, the way the rooms are connected, and its own location.
"Coreesonding author: Jurgis Pasukonis ejurgisp gmail.com>

---

Published as a conference paper at ICLR 2023
to solve completely unrelated challenges, such finetuning large language models from human preferences (Ziegler et al., 2019), optimizing video compression parameters (Mandhane et al., 2022), or promising results in controlling the plasma of nuclear fusion reactors (Degrave et al., 2022).
Despite the progress RI., many current algorithms are still limited to environments that are mostly fully observed and struggle in partially-observed scenarios where the agent needs to integrate and retain information over many time steps (Wayne et al., 2018). Despite this, the ability to remember over long time horizons is central aspect of human intelligence and major limitation applicability of current algorithms. While many existing benchmarks are partially observable to some extent, memory is rarely the limiting factor of agent performance (Oh et al., 2015; Cobbe et al., 2020; Beattie et al., 2016; Hafner, 2021). Instead, these benchmarks evaluate a wide range of skills at once, making it challenging measure improvements in an agent's ability remember.
the
Ideally, we would like a memory benchmark to fulfill the following requirements: (1) isolate the challenge of long-term memory from confounding challenges such as exploration and credit assignment, so that performance improvements can be attributed to better memory. (2) The tasks should challenge average human player but be solvable for them, giving estimate of how far current algorithms are away from human memory abilities. (3) The task requires remembering multiple pieces of information rather than single bit or position, e.g. whether go left or right at the end of a long corridor. (4) The benchmark should be open source and easy to use.
We introduce the Memory Maze, a benchmark platform for evaluating long-term memory in RL agents and sequence models. The Memory Maze features randomized 3D mazes in which the agent is tasked with repeatedly navigating one of the multiple objects. To find the objects quickly, the agent has to remember their locations, the wall layout of the maze, as well as its own location. The contributions of this paper are summarized as follows:
Environment
We introduce the Memory Maze environment, which is specifically designed to measure memory isolated from other challenges and overcomes the limitations of existing benchmarks. We open source the environment and make it easy install and use.
Human Performance We record the performance of human player and find that the benchmark challenging but solvable for them. This offers an estimate of how far current algorithms are from the memory ability of a human.
Memory Challenge We confirm that memory is indeed the leading challenge in this benchmark. by observing that the rewards of the human player increases within each episode, as well as by finding strong improvements of training agents with truncated backpropagation through time.
Offline Dataset We collect diverse offline dataset that includes semantic information, such as the town view, object positions, and the wall layout. This enables offline RL as well as evaluating representations through probing of both task-specific and task task-agnostic information.
Baseline Scores We benchmark a strong model-free and model-based agent on the four sizes of the Memory Maze and find that they make progress on the smaller mazes but lag far behind human performance on the larger mazes, showing that the henchmark of appropriate difficulty.
2 RELATED WORK
Several benchmarks for measuring memory abilities have been proposed. This section summarizes important examples and discusses the limitations that motivated the design of the Memory Maze.
DMLab (Beattie et al., 2016) features various tasks, some of which require memory among other challenges. Parisotto et al. (2020) identified subset of 8 DMLab tasks relating to memory but these tasks have largely been solved by R2D2 and IMPALA (see Figure 11 in Kapturowski al. (2018)). Moreover, DMLab features a skyline in the background that makes it trivial for the agent to localize itself, so the agent does not need to remember its location in the maze. (Wayne et al., 2018) used larger battery of tasks, but only a subset of those was included in the public release of DMLab.
SimCore (Gregor et al., 2019) studied the memory abilities of agents by probing representations and compared a range of agent objectives and memory mechanisms, an approach that we build upon in this paper. However, their datasets and implementations were not released, making it difficult for the research community to build upon the work. A standardized probe benchmark is available for Atari (Anand et al., 2019), but those tasks require almost no memory.
2

---

Published as a conference paper at ICLR 2023
Memory 9x9
Memory 11x11
Memory 13x13
Memory 15x 15
Figure 2: Examples of randomly generated Memory Maze layouts of the four sizes.
DM Memory Suite (Fortunato et al., 2019) consists existing DMLab tasks and 7 variations of T-Maze and Watermaze tasks implemented in the Unity game engine, which neccessitates interfacing with a provided Docker container via networking. These tasks pose an exploration challenge due the initialization far away from the goal, creating confounding factor in agent performance. Moreover, the tasks tend to require only small memory capacity, namely bit for T-Mazes and 1 coordinate for Watermazes.
THE MEMORY MAZE
Memory Maze is domain of randomized mazes specifically designed for evaluating the long- term memory abilities of RL agents. The agent navigates the maze using first-person camera observations of the 3D scene while moving on a 2D planar layout. Memory Maze isolates long-term memory from confounding agent abilities, such as exploration, and requires remembering several pieces of information: the positions of objects, the wall layout, and the agent's position. This section introduces three aspects of the benchmark: (1) an online reinforcement learning environment with four tasks, (2) offline dataset, and (3) a protocol for evaluating representations this dataset by probing.
3.1 ENVIRONMENT
The Memory Maze environment is implemented using MuJoCo (Todorov et al., 2012) as the physics and graphics engine and the dm_control (Tunyasuvunakoo) et al., 2020) library for building RL environments. The environment can be installed as a pip package nemory-maze code, available on the project website There are four Memory Maze tasks with varying sizes and difficulty: Memory 9x9, Memory 1 11x11, Memory 13x13, and Memory 15x15.
from the source
The task is inspired by a game known as scavenger hunt or treasure hunt. The agent starts randomly generated maze containing several objects of different colors. The agent is prompted to find the target object specific color, indicated by the border color in the observation image. Once the agent finds and touches the correct object, it gets a +1 reward, and the next random object is chosen as a target. If the agent touches the object of the wrong color, there is no effect. Throughout the episode, the maze layout and the locations of the objects do not change. The episode continues for a fixed amount of time, so the total episode return equal to the number of targets the agent can find in the given time. See Figure 1 for illustration.
The episode return is inversely proportional to the average time takes for agent to locate target objects. If the agent remembers the location of the prompted object and how the rooms are connected, the agent can take the shortest path to the object and thus reach it quickly. On the other hand, an agent without memory cannot remember the object position and wall layout and thus has to randomly explore the maze until it sees the requested object, resulting in several times longer duration. Thus, the layout, particularly object locations and paths to them.
the Memory Maze tasks correlates with the ability remember the maze
Memory Maze sidesteps the hard exploration problem present in many T-Maze and Watermaze tasks. Due to the random maze layout in each episode, the agent will sometimes spawn close to the object of the prompted color and casily collect the reward. This allows the agent to quickly bootstrap to a policy that navigates to the target object once visible, and from that point, it can improve by developing memory. This makes training much faster compared to, for example, the "Spot the
htttps://githbb.com///// wwwpp////mrr/aam.ame.eeee.eaeeam
3

---

Published as a conference paper at ICLR 2023
Memory 9x9
Memory 11x11
Memory 13x13
Memory 15x15
Parameter
Number of objects
3
4
5
6
Number of rooms
3
- 6
5-
Room size
3
3 - 5
3-5
3
Episode length (steps at 4Hz)
1000
2000
3000
4000
Mean maximum score (oracle)
34.8
58.0
74.5
87.7
Table I: Memory Maze environment details.
Difference" set of tasks in DM Memory Suite that require the agent any reward signal.
long corridor receive
The sizes are designed such that the Memory 15x15 enviromment is challenging for human player and out of reach for state-of-te ooffthe-art RL algorithms, whereas Memory 9x9 is easy for human player and solvable with RL, with 1xx1 and 13x13 as intermediate stepping stones. See Table I for details and Figure 2 for an illustration.
3.2 OFFLINE DATASET
We collect diverse offline dataset of recorded experience from the Memory Maze environments. This dataset is used the present work for the offline probing benchmark and also enables other applications, such offline RL.
We release datasets: Memory Maze 9x9 (30M) and Memory Maze 15x15 (30M). Each dataset contains 30 thousand trajectories from Memory Maze 9x9 and 15x15 environments respectively, single trajectory is 1000 steps long, even for the larger maze increase the diversity of mazes included while keeping the download size small. The datasets are split into 29k trajectories for training and 1k for evaluation.
The data is generated by running a scripted policy on the corresponding environment. The policy uses MPC planner (Richards, 2005) that performs wreadth-first-seacee chosen points in the maze under action noise. This choice of policy was made to generate diverse trajectories that explore the maze effectively and that form loops in space, which can important for learning long-term memory. We intentionally avoiding recording data with a trained agent to ensure diverse data distribution (Yarats al., 2022) and to avoid dataset bias that could favor some methods over others.
navigate randomly
The trajectories include not only the information visible the agent first- image observations, actions, rewards but also additional semantic information about the environment, including the maze layout, agent position, and the object locations. The details of the data keys are in Table 2.
3.3 OFFLINE PROBING
Unsupervised representation learning aims to learn representations that can later be used for down- stream tasks of interest. In the context of partially observable environments, vised representations to summarize the history of observations into a representation that contains information about the state of the environment beyond what is visible in the current observation by remembering salient information about the environment. Unsupervised representations are commonly evaluated by probing (Oord al., 2018: 8: Chen et al., 2020; Gregor et al., 2019; Anand al., 2019), where separate network is trained to predict relevant properties from the frozen representations.
would like unsuper-
We introduce the following four Memory Maze offline probing benchmarks: Memory 9x9 Walls, Memory 15x15 Walls, Memory 9x9 Objects, and Memory 15x15 Objects. These are based on either using the maze wall layout (maze layout) agent-centric object locations (targets. setts__vee vec) as the probe prediction target, trained and evaluated on either Memory Maze 9x9 (30M) or Memory Maze 15x15 (30M) offline datasets.
The evaluation procedure is as follows. First, a sequence representation model (which may be component of model-based RL agent) is trained on the offline dataset with semi-supervised loss based on the
image observations conditioned by actions. Then separate probe network is trained to predict the probe observation (either maze wall layout agent-centric object locations) from the internal state of the model. Crucially, the gradients from the probe network are not propagated into the model, it only learns to decode the information already present in the
4

---

Published as a conference paper at ICLR 2023
Key
Shape
Type
Description
image
(64, 64,3)
uint8
First-person view observation
action
(6)
binary float
Last action, one-hot encoded
reward
Last reward
Laze_layout (9,9) (15, 15)
binary
Maze layout (wall no wall)
agent_pos
(2)
float
Agent position in global coordinates
agent_dir
(2)
float
unit vector Object locations in global coordinates
Agent orientation
targets_pos
(6,2)
float
targets_vec
(6,2)
float
Object locations in agent-centric coordinates
target_pos
(2)
float
Current target object location, global
target_vee vec
(2)
float
Current target object location, agent centric
target_colon
(3)
float
Current target object color RGB
Table 2: Entries in the offline dataset. The tensors are saved as NPZ files with an additional time step, e.g. image tensor for 1000-step long trajectory is (1001, 64, 64. 3). The first element is the image before the first action and the last element is the image after the last action.
internal state, but it does not drive the representation. Finally, the predictions of the probe network are evaluated on the hold-out dataset. When predicting the wall layout, the evaluation metric is prediction accuracy, averaged across all tiles of the layout. When predicting the object locations, the evaluation metric is the root-mean-squared error (RMSE), averaged over the objects. The final score calculated by averaging the evaluation metric over the second half (500 steps) of each trajectory in the evaluation dataset. This is done to remove the initial exploratory part of each trajectory. during which the model has no way of knowing the full layout of the maze (see Figure C.I). We make this choice so that a model with perfect memory could reach 0.0 RMSE on the Objects benchmark and 100% accuracy on the Walls benchmark.
The architecture of the probe network defined as part of the benchmark to ensure comparability: it is MLP with hidden layers, 1024 units each, with layer normalization and ELU activation after each layer (see Table E.3). The input to the probe network the representation of the model which should be ID vector of length 2048 concatenated with the position and orientation of the agent. The agent coordinate is provided as additional input because the wall layout prediction target presented in the global grid coordinate system, which the agent has no way of knowing from the first-person observations. The probe network is trained with BCE loss when the output is wall layout, and with MSE loss when the output is object coordinates.
We found linear probe to not be suitable for our study because is not powerful enough to extract the desired features (such as the wall layout) from the recurrent state. There is potential concern that a powerful enough probe network could extract any desired information about the history of observations from the recurrent state, which would invalidate the logic of the test. We provide evidence in Appendix that this is not the case, and the choice of the 4-layer network is justified.
4 ONLINE EXPERIMENTS
In this section, we present online RL benchmark results on the four Memory Maze tasks (9x9, 11x11. 13x13, 15x15) introduced in Section 3.1. We have evaluated the following baselines, including strong model-based and model-free RL algorithm each:
Human Player The data was collected from one person who had experience playing first-person computer games but had not previously seen the Memory Maze task. The player first played several episodes familiarize themselves with the task and GUI. They then played the recorded episodes, for which they were instructed to concentrate and strive for the maximum score. We recorded ten episodes for each maze size.
Dreamer For model-based RL baseline we evaluated the DreamerV2 agent (Hafner et al., 2020). We mostly used the default parameters from the Atari agent in (Hafner et al., 2020) but increased the RSSM recurrent state size and tuned the KL scale and entropy scale. The same parameters were used across all four tasks. We increased the speed data generation (environment steps per update) to make the training faster in wall clock time, some expense to the sample
5

---

Published as a conference paper at ICLR 2023
Memory 9x9
Memory 11x11
Memory 13x13
Memory 15x15
60
30
60
75
Retum
40
20
40
50
20
10
20
Dreamer
Dreamer (TBTT)
IMPALA
Human
Oracle
Figure 3: Online RL benchmark results after 100M environment steps of training. Error bars show the standard deviation over five runs. We find that current algorithms benefit from training with truncated backpropagation through time and succeed on small mazes, but fall short of human performance the large mazes, leaving room for future algorithmic designs to be evaluated on the Memory Maze.
Memory 9x9
Memory 11x11
Memory 13x13
Memory 15x15
Method
Oracle
34.8
58.0
74.5
87.7
Human
26.4+1.6
44.342.5
55.543.5
67.7
IMPALA
23.440.4
28.210.3
27.540.6
17.7
Dreamer
28.2
20.4
11.640.7
4.1
Dreamer (TBTT)
33.2 33.2
31.443.1
21.4+3.0
5.6
Table 3: Online RL benchmark results. The RL agents were trained for 100M steps, and the reported scores are averaged over five runs, with the standard deviation indicated as subscript. Human score an average of 10 episodes, with the bootstrapped standard error of the mean shown subscript.
efficiency. We trained Dreamer on 8 parallel environments. The full hyperparameters are listed in Table E.1.
Dreamer (TBTT) Truncated backpropagation through time (TBTT) known to an effective method for training RNNs to preserve information time. For example, R2D2 agent (Kaptur- owski et al., 2018), when trained with the stored state, shows significant improvement on DMLab tasks compared to zero state initialization. Original Dreamer is trained with zero input state each batch, which may limit its ability to form long-term memories. To test this, we implement TBTT training in Dreamer by replaying complete trajectories sequentially and passing the RSSM state from one batch to the next. We alleviate the potential problem of correlated batches by forming each T B batch from B different episodes, and we start replaying the first episode from random offset to avoid synchronized episode starts.
IMPALA We choose IMPALA (V-trace) as strong model-free baseline, which performs well on DMLab-30 (Espeholt et al., 2018) and has reliable implementation available in SEED RL (Espeholt et al., 2019). The hyperparameters used in our experiments are shown in Table E.2. We tuned the entropy loss scale, learning rate and Adam epsilon. We also tried increasing the LSTM size, but it did not improve performance. The efficient implementation allowed us to use 128 parallel environments, which was important for performance achieved on the larger mazes.
Oracle We establish an upper bound the task scores by training an oracle agent that observes complete information about the maze layout and follows the shortest path to the target object. Note that no real agent relying on first-person observations can achieve this upper bound score because the oracle receives the maze layout as input from the beginning of the episode without having to explore over time. However, we can estimate that this initial exploration, if done efficiently, should not take more steps than reaching few targets. So the maximum achievable score for agents is within few points of the Oracle upper bound.
In the future, would be interesting to evaluate additional baselines on Memory Maze, such as MRA (Fortunato et al., 2019) that uses episodic memory and GTrXL (Parisotto et al., 2020) that uses transformer architecture. Unfortunately, the source code of these agents is not publicly available, so were unable include them in comparison.

---

Published as a conference paper at ICLR 2023
Memory 9x9
Memory 15x15
Memory 9x9
Memory 15x15
Walls
Walls
Objects
Objects
Wall Accuracy (%)
Wall Accuracy (%)
Objects RMSE
Objects RMSE
90
90
2
4
80
VAE (no-memory)
RSSM
VAE+GRU (TBTT)
RSSM (TBTT)
Supervised oracle
Memory 9x9 and Memory 15x15 datasets. Left: Average accuracy of wall probing (higher is better), with the perfect score being 100% and VAE indicating a no-memory baseline. Right: Average poot-man--quaree error (RMSE) of object probing (lower is better), measured in grid cell units, with the perfect score being and VAE indicating a no-memory baseline.
Figure 4:
Offline probing results
Memory 9x9 Walls
Memory 15x15 Walls
Memory 9x9
Memory 15x15 Objects
Method
Objects
Constant baseline
80.8
78.3
4.9
8.0
VAE (no memory)
84.1
78.7+0.1
4.1
7.5
Supervised oracle
99.7
88.5+0.1
0.5
3.0
RSSM
89.
89.0+0.3
80.0
2.9
6.310.1
VAE+GRU (TBTT)
90.040.1
80.2
2.8
5.940.1
RSSM (TBTT)
94.0
80.5+0.2
2.5
5.840.1
Table 4: Offline probing benchmark results. The score is measured in wall prediction accuracy (%) for the Walls benchmarks and in RMSE (measured in grid cell units) for the Object benchmarks. The scores are calculated over the second half of the evaluation episodes to remove the effect of the initial memory burn-in (see Figure C.I). The models were trained for 1M gradient steps, and we report the mean score over three training runs, with the standard deviation indicated as a subscript.
All agents were evaluated after 100 million environment steps of training. For each baseline and task, trained five agents with different random seeds and report the average scores. A single Dreamer training run took 14 days to train using one GPU learner and 8 CPU actors. A single IMPALA training took 20 hours to train using GPU learner and 128 CPU actors. Our experimental results are summarized in Figure 3 and Table 3. The training curves are provided Appendix A.
First, we observe that the task is challenging but solvable for human player. The mean human score approximately 75% of the oracle upper bound across all four maze sizes. Inspection of episode replays shows that the player is slow at collecting the first few rewards while exploring the new unknown layout, after which the remaining rewards are collected relatively quickly (see Appendix B). This indicates that the task indeed relies on the ability to remember. Moreover, the learning phase becomes longer in the larger mazes.
Second, we note that the performance of RL agents exceeds the human performance on the smallest 9x9 maze but is far below the human baseline the largest 15x15 maze, with 1xx1 and 13x13 interpolating between the two extremes. This shows that our benchmark uncovers the limits of the current RI methods and enables measurable progress for future algorithms.
Third, we observe the utility of training Dreamer with TBTT, which shows a clear boost to the original Dreamer across all maze sizes. For example, on the Memory Maze 9x9, only the Dreamer (TBTT) agent achieves near-optimal performance.
Finally, even though Dreamer (TBTT) outperforms model-free IMPALA on the smaller mazes, on the larger mazes IMPALA is the best RL agent, making steady progress even on the Memory Maze 15x15. We speculate that IMPALA better at remembering tesk-relevant information for longer because model-free policy training encourages the encoder and RNN to only process and retain task-relevant information and ignore the rest. In contrast, the world model of Dreamer tries to remember as much information about the environment possible, which may limit the memory horizon.
7

---

Published as a conference paper at ICLR 2023
Agent Inputs
Probe Underlying Accuracy Trajectory
True pos
True neg
False pos
False neg
==0
50
100
150
200
250
Figure 5: Probing the representations of trained RSSM (TBTT) model for the information about the global layout of the maze. The model only raw pixels of the 3D first-person view (top row). The probe predictions of the wall layout are shown in the bottom row. After initial exploration, almost the full layout be decoded from the representation in the small maze, demonstrating its ability to retain information over time.
5 OFFLINE EXPERIMENTS
In this section, we present offline probing experiments the four benchmarks: Memory 9x9 (Walls), Memory 9x9 (Objects), Memory 15x15 (Walls), and Memory 15x15 (Objects), that were introduced in Section 3.3. We trained and evaluated the following sequence representation learning models:
RSSM Recurrent State Space Model (Hafner et al., 2018) the world model component of Dreamer agent. We use the exact same model that is part of the DreamerV2 agent (Hafner et al., 2020) that was evaluated on the online benchmark, with the hyperparameters listed Table E. The probe network receives the full internal state as input, which is concatenation of deterministic and stochastic states of RSSM.
RSSM (TBTT) As in the online experiments, evaluate the effect of training RSSM with truncated backpropagation through time (TBTT). Instead of starting with zero recurrent state on each training batch, we sample training sequences from trajectories sequentially and carry over the state from one batch to the next. Note that during inference time, both RSSM and RSSM (TBTT) propagate the state over the complete 1000-step trajectory. The difference is only during training, where RSSM effectively resets the state to zero every 48 steps, and RSSM (TBTT) does not.
VAE+GRU (TBTT) This baseline is recurrent model that operates separately learned representations (Ha and Schmidhuber, 2018). The image embeddings are trained with a standard VAE (Kingma and Welling, 2013) on the full dataset, without consideration of their sequential ordering. GRU-based RNN summarizes the sequence embeddings and actions a recurrent state, and dynamics MIP predicts the next observation embedding, given the current state and action. This network is trained with an MSE loss for the next-embedding prediction. The hyperparameters chosen match RSSM where relevant (see Table E.4). The probe predictor uses the GRU hidden state as the representation. This model is also trained with TBTT.
Supervised Oracle For comparison, evaluate baseline trained in supervised manner, allowing the gradients of the probe prediction loss to propagate into the whole model. We use the exact same architecture VAE+GRU, with the only difference of not stopping the probe gradients during training. It is an "oracle" in the sense that it does not follow the standard evaluation procedure, and uses hidden information (probe observations) to train the representation model. This baseline is useful as an upper (optimistic) bound for models with similar architecture.
VAE (no-memory) A model that simply uses the VAE image embeddings as the input to the probe prediction. It has no memory by design, and it is able to only predict the part of the probe observation which can be inferred from the current first-person view. serves lower (pessimistic) bound for memory models.
All models were trained for IM gradient steps, and repeated each training with three random seeds. The evaluation results are summarized in Figure and Table 4. In addition to the trained models, include Constant baseline that outputs the training mean prediction. This score is relatively
8

---

Published as a conference paper at ICLR 2023
Agent Inputs
Probe Underlying
Accuracy Trajectory
50
100
150
170
230
300
Figure 6: Probing the representations of trained RSSM (TBTT) model for the object locations. The model only sees raw pixels of a 3D first-person view (top row). The bottom row shows the object locations predicted from the frozen representation (x) versus the actual locations (o). The output of the probe decoder is the agent-centric object locations, here they are transformed into the global coordinates for visualization.
high on Walls (e.g. 80.8% for 9x9) because a relatively small number of grid cells vary between different layouts. The performance of models falls on the scale between the Constant baseline, as the minimum, and 100% accuracy or 0.0 RMSE as the maximum.
We observe that all evaluated models show some memory capacity (being better than the no-memory baseline), but fall short of perfect memory. The 15x15 Walls is an especially challenging benchmark, where the models barely outperform the no-memory baseline. This is consistent with the low performance of the corresponding agents on the 15x15 online RL benchmark, and suggests that Memory Maze can be fruitful platform for further research, both in the online and offline settings.
Among the models, RSSM (TBTT) reaches the highest performance, outperforming RSSM and VAE+GRU (TBTT). This shows that both truncated backpropagation through time and the stochastic bottleneck are helpful ingredients for learning to remember. On Memory 9x9 (Walls), RSSM (TBTT) correctly predicts 94% of the wall layout after the initial burn-in phase, compared the 84.1% no-memory baseline. This is consistent with the fact that Dreamer (TBTT) achieves near- perfect performance on the Memory 9x9 online RL benchmark. An illustration of sample evaluation trajectory, observations, and the corresponding probe predictions is presented in Figure 5 and Figure 6.
DISCUSSION
We introduced Memory Maze, an environment evaluating the memory abilities of agents. We conduced human study to confirm that the tasks are challenging but solvable for a human player and found that human rewards increase within each episode, demonstrating the need to build up memory to solve the tasks. We collected diverse offline dataset that includes semantic information to evaluate learned representations through probing, by predicting both task-specific and task-agnostic information. Empirically,
find that truncated backpropagation through time offers significant boost in probe performance, further confirming that memory is the key challenge in this benchmark. An evaluation of strong model-free and model-based RL algorithm each show learning progress smaller mazes but fall short human performance on larger mazes, presenting a challenge for future algorithms research.
Acknow Acdnowledgements We thank Google Research infrastructure team to provide JP with access to computational resources. DH was Google Research for part of the project before starting an internship at DeepMind.

---

Published as a conference paper at ICLR 2023
REFERENCES
1. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert. G. Powell, R. Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint 1rXiv:1110.07113, 2019.
A. Anand, F. Racah, S. Ozair, Y. Bengio, M.-A. Côté, and R. D. Hjelm. Unsupervised state representation learning in atari. Advances neural information processing systems, 32, 2019.
A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. D. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In International Conference Machine Learning, pages 507-517. PMLR, 2020.
C. Beattie, J. Z. Leibo, Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefraneq, S. Green, V. Valdés, A. Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation to benchmark reinforcement learning. International conference on machine learning, pages 2048-2056. PMLR, 2020.
Degrave, Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, Hafner, A. Abdol- maleki, D. de Las Casas, al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414 419, 2022.
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, Dunning, et al. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. arXiv preprint 102.v.180..0/56/, 2018.
L. Espeholt, R. Marinier, P. Stanczyk, K. Wang, and M. Michalski. SEED RL: Scalable and efficient deep-RL with accelerated central inference. arXiv preprint arXiv:1910.06599, 2019.
M. Fortunato, M. Tan, R. Faulkner, S. Hansen, A. P. Badia, G. Buttimore, C. Deck, J. Z. Leibo, and C. Blundell. Generalization of reinforcement learners with working and episodic memory. arXiv preprint arXiv:1910.13400 2019.
K. Gregor. D. Jimenez Rezende, Besse, Y. Wu, H. Merzie, and A. van den Oord. Shaping belief states with generative environment models for RL. Advances in Neural Information Processing Systems, 32, 2019.
D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems, 31, 2018.
D. Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.0678) 2021.
D. Hafner, Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.045571 2018.
D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv22010.02199. 2020.
Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference learning representations, 2018.
D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114, 2013.
10

---

Published as a conference paper at ICLR 2023
A. Mandhane, A. Zhernov, M. Rauh, C. Gu, M. Wang. F. Xue, W. Shang, D. Pang. R. Claus, C.-H. Chiang, et al. Muzero with colpetteeniion for rate control in vp9 video compression. arXiv preprint ar Stt1::222200662666 2022.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Oh, X. Guo, H. Lee, R. L. Lewis, and Singh. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems, pages 2863-2871, 2015.
A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 087000778.... 2018.
E. Parisotto. F. Song, Rae, Pascanu, C. Gulcehre, Jayakumar, M. Jaderberg, R. L. Kaufman,
A. Clark, S. Noury, et al. Stabilizing transformers for reinforcement learning, In International Conference on Machine Learning, pages 7487-7498. PMLR, 2020.
A. G. Richards. Robust constrained model predictive control. PhD thesis, Massachusetts Institute of Technology, 2005.
J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint 19/1: 2019.
D. Silver, A. Huang, J. Maddison, Guez, Sifre, G. Van Den Driessche, Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of with deep neural networks and tree search. Nature, 429(7587):484, 2016.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXi::1801.006% 2018.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. doi: 1...11999RR.0.2...0
Tunyasuvunakool, Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, and Y. Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638.
G. Wayne, C.-C. Hung, D. Amos, M. Mirza, A. Ahuja, A. Grabska- Baiska-Barwinska.. J. Rac, Mirowski, J. Z. Leibo, A. Santoro, et al. Unsupervised predictive memory in goal-directed agent. arXiv preprint arxi 10760, 2018.
P. Wu, A. Escontrela, D. Hafner, K. Goldberg, and P. Abbeel. Daydreamer: World models for physical robot learning, arXiv preprint arXiv:2206.144 14176. 2022.
D. Yarats, Brandfonbrener, H. Liu, M. Laskin, Abbeel, A. Lazaric, and L. Pinto. Don't tt change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint 2rXiv:2201.13422 2022.
D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909.08593, 2019.
11

---

Published as a conference paper at ICLR 2023
A
ONLINE TRAINING CURVES
Memory 9x9
Memory 11x11
Memory 13x13
Memory 15x15
30
40
50
60
Return
24
32
40
45
24
30
30
12
16
20
10
15
50M
100M
50M
100M
50M
100M
0
50M
100M
Environment steps
Environment steps
Environment steps
Environment steps
Dreamer (TBTT) Dreamer
IMPALA
Human
Figure A.1: Training curves on the Memory Maze tasks. The scores are averaged over five runs, standard deviation is shown as shaded area. After training for 100M environment steps, Dreamer (TBTT) shows the best performance IMPALA is the best on the larger mazes 13x13 and 15x15.
the two smaller benchmarks Memory 9x9 and 11x11, and
Memory 9x9
Memory 11x11
Memory 13x13
Memory 15x15
30
40
50
60
Return
32
40
45
18
24
30
30
20
10
15
200M
400M
200M
400M
200M
400M
200M
400M
Environment steps
Environment steps
Environment steps
Environment steps
Dreamer (TBTT)
Dreamer
IMPALA
Human
Memory Mazes over 400M steps. On Memory Maze 15x15 it makes steady progress beyond 100M steps reaching score of 27.7 + 0.2, but it is flattening far below the Human baseline.
Figure A.2: Extended IMPALA training
B HUMAN SCORES
Memory 9x9
Memory 11x11
Memory 13x13
Memory 15x15
Return per 100 steps
3
1
0
500
1000
1000
2000
2000
2000
4000
Step of Episode
Step of Episode
Step of Episode
Step of Episode
Figure B.l: The average reward collected as a function of the episode step within the episode, summed over 100 step bins, averaged over the episodes the human player. The reward collection slow in the beginning of the episode, when the maze layout is unknown and the player has to explore the maze. The pace increases as the player memorizes the maze and object locations, and can effectly navigate between them. This shows that the task relies on long-term memory.
12

---

Published as a conference paper at ICLR 2023
C
EXTRA OFFLINE PROBING RESULTS
Memory 9x9
Memory 15x15 Walls
Memory 9x9 Objects
Memory 15x15 Objects
Walls
Wall Accuracy (9)
100
100
Objects RMSE
4
96
95
6
3
92
90
2
88
85
1
2
80
0
1000
500
1000
500
1000
500
1000
Step of Episode
Step of Episode
Step of Episode
Step of Episode
VAE (no-mem)
RSSM
VAE+GRU (TBTT)
RSSM (TBTT)
Supervised oracle
Figure C.I: Offlin probe prediction performance function of the episode step. The prediction error starts high at the beginning of the episode, when the layout of the maze is impossible to predict, and then drops the agent explores the maze. The benchmark results reported in Table 4 are equal to the mean value of these plots over the range from 500 to 1000, where the prediction error is stable.
D PROBE NETWORK
Here motivate the choice of using 4-layer MLP probe network as opposed more typical linear probe. Figure D.1 shows how probe accuracy the Memory Maze 9x9 (Walls) benchmark depends on the number layers in the MLP. The linear probe performs poorly; the accuracy increases up to layers. More layers provide little additional gain. Note the probe has to take into account the input about the position of the agent, which is infeasible with a linear probe. At the other extreme, one may worry that powerful enough network could extract any desired information about the history of observations, but we show that I) additional layers don't I do better and 2) a probe MLP trained on a random untrained RSSM is only marginally better than the constant baseline (80.8%):
linear
layer
2 layers
4 layers
5 layers
layers
layers
untrained RSSM
80
85
90
95
Map Accuracy (%)
Figure D.1: Average map probe accuracy on Memory Maze 9x9 (Walls) offline benchmark as function of the number of layers in the probe MLP. The last row shows the performance of 4-layer MLP probe trained to decode information from a randomly initialized RSSM.
13

---

Published as a conference paper at ICLR 2023
E HYPERPARAMETERS
For tuning hyperparameters we performed one-dimensional grid searches for one parameter at a time. We evaluated parameters on the Memory 11x11 environment, since it is the smallest challenging one, and then used the best values across all environments. For Dreamer agent we scanned over the following parameter values: recurrent state size (512, 1024, 2048, 4096), KL scale (0.1, 0.3, 1.0, 3.0), entropy scale (3 10-1, 1- 1.10=,.3 3 10-3, 1 10--). For IMPALA we tuned the entropy scale (3 10-1 1. 10-3 3- 3.3-10 10-33 1 10-2), learning rate (1 10-1,
, 2 10-1, 3 10-1, 4 10-1) and Adam
epsilon (3 10-9, 10-?, 1006. 9)
Parameter
Value
Model
RSSM recurrent state size
2048
RSSM stochastic discrete latent size
32 x 32
RSSM number of hidden units
1000
Decoder MLP number of layers
Decoder MLP number of units
400
Probe MLP number of layers
Probe MLP number units
1024
Objective
Discount
0.995
X-target parameter
0.95
Actor entropy loss scale
0.001
RSSM KL. loss scale
1.0
RSSM KL balancing
0.8
Training
Parallel environments
Replay buffer size
10M steps 32 sequences
Batch size
Sequence length
48
Imagination horizon Environment steps per update
25
Slow critic update interval
100
World model learning rate
3 10
Actor learning rate
1 10
Critic learning rate
1 1-10-
Adam W epsilon
10-5 5
Adam weight decay
10-2
Gradient clipping
200
Table E.1: Hyperparameters used when training Dreamer and Dreamer (TBTT) agents. The dif- ferences from the parameters of the original Dreamer Atari model (Table D.1 in Hafner et al. (2020)) are shown in bold face. The most substantial differences are: RSSM recurrent state size (600- + 2048), replay buffer size (10" - 10"), environment steps per update (4 -> 25) and KI. loss scale (0.1 - 1.0).
14

---

Published as a conference paper at ICLR 2023
Parameter
Value
Parallel environments
128
LSTM size
256
Discount
0.99
X-target parameter
0.95
Actor entropy loss scale
0.001
Batch size
32 sequences
Sequence length
100
Adam learning rate
10-4
Adam epsilon
10-7
Table E.2: Hyperparameters used when training IMPALA agent. The parameters were based on those used in (Espeholt et al., 2019) on the DMLab-30 tasks, with the additional tuning of entropy scale, learning rate and Adam epsilon. We also tested larger LSTM size, but provided no extra performance.
Parameter
Value
Probe MLP number of layers
Probe MLP number of units
1024 ELU
Probe MLP activation
Probe MLP layer norm epsilon
10-3
Adam learning rate
3 10-4
Adam epsilon
10-5
Table E.3: Hyperparameters of the probe decoder used in the offline probing benchmarks.
Parameter
Value
GRU state size
2048
VAE stochastic latent size
32
Encoder
(same as Dreamer) (same as Dreamer)
Decoder
Dynamics MLP number of layers
2
Dynamics MLP number of units VAE KL loss scale
400
0.1
Batch size
32 sequences
Sequence length
48
Adam W learning rate
3 10-4
AdamW epsilon
10
AdamW weight decay
10 10-2
Gradient clipping
200
Table E.4: Hyperparameters used for the GRU+VAE offline probing baseline.
Parameters are chosen to match Dreamer model Table E. where relevant.
15

---

Published as a conference paper at ICLR 2023
F ENVIRONMENT DETAILS
Index
Action
Forward/Backward
Left/Right
noop
0.0
0.0
forward turn left
+1.0
0.0
0.0
-1.0
turn right
0.0
+1.0
forward left
+1.0
1.0
forward right +1.0
+1.0
Table F.1: Memory Maze 6-dimensional discrete action space mapping the underlying DMC continuous action space.
16


Reviews:
[{"review_id": "d2zowMlOc6S", "reviewer": "Reviewer_9HKy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new benchmark, Memory Maze, for evaluating the long-term memory capabilities of RL agents. MemoryMaze consists of randomly generated 3D mazes wherein an agent gets only local camera observations along with a specified goal item to locate. In each episode, the environment presents the agent with a sequence of random target items in a random maze, and the agent receives a reward for each target successfully reached before a time limit runs out. The task emphasizes memory because after observing an item's location the first time, the agent should be able to remember it, along with the maze layout, to find it more quickly the next time within the same episode. Various maze sizes are tested to tune the difficulty of the memory challenge. The orthogonal exploration challenge is mitigated by the fact that the start location is randomized in each episode, thus the agent will occasionally be able to spawn close to a goal item to observe reward. Experiments benchmark the performance of a human against strong reinforcement learning agents. The first baseline is model-free IMPALA, the second is model-based Dreamerv2. For DreamerV2, one variant is tested where the RNN state is zeroed at the start of each training sequence, and another is tested where the RNN state is maintained for whole episodes, but gradients are propagated only for a fixed sequence length (called truncated backpropagation through time in the paper). The result indicated that, in the smaller mazes, strong RL agents can perform similarly to or better than humans, while for larger mazes the RL agents fall far behind, indicating room for future progress. In addition to the RL experiments, offline probing experiments are included which run sequence models on prerecorded episodes and probe the latent state of a sequence model for information about the location of items and walls in the maze.", "summary_of_the_review": "This paper presents a well-thought-out benchmark for evaluating the long-term memory ability of current reinforcement learning systems. I believe this is an important contribution, as such environments are necessary for progress. Experiments are generally good and give insight into the utility of the environment for the stated purpose. My main concerns are a few issues with unclear claims and empirical details that could be added. The technical novelty is also not high as similar environments have been used to test memory before, but I still see this paper's aim of proposing and evaluating the environment itself as a worthwhile contribution.", "strength_and_weaknesses": "Strengths\n=========\n* Good benchmarks to test specific agent capabilities facilitate progress in reinforcement learning research.\n* The proposed environment seems well thought out and it's generally well explained how it aims to test memory specifically while mitigating dependency on other challenges like exploration.\n* The paper is for the most part very well-written and clear.\n* The empirical results are interesting and seem generally well done. They do a good job of highlighting the utility of the environment for evaluating the memory capabilities of strong reinforcement learning agents. I like the inclusion of the offline probing task and associated empirical results as a way to evaluate memory capability separated from the other challenges of reinforcement learning within the same environment.\n\nWeaknesses\n==========\n* Certain claims are not clearly demonstrated in the paper or are overly vague. For example \"This makes training much faster compared to, for example, DM Memory Suite\", while perhaps plausible, this is not meaningfully demonstrated. Moreover, this statement refers to a whole suite of tasks and I don't believe it's true for all of them so this should at least be clarified and ideally demonstrated in some manner. Another example is the statement \"on the larger mazes IMPALA is the best RL agent\" and the associated speculative explanation. This doesn't seem like a fair comparison at all since dreamer is trained with 8 actors and IMPALA with 128. Does this not mean IMPALA gets 16 times more data than dreamer? Of course, there is a lot of nuance to this since dreamer is also much slower to train, however, I don't think the way this is currently explained is particularly insightful.\n* Human data is collected using only one person. Evaluating a larger group of people could allow measurement of within-group variance and gives a better sense of the consistency of human performance on this task. This also makes the use of the plural in \"We conducted a human study to confirm that the tasks are challenging but solvable for average human players\" debatably incorrect.\n* Some empirical details such as how precisely hyperparameter tuning was done are missing. For example, for Dreamer, it just says they \"tuned the KL and entropy scales to the new environment\", tuned how (grid search, manual tweaking...)? On which version of the environment (since this hyperparameter is shared across maze sizes)? For Impala \"we tuned the entropy loss scale by scanning different values and then used the same hyperparameters across all four environments\", scanning what values? And again for which version of the environment?\n* Technical novelty is perhaps not that high, since previous work (for example the work of Wayne et. al. (2018), which is not cited) has investigated training agents on very similar tasks where an agent must repeatedly find objects in a randomized maze. However, given the focus of this work is on demonstrating the utility of the environment itself, which I think is worthwhile, I don't see this as a huge issue. On that note, it would be good to add a citation to Wayne et. al. (2018) as they also propose several tasks for evaluating memory, including one in which episodes consist of repeatedly locating a specific item in a randomized maze (albeit I believe only one item and not multiple as in Memory Maze).\n\nMinor Comments and Corrections\n==============================\n* In Introduction (page 1): \"solve later solve\"\n* In Introduction (page 2): \"this offers and estimate\"\n* Section 4 (page 6): \"...indicating that the human player had to observe the object positions multiple times before remembering them.\", I don't quite understand how\n* I can't see that code is made available in the present draft, though it is stated that \"we open source the environment and make it easy to install and use\". I think acceptance should definitely be conditional on this release given the testbed itself is the central contribution.\n* Section 5 (page 8): \"RSSM were relevant\"\n* For the MSE shown in Figure 4, it would be good to give some reference to understand the magnitude of these values relative to the maze (is each maze cell one unit?). Additionally, it would be good to show root MSE, or simply mean distance, instead of MSE so that one can more easily relate these errors to distances in the maze.\n\nReference\n=========\nWayne, Greg, et al. \"Unsupervised predictive memory in a goal-directed agent.\" arXiv preprint arXiv:1803.10760 (2018).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EhLJhrUUetr", "reviewer": "Reviewer_kp2x", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to reinforce the research about memory-dependence in decision making, especially reinforcement learning (RL). The core contribution is the introduction of a new task(environment) set, a kind of Maze with first-person image as observation. To get more rewards, the agent needs to \"remember\" (or \"understand\" with internal representations) the layout of the maze so as to reach each target position. The authors performed extensive deep RL experiments to verify (1) Long-term memory is the leading difficulty of the tasks (2) The tasks difficulties range from \"solvable\" to \"challenging\" to deep RL agents.  Also, the the authors provide human-demonstrated dataset for offline RL studies. In sum, the paper contributes by introducing a novel benchmark for memory-dependent decision-making studies.", "summary_of_the_review": "Based on the comments above, I think this is a good paper and I vote for acceptance.", "strength_and_weaknesses": "## [Strength]\n- The study addresses an important open question in decision making\n- The proposed environments are intuitive while challenging.\n- The paper was written excellently. Both the high-level idea and the details are clear.\n- The experiments are comprehensive and convincible with abalation studies.\n\n## [Comments]\n- The action space seems to be discrete (Table F.1). I wonder whether the authors can provide a version with continous actions since it is more realistic. \n- The title \"3D maze\" is a bit ambiguous: the agent is moving on a 2D space. \n- Some references can be more formal, e.g., \"world models .. arxiv\" should be the NeurIPS version, and \"rl\" should be capital letters.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4WfUpb9iBNg", "reviewer": "Reviewer_pb7h", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a new RL environment, Memory Maze, designed to evaluate an agent’s ability to maintain moderately long-term memories of observations, independent of the common challenges of exploration and credit assignment. A dataset of expert trajectories is provided to support offline RL research. Baseline (including human) results demonstrate that the benchmark spans the range from currently solvable to very challenging. Further support is provided for probing the agent’s representations.", "summary_of_the_review": "While this work is not flashy, and does not establish new SOTA results, the benchmark and baselines are likely to prove valuable in research to improve the memory of RL agents, which currently falls dramatically short of human-level.\n\nUpdate (Nov 18): I have read all reviews and author responses. I still view this work as a solid contribution. My score remains unchanged.", "strength_and_weaknesses": "**Strengths**\n\nDespite the ubiquity of partially observable environments, and many existing RL tasks that require some degree of memory, most fall short in ways detailed by this paper. The design of the Memory Maze environment is well-motivated, and carefully positioned with regards to prior work. The result is a compelling RL environment for evaluating the memory abilities of agents. The episodes are appropriately lengthy, to measure memories that persist for thousands of timesteps. \n\nThe range of maze sizes, rooms, and objects is crafted to enable graduated evaluation of agent abilities. The baseline agents are well chosen, and include human results. The Dreamer baseline results are particularly interesting, outperforming a standard model-free agent (Impala) on smaller mazes, and under-performing on larger mazes. And the human results in Fib B.1 show how much room there is for RL methods to improve.\n\n**Weaknesses**\n\nThe paper would be stronger if it included (along with the baselines) a novel agent architecture that established state-of-the-art results on Memory Maze. Without that, the benchmark’s impact will depend on the extent to which the community chooses to use it.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster