Title: Reinforcement Learning with Fast and Forgetful Memory
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

partially observable

POMDP

memory

rnn

transformer
Abstract: Nearly all real world tasks are inherently partially observable, necessitating the use of memory in Reinforcement Learning (RL). Most model-free approaches summarize the trajectory into a latent Markov state using memory models borrowed from Supervised Learning (SL), even though RL tends to exhibit different training and efficiency characteristics. Addressing this discrepancy, we introduce Fast and Forgetful Memory, an algorithm-agnostic memory model designed specifically for RL. Our approach constrains the model search space via strong structural priors inspired by computational psychology. It is a drop-in replacement for recurrent neural networks (RNNs) in recurrent RL algorithms, achieving greater reward than RNNs across various recurrent benchmarks and algorithms _without changing any hyperparameters_. Moreover, Fast and Forgetful Memory exhibits training speeds two orders of magnitude faster than RNNs, attributed to its logarithmic time and linear space complexity. Our implementation is available at https://github.com/proroklab/ffm.

Reinforcement Learning with Fast and Forgetful Memory

Abstract

Nearly all real world tasks are inherently partially observable, necessitating the use of memory in Reinforcement Learning (RL). Most model-free approaches summarize the trajectory into a latent Markov state using memory models borrowed from Supervised Learning (SL), even though RL tends to exhibit different training and efficiency characteristics. Addressing this discrepancy, we introduce Fast and Forgetful Memory, an algorithm-agnostic memory model designed specifically for RL. Our approach constrains the model search space via strong structural priors inspired by computational psychology. It is a drop-in replacement for recurrent neural networks (RNNs) in recurrent RL algorithms, achieving greater reward than RNNs across various recurrent benchmarks and algorithms without changing any hyperparameters . Moreover, Fast and Forgetful Memory exhibits training speeds two orders of magnitude faster than RNNs, attributed to its logarithmic time and linear space complexity. Our implementation is available at https://github.com/proroklab/ffm .

1 Introduction

Reinforcement Learning (RL) was originally designed to solve Markov Decision Processes (MDPs) [Sutton and Barto, 2018], but many real world tasks violate the Markov property, confining superhuman RL agents to simulators. When MDPs produce noisy or incomplete observations, they become Partially Observable MDPs (POMDPs). Using memory , we can summarize the trajectory into a Markov state estimate, extending convergence guarantees from traditional RL approaches to POMDPs [Kaelbling et al., 1998].

In model-free RL, there are two main approaches to modeling memory: (1) RL-specific architectures that explicitly model a probabilistic belief over Markov states [Kaelbling et al., 1998] and (2) general-purpose models such as RNNs or transformers that distill the trajectory into a fixed-size latent Markov state. Ni et al. [2022] and Yang and Nguyen [2021] reveal that with suitable hyperparameters, general-purpose memory often outperforms more specialized belief-based memory.

Most applications of memory to RL tend to follow Hausknecht and Stone [2015], using RNNs like Long Short-Term Memory (LSTM) or the Gated Recurrent Unit (GRU) to summarize the trajectory, with other works evaluating transformers, and finding them tricky and data-hungry to train [Parisotto et al., 2020]. Morad et al. [2023] evaluate a large collection of recent memory models across many

Ryan Kortvelesy

Amanda Prorok

partially observable tasks, finding that the GRU outperformed all other models, including newer models like linear transformers. Interestingly, they show that there is little correlation between how well memory models perform in SL and RL. Even recent, high-profile RL work like Hafner et al. [2023], Kapturowski et al. [2023] use RNNs over more modern alternatives, raising the question question: why do older memory models continue to overshadow their contemporary counterparts in modern RL?

Training model-free RL policies today is sample inefficient, alchemical, and prone to collapse [Schulman et al., 2015, Zhang et al., 2018], and reasoning over the entire trajectory magnifies these issues. SL can better utilize scale, compute, and dataset advancements than RL, largely removing the need for strong inductive biases . For example, transformers execute pairwise comparisons across all inputs, in a somewhat 'brute force' approach to sequence learning. Similarly, State Space Models [Gu et al., 2021] or the Legendre Memory Unit [Voelker et al., 2019] are designed to retain information over tens or hundreds of thousands of timesteps, growing the model search space with each additional observation.

Through strong inductive biases, older recurrent memory models provide sample efficiency and stability in exchange for flexibility. For example, the GRU integrates inputs into the recurrent states in sequential order, has explicit forgetting mechanics, and keeps recurrent states bounded using saturating activation functions. These strong inductive biases curtail the model search space, improve training stability and sample efficiency, and provide a more 'user-friendly' training experience. If inductive biases are indeed responsible for improved performance, can we better leverage them to improve memory in RL?

Contributions We introduce a memory model that summarizes a trajectory into a latent Markov state for a downstream policy. To enhance training stability and efficiency, we employ strong inductive priors inspired by computational psychology. Our model can replace RNNs in recurrent RL algorithms with a single line of code while training nearly two orders of magnitude faster than RNNs. Our experiments demonstrate that our model attains greater reward in both on-policy and off-policy settings and across various POMDP task suites, with similar sample efficiency to RNNs.

2 Related Work

RL and SL models differ in their computational requirements. While in SL the model training duration is primarily influenced by the forward and backward passes, RL produces training data through numerous inference workers interacting with the environment step by step. Consequently, it's imperative for RL memory to be both fast and efficient during both training and inference . However, for memory models, there is often an efficiency trade-off between between the training and inference stages, as well as a trade-off between time and space complexity.

Recurrent Models Recurrent models like LSTM [Hochreiter and Schmidhuber, 1997], GRUs [Chung et al., 2014], Legendre Memory Units (LMUs) [Voelker et al., 2019], and Independent RNNs [Li et al., 2018] are slow to train but fast at inference. Each recurrent state S over a sequence must be computed sequentially

where x j , y j are the inputs and outputs respectively at time j , and f updates the state S incrementally. The best-case computational complexity of recurrent models scales linearly with the length of the sequence and such models cannot be parallelized over the time dimension , making them prohibitively slow to train over long sequences. On the other hand, such models are quick at inference, exhibiting constant-time complexity per inference timestep and constant memory usage over possibly infinite sequences.

Parallel Models Parallel or 'batched' models like temporal CNNs [Bai et al., 2018] or transformers [Vaswani et al., 2017] do not rely on a recurrent state and can process an entire sequence in parallel i.e.,

Given the nature of GPUs, these models exhibit faster training than recurrent models. Unfortunately, such models require storing all or a portion of the trajectory, preventing their use on long or infinite

sequences encountered during inference. Furthermore, certain parallel models like transformers require quadratic space and n comparisons at each timestep during inference. This limits both the number of inference workers and their speed, resulting in inefficiencies, especially for on-policy algorithms.

Hybrid Models Linear transformers [Katharopoulos et al., 2020, Schlag et al., 2021, Su et al., 2021] or state space models [Gu et al., 2021, 2020] provide the best of both worlds by providing equivalent recurrent and closed-form (parallel) formulations

Training employs the batched formula to leverage GPU parallelism, while inference exploits recurrent models' low latency and small memory footprint. Thus, hybrid models are well-suited for RL because they provide both fast training and fast inference, which is critical given the poor sample efficiency of RL. However, recent findings show that common hybrid models typically underperform in comparison to RNNs on POMDPs [Morad et al., 2023].

3 Problem Statement

We are given a sequence of actions and observations ( o 1 , 0 ) , ( o 2 , a 1 ) , . . . ( o n , a n -1 ) up to time n . Let the trajectory X be some corresponding encoding x = ε ( o , a ) of each action-observation pair

Our goal is to summarize X n into a latent Markov state y n using some function f . For the sake of both space and time efficiency, we restrict our model to the space of hybrid memory models. Then, our task is to find an f , such that

where X k : n is shorthand for x k , x k +1 , . . . , x n . Note that by setting k = n , we achieve a one-step recurrent formulation that one would see in an RNN, i.e., y n , S n = f ( X n : n , S n -1 ) = f ( x n , S n -1 ) .

4 Background

In the search for inductive priors to constrain our memory model search space, we turn to the field of computational psychology. In computational psychology, we use the term trace to refer to the physical embodiment of a memory - that is, the discernable alteration in the brain's structure before and after a memory's formation. Computational psychologists model long-term memory as a collection of traces, often in matrix form

where x j are individual traces represented as column vectors and S n is the memory at timestep n [Kahana, 2020].

Composite Memory Composite memory [Galton, 1883] approximates memory as a lossy blending of individual traces x via summation, providing an explanation for how a lifetime of experiences can fit within a fixed-volume brain. Murdock [1982] expresses this blending via the recurrent formula

where γ ∈ (0 , 1) is the forgetting parameter and B n is a diagonal matrix sampled from a Bernoulli distribution, determining a subset of x n to add to memory. In Murdock's model, S is a vector, not a matrix. We can expand or 'unroll' Murdock's recurrence relation, rewriting it in a closed form

making it clear that memory decays exponentially with time. Armed with equivalent recurrent and parallel formulations of composite memory, we can begin developing a hybrid memory model.

Contextual Drift It is vital to note that incoming traces x capture raw sensory data and lack context . Context, be it spatial, situational, or temporal, differentiates between seemingly identical raw sensory inputs, and is critical to memory and decision making. The prevailing theory among computational psychologists is that contextual and sensory information are mixed via an outer product [Howard and Kahana, 2002]

where η n is the contextual state at time n , and ρ ensures smooth or gradual changes between contexts. Although context spans many domains, we focus solely on temporal context.

5 Fast and Forgetful Memory

Fast and Forgetful Memory (FFM) is a hybrid memory model based on theories of composite memory and contextual drift. It is composed of two main components: a cell and an aggregator . The cell receives an input and recurrent state x , S and produces a corresponding output and updated recurrent state y , S (Figure 1). The aggregator resides within the cell and is responsible for updating S (Figure 2). We provide the complete equations for both the aggregator and cell and end with the reasoning behind their design.

Aggregator The aggregator computes a summary S n of X 1: n , given a recurrent state S k -1 and inputs X k : n

where ⊙ is the Hadamard product (or power), m is the trace size, c is the context size, and α ∈ R m + , ω ∈ R c are trainable parameters representing decay and context respectively. Multiplying column a vector by 1 ⊤ c 'broadcasts' or repeats the column vector c times.

In Appendix C, we derive a memory-efficient and numerically stable closed form solution to compute all states S k : n in parallel over the time dimension. Since our model operates over relative time, we map absolute time k, . . . , n to relative time p ∈ 0 , . . . , t , where t = n -k . The closed form to compute a state S k + p given S k -1 is then

We can rewrite the aggregator's closed form (Equation 13) in matrix notation to highlight its timeparallel nature, computing all states S k : n = S k , S k +1 , . . . , S k + n at once

The cumulative sum term in the rightmost matrix can be distributed across t processors, each requiring O (log t ) time using a prefix sum or scan [Harris et al., 2007].

Cell The aggregator alone is insufficient to produce a Markov state y . S is complex-valued and contains a large amount of information that need not be present in y at the current timestep, making

1

n

0

x

1

1

n

ℜ

(

-

x

1

)

ℜ

(

x

1

)

1

n

1

n

ℑ

(

-

x

1

)

ℑ

(

x

1

)

1

n

⊙

e

-

tα

x

1

·

(

e

-

ti

ω

)

⊤

S

⊕

⊙

γ

ℓ

1

(

·

)

⊙

σ

(

ℓ

2

(

·

))

LN(

ℓ

3

(

·

))

⊙

σ

(

ℓ

4

(

·

))

(1

-

σ

(

ℓ

4

(

·

)))

⊙

ℓ

5

(

·

)

⊕

y

k

S

k

-

1

x

k

S

k

˜

x

k

it cumbersome to interpret for a downstream policy. Furthermore, X could benefit from additional preprocessing. The cell applies input gating to X and extracts a real-valued Markov state y from S .

Agg represents the aggregator (Equation 14) and ℓ represents linear layers with mappings ℓ 1 , ℓ 2 : R d → R m , ℓ 3 : R m × 2 c → R d , ℓ 4 , ℓ 5 : R d → R d . ℜ , ℑ extract the real and imaginary components of a complex number as reals, Flatten reshapes a matrix ( m × c → mc ) and || is the concatenation operator. LN is nonparametric layer norm, and σ is sigmoid activation. Equation 15 applies input gating, Equation 16 computes the recurrent states, Equation 17 projects the state into the real domain, and Equation 18 applies output gating.

Modeling Inductive Biases To justify FFM's architecture, we detail its connections to computational psychology. Drawing from the theory of composite memory, we integrate sigmoidal gating to x in Equation 15, approximating the sparsity-driven Bernoulli term, B , presented in (Equation 7). This inductive bias represents that only a small subset of each sensory experience is stored in memory.

Following the blending constraint imposed by composite memory (Equation 8), we sum the traces together (Equation 11). We suspect this blending prior yields a model that is more robust to the bad policy updates that plague RL, compared to RNNs that apply nonlinear and destructive transforms (e.g. deletion) to the recurrent state.

In Equation 12, the exponential component exp( -t α ) ensures that traces are forgotten according to composite memory decay from Equation 8. We choose an exponential decay e -| α | , approximating Murre and Dros [2015]. This inductive bias asymptotically decays traces, reducing the size of the model optimization space. This is in direct contrast to recent SL memory models which aim to store as much information as possible for as long as possible [Gu et al., 2021, Voelker et al., 2019, Schlag et al., 2021]. In our ablations, we show that forgetting is critical in RL.

Traditionally, contextual coding and composite memory are separate mechanisms, but we incorporate both through γ . The exp( -i ω t ) component of γ in Equation 12 applies temporal context, enabling relative-time reasoning. Following contextual coding theory, we take the outer product of a trace vector, in our case decayed by α , with a context vector produced by ω . We can rewrite γ t = γ t -1 ⊙ γ , mirroring the gradual recurrent context changes from Equation 10.

The recurrent state S is a complex-valued matrix, so we project it back to the real domain as z (Equation 17). z can be large for dimensions where α is near zero (minimal decay), so we find it crucial to apply layer normalization in Equation 18. Finally, we apply a gated residual connection to improve convergence speed, letting gradient descent find the optimal mixture of input x to memory z for the Markov state y .

The Mathematics Underpinning FFM The goal of FFM is to produce a memory mechanism which can be parallelized across the time dimension. As discussed in section 4, we can achieve this through a summation of sensory information mixed by some arbitrary function ϕ with a temporal context ω j = ψ ( j ) , generating an aggregator of the form

Any implementation under this formulation would provide memory with temporal context for absolute time. However, in order to maximize generalization, we wish to operate over relative time. This introduces a problem, as the relative temporal context ψ ( n -j ) associated with each x j must be updated at each timestep. Therefore, we must find a function h that updates each temporal context with some offset: h ( ψ ( j ) , ψ ( k )) = ψ ( j + k ) . Solving for h , we get: h ( j, k ) = ψ ( ψ -1 ( j ) + ψ -1 ( k )) . If we also set ϕ = h , then applying a temporal shift h ( · , k ) to each term in the sum yields the same result as initially inserting each x j with ψ ( j + k ) :

Consequently, it is possible to update the terms at each timestep to reflect the relative time n -j . However, it is intractable to recompute n relative temporal contexts at each timestep, as that would result in n 2 time and space complexity for a sequence of length n . To apply temporal context in a batched manner in linear space, we can leverage the distributive property. If we select ϕ ( a, b ) = a · b , then the update distributes over the sum, updating all of the terms simultaneously. In other words, we can update the context of n terms in constant time and space (see Appendix C for the full derivation). As ψ is defined as a function of ϕ , we can solve to find ψ ( t ) = e ξt (although any exponential constitutes a solution, we select the natural base). This yields an aggregator of the form

If ξ ∈ R + , x j = 0 , then the recurrent state will explode over long sequences: lim n →∞ e ξ ( n -j ) x j = ∞ , so the real component of ξ should be negative. Using different decay rates ξ j , ξ k ∈ R -, we can deduce the temporal ordering between terms x j and x k . Unfortunately, this requires both x j , x k eventually decay to zero - what if there are important memories we do not want to forget, while simultaneously retaining their ordering? If ξ is imaginary, then we can determine the relative time between x j , x k as the terms oscillate indefinitely without decaying. Thus, we use a complex ξ with a negative real component, combining both forgetting and long-term temporal context. In the following paragraphs, we show that with a complex ξ , the FFM cell becomes a universal approximator of convolution .

̸

Universal Approximation of Convolution Here, we show that FFM can approximate any temporal convolutional. Let us look at the z term from Equation 17, given the input ˜ x , with a slight change

in notation to avoid the overloaded subscript notation: ˜ x ( τ ) = ˜ x τ , z ( n ) = z n . For a sequence 1 to n , z ( n ) , the precursor to the Markov state y n , can be written as:

where A , b is the weight and bias from linear layer ℓ 3 , indexed by subscript. Looking at a single input dimension k of ˜ x , we have

Equation 24 is a temporal convolution of ˜ x k ( t ) using a Fourier Series filter with c terms ( ω ∈ C c ), with an additional learnable 'filter extent' term α k . The Fourier Series is a universal function approximator, so FFM can approximate any convolutional filter over the signal ˜ x ( τ ) . Appendix A further shows how this is related Laplace transform. Unlike discrete convolution, we do not need to explicitly store prior inputs or engage in zero padding, resulting in better space efficiency. Furthermore, the filter extent α k is learned and dynamic - it can expand for sequences with long-term temporal dependencies and shrink for tasks with short-term dependencies. Discrete temporal convolution from methods like Bai et al. [2018] use a fixed-size user-defined filter extent.

Interpretability Unlike other memory models, FFM is interpretable. Each dimension in S has a known decay rate and contextual period. We can determine trace durability (how long a trace lasts) t α and the maximum contextual period t ω of each dimension in S via

where β determines the strength at which a trace is considered forgotten. For example, β = 0 . 01 would correspond to the time when the trace x k contributes 1% of its original value to S n . FFM can measure the relative time between inputs up to a modulo of t ω .

6 Experiments and Discussion

We evaluate FFM on the two largest POMDP benchmarks currently available: POPGym [Morad et al., 2023] and the POMDP tasks from [Ni et al., 2022], which we henceforth refer to as POMDPBaselines. For POPGym, we train a shared-memory actor-critic model using recurrent using Proximal Policy Optimization [Schulman et al., 2017]. For POMDP-Baselines, we train separate memory models for the actor and critic using recurrent Soft Actor Critic (SAC) [Haarnoja et al., 2018] and recurrent Twin Delayed DDPG (TD3) [Fujimoto et al., 2018]. We replicate the experiments from the POPGym and POMDP-Baselines papers as-is without changing any hyperparameters . We use a single FFM configuration across all experiments, except for varying the hidden and recurrent sizes to match the RNNs, and initialize α , ω based on the max sequence length. See Appendix D for further details.

Table 2 lists memory baselines, Table 3 ablates each component of FFM, Figure 3 contains a summary of training statistics for all models (wall-clock train time, mean reward, etc), and Figure 4

10

3

10

4

10

5

Train Time (ms)

TCN

PosMLP

MLP

Stack

FWP

FART

FFM

S4D

FFM (CPU)

GRU

Elman

LSTM

IndRNN

LMU

DNC

0.0

0.5

1.0

1.5

2.0

Inference Latency (ms)

MLP

Elman

LMU

PosMLP

GRU

LSTM

IndRNN

Stack

TCN

FFM

FWP

FART

S4D

DNC

0

50

100

150

200

Elman

TCN

LMU

IndRNN

MLP

FWP

PosMLP

FART

Stack

FFM

GRU

S4D

LSTM

DNC

0.2

0.0

0.2

0.4

Reward

S4D

PosMLP

DNC

FWP

FART

Stack

LMU

TCN

Elman

LSTM

IndRNN

GRU

FFM

contains a detailed comparison of FFM against the best performing RNN and transformer on POPGym. Figure 5 evaluates FFM on the POMDP-Baselines tasks and Figure 6 investigates FFM interpretability. See Appendix E and Appendix F for more granular plots. Error bars in all cases denote the 95% bootstrapped confidence interval.

Practicality and Robustness: FFM demonstrates exceptional performance on the POPGym and POMDP-Baselines benchmarks, achieving the highest mean reward as depicted in Figure 3 and Figure 5 without any changes to the default hyperparameters . FFM also executes a PPO training epoch nearly two orders of magnitude faster than a GRU (Figure 3, Table 1). Without forgetting, FFM underperforms the GRU, showing that forgetting is the most important inductive prior (Table 3). We use a single FFM configuration for each benchmark, demonstrating it is reasonably insensitive to hyperparameters. All in all, FFM outperforms all other models on average, across three RL algorithms and 52 tasks. Surprisingly, there are few occurrences where FFM is noticeably worse than others, suggesting it is robust and a good general-purpose model.

Explainability and Prior Knowledge: Figure 6 demonstrates that FFM learns suitable and interpretable decay rates and context lengths. We observe separate memory modes for the actor and

AutoencodeEasy

MineSweeperEasy

RepeatPreviousEasy

CountRecallHard

NoisyStatelessPendulumMedium

NoisyStatelessPendulumEasy

NoisyStatelessPendulumHard

ConcentrationHard

LabyrinthEscapeEasy

RepeatFirstMedium

StatelessCartPoleMedium

StatelessCartPoleHard

StatelessCartPoleEasy

RepeatFirstEasy

StatelessPendulumEasy

NoisyStatelessCartPoleEasy

HigherLowerEasy

HigherLowerHard

StatelessPendulumMedium

StatelessPendulumHard

ConcentrationMedium

HigherLowerMedium

MineSweeperHard

NoisyStatelessCartPoleHard

LabyrinthExploreMedium

LabyrinthExploreHard

NoisyStatelessCartPoleMedium

AutoencodeMedium

MineSweeperMedium

RepeatFirstHard

MultiarmedBanditEasy

AutoencodeHard

BattleshipMedium

RepeatPreviousMedium

BattleshipEasy

MultiarmedBanditMedium

MultiarmedBanditHard

ConcentrationEasy

RepeatPreviousHard

CountRecallMedium

LabyrinthEscapeMedium

CountRecallEasy

LabyrinthEscapeHard

Env

-10%

-1%

0%

1%

10%

100%

Reward Relative to GRU

Walker-V

Ant-P

HalfCheetah-V

Walker-P

Hopper-V

Hopper-P

Env

0

1000

2000

3000

Reward

Model+Alg

RNN+SAC

FFM+SAC

RNN+TD3

FFM+TD3

0

50

100

150

200

Epoch

0

100

200

300

400

tω

t

= 64

)

t

= 104

)

0

50

100

150

200

Epoch

0

100

200

300

400

tα

t

= 32)

t

104)

0

50

100

150

200

Epoch

-

0

.

5

-

0

.

4

-

0

.

3

-

0

.

2

Reward

FFM-32,104

FFM-1,1024

GRU

critic, and we can initialize α , ω using prior knowledge to improve returns (Figure 6). This sort of prior knowledge injection is not possible in RNNs, and could be useful for determining the number of 'burn-in' steps for methods that break episodes into fixed-length sequences, like R2D2 [Kapturowski et al., 2019] or MEME [Kapturowski et al., 2023].

Limitations and Future Work

Scaling Up: We found that large context sizes c resulted in decreased performance. We hypothesize that a large number of periodic functions results in a difficult to optimize loss landscape with many local extrema. Interestingly, transformers also employ sinusoidal encodings, which might explain the difficulty of training them in model-free RL. We tried multiple configurations of FFM cells in series, which helped in some tasks but often learned much more slowly. In theory, serial FFM cells could learn a temporal feature hierarchy similar to the spatial hierarchy in image CNNs. In many cases, serial FFM models did not appear fully converged, so it is possible training for longer could solve this issue.

Additional Experiments: Hyperparameter tuning would almost guarantee better performance, but would also result in a biased comparison to other models. We evaluated FFM with on-policy and off-policy algorithms but did not experiment with offline or model-based RL algorithms. In theory, FFM can run in continuous time or irregularly-spaced intervals simply by letting t be continuously or irregular, but in RL we often work with discrete timesteps at regular intervals so we due not pursue this further.

Numerical Precision: FFMexperiences a loss of numerical precision caused by the repeated multiplication of exponentials, resulting in very large or small numbers. Care must be taken to prevent overflow, such as upper-bounding α . Breaking a sequence into multiple forward passes while propagating recurrent state (setting n -k less than the sequence length) fixes this issue, but also reduces the training time efficiency benefits of FFM. We found FFM performed poorly using single precision floats, and recommend using double precision floats during multiplication by γ . We tested a maximum sequence length of 1024 per forward pass, although we could go higher by decreasing the maximum decay rate. With quadruple precision floats, we could process sequences of roughly 350,000 timesteps in a single forward pass with the max decay rate we used. Unfortunately, Pytorch does not currently support quads.

7 Conclusion

The inductive priors underpinning FFM are key to its success, constraining the optimization space and providing parallelism-enabling structure. Unlike many memory models, FFM is interpretable and even provides tuning opportunities based on prior knowledge about individual tasks. FFM provides a low-engineering cost 'plug and play' upgrade to existing recurrent RL algorithms, improving model efficiency and reward in partially observable model-free RL with a one-line code change.

References

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence , 101(1-2):99-134, 1998. Publisher: Elsevier.

Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs. In Proceedings of the 39th International Conference on Machine Learning , pages 16691-16723. PMLR, June 2022. URL https://proceedings.mlr. press/v162/ni22a.html . ISSN: 2640-3498.

Zhihan Yang and Hai Nguyen. Recurrent Off-policy Baselines for Memory-based Continuous Control. Deep RL Workshop, NeurIPS 2021 , 2021.

Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series , 2015.

Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Rapha¨ el Lopez Kaufman, Aidan Clark, Seb Noury, Matthew Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing Transformers for Reinforcement Learning. In Hal Daum´ e III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 7487-7498. PMLR, July 2020. URL https://proceedings.mlr.press/v119/parisotto20a.html .

Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. POPGym: Benchmarking Partially Observable Reinforcement Learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= chDrutUTs0K .

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering Diverse Domains through World Models. arXiv preprint arXiv:2301.04104 , 2023.

Steven Kapturowski, V´ ıctor Campos, Ray Jiang, Nemanja Rakicevic, Hado van Hasselt, Charles Blundell, and Adria Puigdomenech Badia. Human-level atari 200x faster. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/ forum?id=JtC6yOHRoJJ .

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A Study on Overfitting in Deep Reinforcement Learning. arXiv , 2018. URL https://arxiv.org/pdf/1804.06893.pdf .

Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R´ e. Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers. In Advances in Neural Information Processing Systems , volume 34, pages 572-585. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 05546b0e38ab9175cd905eebcc6ebb76-Abstract.html .

Aaron Voelker, Ivana Kaji´ c, and Chris Eliasmith. Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://papers.nips.cc/paper/ 2019/hash/952285b9b7e7a1be5aa7849f32ffff05-Abstract.html .

Sepp Hochreiter and J¨ urgen Schmidhuber. Long Short-Term Memory. Neural Computation , 9 (8):1735-1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735 .

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. Technical Report arXiv:1412.3555, arXiv, December 2014. URL http://arxiv.org/abs/1412.3555 . arXiv:1412.3555 [cs] type: article.

Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5457-5466, Salt Lake City, UT, June 2018. IEEE. ISBN 978-1-5386-6420-9. doi: 10.1109/CVPR.2018.00572. URL https: //ieeexplore.ieee.org/document/8578670/ .

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 , 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html .

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are RNNs: fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning , ICML'20, pages 5156-5165. JMLR.org, July 2020.

Imanol Schlag, Kazuki Irie, and J¨ urgen Schmidhuber. Linear Transformers Are Secretly Fast Weight Programmers. In Proceedings of the 38th International Conference on Machine Learning , pages 9355-9366. PMLR, July 2021. URL https://proceedings.mlr.press/v139/schlag21a. html . ISSN: 2640-3498.

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864 , 2021.

Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R´ e. HiPPO: Recurrent Memory with Optimal Polynomial Projections. In Advances in Neural Information Processing Systems , volume 33, pages 1474-1487. Curran Associates, Inc., 2020. URL https://proceedings. neurips.cc/paper/2020/hash/102f0bb6efb3a6128a3c750dd16729be-Abstract.html .

Michael J Kahana. Computational models of memory search. Annual Review of Psychology , 71: 107-138, 2020. Publisher: Annual Reviews.

Francis Galton. Inquiries into human faculty and its development . Macmillan, 1883.

Bennet B Murdock. A Theory for the Storage and Retrieval of Item and Associative Information. Psychological Review , 89(6):609, 1982. Publisher: American Psychological Association.

Marc W. Howard and Michael J. Kahana. A Distributed Representation of Temporal Context. Journal of Mathematical Psychology , 46(3):269-299, June 2002. ISSN 0022-2496. doi: 10.1006/jmps.2001.1388. URL https://www.sciencedirect.com/science/article/ pii/S0022249601913884 .

Mark Harris, Shubhabrata Sengupta, and John D Owens. Parallel prefix sum (scan) with CUDA. GPU gems , 3(39):851-876, 2007.

Jaap M. J. Murre and Joeri Dros. Replication and Analysis of Ebbinghaus' Forgetting Curve. PLOS ONE , 10(7):e0120644, July 2015. ISSN 1932-6203. doi: 10.1371/journal.pone.0120644. URL https://dx.plos.org/10.1371/journal.pone.0120644 .

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actorcritic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, and others. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015. Publisher: Nature Publishing Group.

Jeffrey L Elman. Finding structure in time. Cognitive science , 14(2):179-211, 1990. Publisher: Wiley Online Library.

Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka GrabskaBarwi´ nska, Sergio G´ omez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, and others. Hybrid computing using a neural network with dynamic external memory. Nature , 538(7626):471-476, 2016. Publisher: Nature Publishing Group.

Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations , 2019.

A Relationship to the Laplace Transform

We often think of RL in terms of discrete timesteps, but the sum in Equation 11 could be replaced with an integral for continuous-time domains, where ˜ x is a function of time:

If we look at just a single dimension of ˜ x , we have

which is equivalent to the Laplace Transform L of a function ˜ x ( τ ) . One way to interpret the connection between our aggregator and the Laplace transform is that the aggregator transforms the memory task into a pole-finding task in the S-plane, a well-known problem in control theory. When we train FFM for an RL task, we are attempting to find the poles and zeros that minimize the actor/critic loss for a batch of sequences.

B Weight Initialization

We find that our model is sensitive to initialization of the α , ω values. We compute the upper limit for α such that memory will retain β = 0 . 01 = 1% of its value after some elapsed number of timesteps we call t e . This can be computed mathematically via

The lower limit is set based on the maximum value a double precision float can represent, minus a small ϵ

Note that if needed, we can choose a smaller lower bound, but the input must be chunked into sequences of length t e and run in minibatches. Since we can compute minibatches recurrently, the gradient spans across all minibatches rather than being truncated like a quadratic transformer. Nonetheless, we did not need to do this in any of our experiments. We set α to be linearly spaced between the computed limits

We initialize the ω terms using a similar approach, with linearly spaced denominators between one and t e

All other parameters are initialized using the default Pytorch initialization.

C Computing States in Parallel with Linear Space Complexity

Here, we show how the recurrent formulation can be computed in parallel. A naive implementation would use O ( n 2 ) space, but using a trick, we can accomplish this in O ( n ) space.

Assume we have already computed hidden state S k -1 for some sequence. Our job is now to compute the next n -k +1 recurrent states S k , S k +1 , . . . , S n in parallel. We can rewrite Equation 11 in matrix

form:

We can write the closed form for the p th row of the matrix, where 0 ≥ p ≥ n -k

Unfortunately, it appears we will need to materialize ( n -k +1) 2 2 terms:

However, we can factor out γ -p from Equation 35 via a combination of the distributive property and the product of exponentials

Now, each γ j is associated with a single x j , requiring just n -k +1 terms:

We can represent each of these rows as a slice of a single n -k + 1 length tensor, for a space complexity of O ( n -k ) or for k = 1 , O ( n ) .

Finally, we want to swap the exponent signs because it provides better precision when working with floating point numbers. Computing small values, then big values is more numerically stable than the other way around. We want to compute the inner sum using small numbers ( γ + results in small numbers while γ -produces big numbers; n -k is positive and k -n is negative). We can factor γ k -n out of the sum and rewrite Equation 40 as

If we let t = n -k , this is equivalent to Equation 13:

We also need to compute 2( n -k +1) gamma terms: γ k -n , . . . γ n -k +1 , resulting in linear space complexity O ( n ) for a sequence of length n .

D Benchmark Details

We utilize the episodic reward and normalized episodic reward metrics. We record episodic rewards as a single mean for each epoch, and report the maximum reward over a trial.

We do not modify the hyperparameters for PPO, SAC, or TD3 from the original benchmark papers. We direct the reader to [Morad et al., 2023] for PPO hyperparameters and [Ni et al., 2022] for SAC and TD3 hyperparameters.

D.1 Hardware and Efficiency Information

We trained on a server with a Xeon CPU running Torch version 1.13 with CUDA version 11.7, with consistent access to two 2080Ti GPUs. Wandb reports that we used roughly 161 GPU days of compute to produce the POMDP-Baselines results, 10 GPU days for the POPGym results, and 45 GPU days for the FFM ablation using POPGym. This results in a total of 216 GPU days. Since we ran four jobs per GPU, this corresponds to 54 days of wall-clock time for all experiments.

D.2 PPO and POPGym Baselines

Morad et al. [2023] compares models along the recurrent state size, with a recurrent state size of 256. For fairness, We let m = 32 and c = 4 , which results in a complex recurrent state of 128, which can be represented as a 256 dimensional real vector. We initialize α , ω following Appendix B for t e = 1024 , β = 0 . 01 . We run version 0.0.2 of POPGym, and compare FFM numerically with the MMERscore from the paper in Table 4.

D.3 SAC, TD3, and POMDP-Baselines

[Ni et al., 2022] does not compare along the recurrent state size, but rather the hidden size, while utilizing various hidden sizes h . In other words, the LSTM recurrent size is twice that of the GRU. We let c = h/ 32 and m = h/c , so that mc = h . This produces an equivalent FFM configuration to the POPGym baseline when h = 128 , with equivalent recurrent size in bytes to the LSTM (or equivalent in dimensions to the GRU). The paper truncates episodes into segments of length 32 or 64 depending on the task, so we let t e = 128 to ensure that information can persist between segments. Thanks to the determinism provided in the paper, readers should be able to reproduce our exact results using the random seeds 0 , 1 , 2 , 3 , 4 . We utilize separate memory modules for the actor and critic, as done in the paper. We base our experiments off of the most recent commit at the time of writing, 4d9cbf1.

E Benchmark Lineplots

We provide cumulative max reward lineplots in Figure 9, Figure 7, and Figure 8 for all the experiments we ran.

1000

0

1000

2000

3000

Max Cumulative Reward

Env = Ant-P

Env = HalfCheetah-V

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Env Steps

1e6

Env = Hopper-P

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Env Steps

1e6

1000

0

1000

2000

3000

Max Cumulative Reward

Env = Hopper-V

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Env Steps

1e6

Env = Walker-P

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Env Steps

1e6

Env = Walker-V

Model

GRU/LSTM

FFM

1000

0

1000

2000

3000

Max Cumulative Reward

Env = Ant-P

Env = HalfCheetah-V

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Env Steps

1e6

Env = Hopper-P

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Env Steps

1e6

1000

0

1000

2000

3000

Max Cumulative Reward

Env = Hopper-V

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Env Steps

1e6

Env = Walker-P

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Env Steps

1e6

Env = Walker-V

Model

GRU/LSTM

FFM

0.50

0.45

0.40

0.35

0.30

0.25

Max Cumulative Reward

Env = AutoencodeEasy

0.51

0.50

0.49

0.48

0.47

0.46

0.45

0.44

0.43

Env = AutoencodeHard

0.50

0.48

0.46

0.44

0.42

0.40

Env = AutoencodeMedium

0.6

0.5

0.4

0.3

0.2

Env = BattleshipEasy

0.7

0.6

0.5

0.4

0.60

0.55

0.50

0.45

0.40

0.35

0.30

Max Cumulative Reward

Env = BattleshipMedium

0.5

0.4

0.3

0.2

0.1

0.0

0.1

0.2

Env = ConcentrationEasy

0.865

0.860

0.855

0.850

0.845

0.840

0.835

0.830

0.825

Env = ConcentrationHard

0.6

0.5

0.4

0.3

0.2

Env = ConcentrationMedium

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Env = CountRecal Easy

0.9

0.8

0.7

0.6

0.5

Max Cumulative Reward

Env = CountRecal Hard

0.9

0.8

0.7

0.6

0.5

0.4

Env = CountRecal Medium

0.0

0.1

0.2

0.3

0.4

0.5

Env = HigherLowerEasy

0.0

0.1

0.2

0.3

0.4

0.5

Env = HigherLowerHard

0.0

0.1

0.2

0.3

0.4

0.5

Env = HigherLowerMedium

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Max Cumulative Reward

Env = LabyrinthEscapeEasy

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

Env = LabyrinthEscapeHard

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

Env = LabyrinthEscapeMedium

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

Env = LabyrinthExploreHard

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Max Cumulative Reward

Env = LabyrinthExploreMedium

0.2

0.0

0.2

0.4

0.6

Env = MineSweeperEasy

0.40

0.35

0.30

0.25

0.20

Env = MineSweeperHard

0.4

0.3

0.2

0.1

0.0

Env = MineSweeperMedium

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Env = MultiarmedBanditEasy

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Max Cumulative Reward

Env = MultiarmedBanditHard

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Env = MultiarmedBanditMedium

0.2

0.4

0.6

0.8

1.0

Env = NoisyStatelessCartPoleEasy

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Env = NoisyStatelessCartPoleHard

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Env = NoisyStatelessCartPoleMedium

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Max Cumulative Reward

Env = NoisyStatelessPendulumEasy

0.1

0.2

0.3

0.4

0.5

0.6

Env = NoisyStatelessPendulumHard

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Env = NoisyStatelessPendulumMedium

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Env = RepeatFirstEasy

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Env = RepeatFirstHard

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Max Cumulative Reward

Env = RepeatFirstMedium

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Env = RepeatPreviousEasy

0.5

0.4

0.3

0.2

0.1

0.0

0.1

0.2

Env = RepeatPreviousHard

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Env = RepeatPreviousMedium

0.2

0.4

0.6

0.8

1.0

Env = StatelessCartPoleEasy

0

50

100

150

200

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Max Cumulative Reward

Env = StatelessCartPoleHard

0

50

100

150

200

Epoch

0.2

0.4

0.6

0.8

1.0

Env = StatelessCartPoleMedium

0

50

100

150

200

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Env = StatelessPendulumEasy

0

50

100

150

200

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Env = StatelessPendulumHard

0

50

100

150

200

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Env = StatelessPendulumMedium

Model

DNC

Elman

FART

FFM

FWP

GRU

IndRNN

LMU

LSTM

MLP

PosMLP

S4D

Stack

TCN

F POPGym Comparisons by Model

We provide Figure 10, Figure 11, Figure 12 showing the relative FFM return compared to the other 12 POPGym models, including temporal convolution, linear transformers, and more.

ConcentrationMedium

LabyrinthEscapeEasy

ConcentrationHard

LabyrinthExploreMedium

HigherLowerHard

HigherLowerMedium

HigherLowerEasy

StatelessCartPoleEasy

StatelessCartPoleMedium

LabyrinthExploreHard

AutoencodeHard

StatelessCartPoleHard

BattleshipMedium

NoisyStatelessCartPoleHard

NoisyStatelessPendulumHard

AutoencodeMedium

RepeatFirstEasy

NoisyStatelessPendulumMedium

NoisyStatelessCartPoleEasy

MineSweeperEasy

RepeatPreviousHard

NoisyStatelessCartPoleMedium

MultiarmedBanditEasy

StatelessPendulumHard

NoisyStatelessPendulumEasy

ConcentrationEasy

BattleshipEasy

MultiarmedBanditHard

StatelessPendulumMedium

StatelessPendulumEasy

AutoencodeEasy

MultiarmedBanditMedium

MineSweeperHard

RepeatPreviousMedium

MineSweeperMedium

RepeatPreviousEasy

LabyrinthEscapeHard

LabyrinthEscapeMedium

CountRecallHard

RepeatFirstHard

RepeatFirstMedium

CountRecallMedium

CountRecallEasy

Env

-100%

-10%

-1%

0%

1%

10%

100%

1000%

Reward Relative to DNC

RepeatPreviousEasy

AutoencodeEasy

MineSweeperEasy

ConcentrationMedium

StatelessCartPoleMedium

StatelessCartPoleHard

StatelessCartPoleEasy

RepeatFirstEasy

LabyrinthEscapeEasy

NoisyStatelessCartPoleEasy

StatelessPendulumMedium

HigherLowerHard

ConcentrationHard

StatelessPendulumEasy

HigherLowerEasy

StatelessPendulumHard

HigherLowerMedium

NoisyStatelessCartPoleHard

NoisyStatelessCartPoleMedium

NoisyStatelessPendulumHard

LabyrinthExploreMedium

MultiarmedBanditEasy

MineSweeperMedium

NoisyStatelessPendulumEasy

MineSweeperHard

NoisyStatelessPendulumMedium

BattleshipEasy

AutoencodeMedium

AutoencodeHard

BattleshipMedium

CountRecallHard

LabyrinthExploreHard

MultiarmedBanditMedium

CountRecallMedium

ConcentrationEasy

RepeatPreviousMedium

RepeatPreviousHard

MultiarmedBanditHard

LabyrinthEscapeMedium

CountRecallEasy

LabyrinthEscapeHard

RepeatFirstHard

RepeatFirstMedium

Env

-10%

-1%

0%

1%

10%

100%

Reward Relative to Elman

ConcentrationMedium

HigherLowerHard

RepeatFirstEasy

StatelessCartPoleEasy

StatelessCartPoleMedium

HigherLowerMedium

StatelessCartPoleHard

ConcentrationHard

HigherLowerEasy

NoisyStatelessCartPoleEasy

RepeatFirstMedium

NoisyStatelessCartPoleHard

CountRecallHard

NoisyStatelessCartPoleMedium

NoisyStatelessPendulumHard

RepeatFirstHard

StatelessPendulumHard

AutoencodeHard

NoisyStatelessPendulumMedium

MineSweeperEasy

BattleshipMedium

AutoencodeMedium

StatelessPendulumMedium

MultiarmedBanditEasy

StatelessPendulumEasy

NoisyStatelessPendulumEasy

CountRecallMedium

AutoencodeEasy

MineSweeperMedium

RepeatPreviousHard

BattleshipEasy

ConcentrationEasy

MineSweeperHard

MultiarmedBanditMedium

MultiarmedBanditHard

RepeatPreviousMedium

RepeatPreviousEasy

LabyrinthEscapeEasy

LabyrinthExploreMedium

LabyrinthEscapeHard

CountRecallEasy

LabyrinthExploreHard

LabyrinthEscapeMedium

Env

-10%

-1%

0%

1%

10%

100%

Reward Relative to FART

CountRecallHard

ConcentrationMedium

StatelessCartPoleEasy

StatelessCartPoleMedium

HigherLowerHard

HigherLowerMedium

HigherLowerEasy

ConcentrationHard

AutoencodeEasy

NoisyStatelessCartPoleEasy

NoisyStatelessCartPoleHard

RepeatFirstHard

StatelessCartPoleHard

AutoencodeHard

MineSweeperEasy

AutoencodeMedium

MultiarmedBanditEasy

NoisyStatelessCartPoleMedium

RepeatFirstMedium

NoisyStatelessPendulumHard

StatelessPendulumHard

RepeatFirstEasy

BattleshipMedium

NoisyStatelessPendulumMedium

RepeatPreviousMedium

RepeatPreviousHard

CountRecallMedium

StatelessPendulumMedium

MineSweeperMedium

MineSweeperHard

MultiarmedBanditMedium

BattleshipEasy

StatelessPendulumEasy

NoisyStatelessPendulumEasy

ConcentrationEasy

MultiarmedBanditHard

RepeatPreviousEasy

LabyrinthEscapeEasy

LabyrinthEscapeHard

CountRecallEasy

LabyrinthExploreMedium

LabyrinthEscapeMedium

LabyrinthExploreHard

Env

-10%

-1%

0%

1%

10%

100%

Reward Relative to FWP

ConcentrationEasy

RepeatPreviousEasy

CountRecallHard

HigherLowerHard

MineSweeperEasy

NoisyStatelessCartPoleHard

RepeatFirstEasy

StatelessCartPoleMedium

StatelessCartPoleHard

StatelessCartPoleEasy

HigherLowerMedium

NoisyStatelessCartPoleEasy

RepeatFirstMedium

HigherLowerEasy

NoisyStatelessCartPoleMedium

StatelessPendulumHard

RepeatFirstHard

AutoencodeHard

AutoencodeMedium

MultiarmedBanditEasy

AutoencodeEasy

BattleshipMedium

MineSweeperMedium

BattleshipEasy

RepeatPreviousHard

MineSweeperHard

StatelessPendulumEasy

MultiarmedBanditMedium

MultiarmedBanditHard

RepeatPreviousMedium

NoisyStatelessPendulumHard

ConcentrationMedium

StatelessPendulumMedium

NoisyStatelessPendulumMedium

ConcentrationHard

NoisyStatelessPendulumEasy

CountRecallMedium

LabyrinthExploreMedium

LabyrinthExploreHard

CountRecallEasy

LabyrinthEscapeEasy

LabyrinthEscapeHard

LabyrinthEscapeMedium

Env

-10%

-1%

0%

1%

10%

100%

Reward Relative to IndRNN

RepeatPreviousMedium

RepeatPreviousHard

MineSweeperEasy

RepeatPreviousEasy

ConcentrationMedium

ConcentrationHard

RepeatFirstEasy

StatelessCartPoleEasy

StatelessCartPoleMedium

StatelessCartPoleHard

StatelessPendulumHard

StatelessPendulumMedium

HigherLowerEasy

HigherLowerMedium

HigherLowerHard

NoisyStatelessCartPoleHard

NoisyStatelessCartPoleEasy

StatelessPendulumEasy

NoisyStatelessPendulumHard

CountRecallHard

AutoencodeHard

LabyrinthEscapeEasy

NoisyStatelessPendulumMedium

AutoencodeEasy

NoisyStatelessCartPoleMedium

BattleshipEasy

BattleshipMedium

AutoencodeMedium

ConcentrationEasy

MineSweeperHard

NoisyStatelessPendulumEasy

MineSweeperMedium

CountRecallMedium

RepeatFirstMedium

MultiarmedBanditEasy

LabyrinthExploreMedium

MultiarmedBanditMedium

MultiarmedBanditHard

LabyrinthExploreHard

LabyrinthEscapeHard

CountRecallEasy

LabyrinthEscapeMedium

RepeatFirstHard

Env

-10%

-1%

0%

1%

10%

100%

Reward Relative to LMU

MineSweeperEasy

RepeatPreviousEasy

CountRecallHard

ConcentrationHard

AutoencodeEasy

ConcentrationMedium

StatelessCartPoleMedium

StatelessCartPoleEasy

RepeatFirstEasy

StatelessCartPoleHard

HigherLowerEasy

HigherLowerHard

NoisyStatelessCartPoleEasy

StatelessPendulumMedium

StatelessPendulumHard

HigherLowerMedium

NoisyStatelessCartPoleHard

NoisyStatelessPendulumHard

StatelessPendulumEasy

NoisyStatelessCartPoleMedium

LabyrinthEscapeEasy

AutoencodeMedium

NoisyStatelessPendulumMedium

RepeatFirstMedium

RepeatPreviousMedium

AutoencodeHard

NoisyStatelessPendulumEasy

RepeatPreviousHard

MultiarmedBanditEasy

BattleshipMedium

MultiarmedBanditMedium

MineSweeperHard

MineSweeperMedium

ConcentrationEasy

CountRecallMedium

MultiarmedBanditHard

BattleshipEasy

CountRecallEasy

RepeatFirstHard

LabyrinthExploreMedium

LabyrinthEscapeHard

LabyrinthExploreHard

LabyrinthEscapeMedium

Env

-10%

-1%

0%

1%

10%

100%

Reward Relative to LSTM

LabyrinthEscapeHard

LabyrinthExploreHard

ConcentrationMedium

LabyrinthExploreMedium

LabyrinthEscapeEasy

HigherLowerHard

HigherLowerMedium

HigherLowerEasy

ConcentrationHard

BattleshipMedium

AutoencodeHard

BattleshipEasy

ConcentrationEasy

MineSweeperHard

AutoencodeMedium

NoisyStatelessCartPoleHard

StatelessCartPoleEasy

MultiarmedBanditHard

MultiarmedBanditEasy

MineSweeperMedium

NoisyStatelessPendulumHard

StatelessPendulumHard

NoisyStatelessPendulumMedium

MultiarmedBanditMedium

RepeatPreviousHard

NoisyStatelessCartPoleMedium

AutoencodeEasy

StatelessPendulumMedium

MineSweeperEasy

NoisyStatelessCartPoleEasy

StatelessPendulumEasy

RepeatFirstEasy

NoisyStatelessPendulumEasy

StatelessCartPoleMedium

RepeatPreviousMedium

StatelessCartPoleHard

LabyrinthEscapeMedium

RepeatPreviousEasy

RepeatFirstHard

RepeatFirstMedium

CountRecallHard

CountRecallMedium

CountRecallEasy

Env

-100%

-10%

-1%

0%

1%

10%

100%

1000%

Reward Relative to MLP

BattleshipEasy

CountRecallHard

ConcentrationMedium

HigherLowerHard

HigherLowerMedium

ConcentrationHard

HigherLowerEasy

StatelessCartPoleEasy

BattleshipMedium

AutoencodeHard

NoisyStatelessCartPoleHard

MineSweeperHard

ConcentrationEasy

CountRecallMedium

AutoencodeMedium

NoisyStatelessPendulumHard

RepeatFirstEasy

MineSweeperEasy

MineSweeperMedium

NoisyStatelessPendulumMedium

StatelessPendulumHard

NoisyStatelessCartPoleMedium

MultiarmedBanditEasy

StatelessPendulumMedium

StatelessCartPoleMedium

LabyrinthExploreMedium

RepeatPreviousHard

AutoencodeEasy

LabyrinthEscapeEasy

NoisyStatelessPendulumEasy

StatelessPendulumEasy

NoisyStatelessCartPoleEasy

MultiarmedBanditMedium

StatelessCartPoleHard

MultiarmedBanditHard

RepeatPreviousMedium

LabyrinthExploreHard

LabyrinthEscapeHard

LabyrinthEscapeMedium

RepeatPreviousEasy

CountRecallEasy

RepeatFirstHard

RepeatFirstMedium

Env

-10%

-1%

0%

1%

10%

100%

Reward Relative to PosMLP

ConcentrationMedium

ConcentrationHard

HigherLowerEasy

HigherLowerMedium

HigherLowerHard

AutoencodeHard

BattleshipMedium

AutoencodeMedium

NoisyStatelessCartPoleHard

RepeatPreviousHard

NoisyStatelessPendulumHard

ConcentrationEasy

BattleshipEasy

StatelessCartPoleEasy

NoisyStatelessPendulumMedium

AutoencodeEasy

NoisyStatelessCartPoleMedium

MineSweeperHard

StatelessPendulumHard

NoisyStatelessPendulumEasy

RepeatPreviousMedium

StatelessPendulumMedium

StatelessPendulumEasy

MultiarmedBanditMedium

MultiarmedBanditHard

MultiarmedBanditEasy

NoisyStatelessCartPoleEasy

MineSweeperMedium

StatelessCartPoleMedium

MineSweeperEasy

StatelessCartPoleHard

RepeatFirstHard

RepeatFirstEasy

RepeatFirstMedium

LabyrinthEscapeEasy

CountRecallHard

RepeatPreviousEasy

LabyrinthEscapeHard

LabyrinthExploreMedium

LabyrinthEscapeMedium

LabyrinthExploreHard

CountRecallMedium

CountRecallEasy

Env

-100%

-10%

-1%

0%

1%

10%

100%

1000%

Reward Relative to S4D

RepeatPreviousEasy

CountRecallHard

ConcentrationHard

ConcentrationMedium

StatelessCartPoleMedium

StatelessCartPoleEasy

RepeatFirstEasy

StatelessPendulumMedium

StatelessPendulumHard

StatelessPendulumEasy

HigherLowerHard

StatelessCartPoleHard

HigherLowerMedium

HigherLowerEasy

NoisyStatelessPendulumEasy

LabyrinthExploreMedium

NoisyStatelessPendulumMedium

NoisyStatelessPendulumHard

NoisyStatelessCartPoleHard

MineSweeperEasy

NoisyStatelessCartPoleEasy

AutoencodeHard

AutoencodeMedium

BattleshipMedium

MultiarmedBanditEasy

CountRecallMedium

NoisyStatelessCartPoleMedium

AutoencodeEasy

BattleshipEasy

MineSweeperHard

MineSweeperMedium

MultiarmedBanditMedium

ConcentrationEasy

RepeatPreviousHard

LabyrinthExploreHard

LabyrinthEscapeEasy

MultiarmedBanditHard

RepeatPreviousMedium

LabyrinthEscapeHard

LabyrinthEscapeMedium

CountRecallEasy

RepeatFirstHard

RepeatFirstMedium

Env

-10%

-1%

0%

1%

10%

100%

Reward Relative to Stack

ConcentrationEasy

ConcentrationMedium

RepeatPreviousEasy

CountRecallHard

MineSweeperHard

ConcentrationHard

StatelessCartPoleMedium

StatelessCartPoleEasy

RepeatFirstEasy

StatelessCartPoleHard

StatelessPendulumEasy

HigherLowerHard

StatelessPendulumHard

LabyrinthExploreMedium

MineSweeperMedium

StatelessPendulumMedium

HigherLowerMedium

HigherLowerEasy

MineSweeperEasy

NoisyStatelessPendulumEasy

NoisyStatelessPendulumMedium

NoisyStatelessPendulumHard

NoisyStatelessCartPoleHard

MultiarmedBanditEasy

MultiarmedBanditMedium

MultiarmedBanditHard

NoisyStatelessCartPoleEasy

AutoencodeHard

BattleshipMedium

AutoencodeMedium

LabyrinthEscapeEasy

BattleshipEasy

CountRecallMedium

NoisyStatelessCartPoleMedium

AutoencodeEasy

LabyrinthExploreHard

RepeatPreviousHard

RepeatPreviousMedium

LabyrinthEscapeHard

CountRecallEasy

LabyrinthEscapeMedium

RepeatFirstMedium

RepeatFirstHard

Env

-10%

-1%

0%

1%

10%

100%

Reward Relative to TCN

G Efficiency Statistic Details

This section explains how we computed the bar plots showing efficiency statistics. We construct the POPGym models and perform one PPO training epoch. The train time metric corresponds to the time spent doing forward and backward passes over the data. One epoch corresponds to the epoch defined in POPGym: 30 minibatches of 65,336 transitions each, with an episode length of 1024, a hidden dimension of 128, and a recurrent dimension of 256. We do this 10 times and compute the mean and confidence interval. Torch GRU, LSTM, and Elman networks have specialized CUDA kernels, making them artificially faster than LMU and IndRNN which are written in Torch and require the use of for loops. We utilize the pure python implementation of these models, wrapping them in a for loop instead of utilizing their hand-designed CUDA kernels. We consider this a fair comparison since this makes the GRU, Elman, and LSTM networks still run slightly faster than IndRNN and LMU (Torch-native RNNs). FFM is also written in Torch and does not have access to specialized CUDA kernels. CUDA programmers more skilled than us could implement a custom FFM kernel that would see a speedup similar to the GRU/LSTM/Elman kernels.

To compute inference latency, we turn off Autograd and run inference for 1024 timesteps, computing the time for each forward pass. We do this 10 times and compute the mean and 95% confidence interval.

To compute memory usage, we utilize Torch's GPU memory tools and record the maximum memory usage at any point during the training statistic. Memory usage is constant between trials.

For reward, we take the mean reward over all environments, split by model and trial. We then report the mean and 95% confidence interval over trials and models.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new memory model for reinforcement learning agents, targeting POMDP setting. The memory (FFM) aggerates past data via an exponentially weighted summation operation where the weights are learnable parameters. The memory is used to compute the Markov state through sophisticated gating mechanisms. The paper evaluates FFM on 2 POMDP benchmarks and shows improvement over baselines such as GRU.\n\nSTRENGTHS:\n- The memory design is motivated by computational psychology \n- The proposed memory is light-weight and fast\n\nWEAKNESSES:\n- The novelty is questionable. The main contribution is Eq. 11, and it is very similar to Eq. 1 in [1]. The difference is that the decay term here is learnable and consists of real and imaginary components. Table 2 reveals that non-learned decay also shows decent performance, which questions the need of learning the decay. \n- The literature review lacks a line of research using MANN on RL [2,3,4]. They are memory models specifically designed for RL. It is fairer to compare FFM with them rather than comparing with general memory models such as DNC or GRU. Please note that general memory models may require heavy hyperparameter tuning and adjustment to make them work on RL, thus comparing with well-established memory-based RL methods is better. For example, Transformer needs some tricks to work on RL [5]. Please consider using this baseline instead of general Transformer models.   \n- The experimental results are not convincing. In POPGym, the gap between FFM and GRU is not clear. The ablation study in Table 2 indicates that the full model does not outperform other configurations significantly, especially when the std is high and only 3 runs are executed. It would suggest that the complicated design might not add much to the performance gain. In Fig. 5, there are many tasks FFM shows similar or worse results than RNN. It questions the benefit of using FFM. \n\n[1] Ba, Jimmy, Geoffrey E. Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin Ionescu. \"Using fast weights to attend to the recent past.\" Advances in neural information processing systems 29 (2016).\n\n[2] Wayne, Greg, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska-Barwinska, Jack Rae et al. \"Unsupervised predictive memory in a goal-directed agent.\" arXiv preprint arXiv:1803.10760 (2018).\n\n[3]  Fortunato, Meire, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomènech Badia, Gavin Buttimore, Charles Deck, Joel Z. Leibo, and Charles Blundell. \"Generalization of reinforcement learners with working and episodic memory.\" Advances in neural information processing systems 32 (2019).\n\n[4] Lampinen, Andrew, Stephanie Chan, Andrea Banino, and Felix Hill. \"Towards mental time travel: a hierarchical memory for reinforcement learning agents.\" Advances in Neural Information Processing Systems 34 (2021): 28182-28195.\n\n[5] Parisotto, Emilio, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg et al. \"Stabilizing transformers for reinforcement learning.\" In International conference on machine learning, pp. 7487-7498. PMLR, 2020.\n\nQUESTIONS:\n- It is unclear why the inductive bias (composite memory and contextual drift) is designed specifically for RL setting. The bias is very general and can be used for both SL and RL. This raises the questions such as whether FFM can show good performance under SL? Why only test it on RL?\n- It is hard to link Eq. (11) to the contextual drift motivation where $\\omega$ should be recurrently computed. \n- It is also unclear why the decay should be complex-valued. The authors should present the explanation for the changes compared to the original motivation Eq. 8-10. \n- The choice of gating for the Cell seems arbitrary. Could you explain the reasons behind these designs, or do you follow any prior papers/ideas?\n\nLIMITATIONS:\nMore investigation is needed on the initialization of the decay. Now, it seems to be heuristic and unclear if it still works for other tasks or settings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-19"}

comments: SUMMARY:
This work introduces a new recurrent neural network layer called Fast and Forgetful Memory (FFM), which, differently from traditional architectures such as GRUs and LSTMs, FFM is explicitly designed for deep reinforcement learning

STRENGTHS:
The authors present a novel and promising memory layer for reinforcement learning, the core ideas of the layer are intuitive. The theoretical foundation is well explained and easy to follow, I didn't spot any errors. Also the empirical analysis is extensive. Authors cover time and space complexity of the layer which is very beneficial to get a general idea of its advantages.

In general I think is a good work and a strong accept

WEAKNESSES:
No major weaknesses that I can tell, just a few minor points:
- In the abstract I would not say so broadly that all model-free approaches are using architectures from SL I think that applies to RNNs but mainstream transformers used in RL were specifically redesigned for this framework
- Line 104, I would explain what traces are
- Line 117, remove brackets from citations that are part of the sentence
- Page 4, I would do an effor to inegrate the footnote in the main text, it would make the section much easier to follow

QUESTIONS:
- Do you foresee any possible issues of instability?
- How long were the temporal dependencies tested in the experiments?

LIMITATIONS:
I think authors did fine work addressing the limitations of their contribution at the end of the paper
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Fast and Forgetful Memory (FFM), a memory model designed for reinforcement learning. FFM is based on theories of composite memory and contextual drift from computational psychology. It consists of two main components: an aggregator and a cell. The aggregator computes complex-valued summaries of inputs, while the cell performs input gating and extracts real-valued outputs from the summaries. Conceptually, the aggregator implements composite memory by blending individual traces and incorporating an exponential forgetting mechanism. It further enables relative-time reasoning by applying a temporal context. On a suite of POMDP tasks from POPGym and POMDP-Baselines, FFM-enabled recurrent RL algorithms outperform those running on standard RNN. Moreover, since FFM belongs to the class of hybrid memory models, which provides equivalent recurrent and closed-form formulae, it achieves linear space complexity and logarithm time complexity with parallelism. This makes FFM an efficient and performant memory model that can be used as a drop-in replacement for RNN in recurrent RL algorithms.\n\nSTRENGTHS:\n- The paper proposes a novel memory mechanism for reinforcement learning by drawing inspiration from the field of computational psychology. It represents its cell state as a collection of traces and features numerous inductive biases for composition, forgetting, and relative-time reasoning. These inductive biases collectively lead to improved performance on RL tasks.\n- The experiments are substantial and thorough, providing ample evidence that FFM can be used as a drop-in replacement for RNN in recurrent RL algorithms. The ablation experiments and analysis provide further insight into the significance of each design choice.\n- The contributions of this paper are significant, especially considering the importance of memory models in partially-observable RL.\n\nWEAKNESSES:\n- The intuition behind some inductive biases is elusive. Please refer to the Questions section.\n\nQUESTIONS:\nBesides drawing inspiration from computational psychology, are there intuitive or formal reasons that the proposed inductive biases lead to better performance on reinforcement learning tasks compared to the various instantiations of RNN? In particular:\n- Why is storing memory as individual traces more desirable than having a single context vector?\n- Is there a benefit to using imaginary temporal context over standard position encoding?\n- Is there evidence that temporal context enables relative-time reasoning?\n\nLIMITATIONS:\nThe authors address the following limitations in their paper:\n- Performance decreases with larger context sizes, presumably because of a larger number of periodic functions resulting in more local extrema in optimization.\n- Limitation in the number of experiments.\n- Repeated multiplication of exponentials can lead to a loss of numerical precision.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to propose an algorithm-agnostic memory model specifically designed for reinforcement learning (RL). The proposed neural network (NN) model is claimed to offer advantages over existing sequence-based memory models like Recurrent Neural Networks (RNNs) and can be seamlessly substituted for RNNs. The key advantage over RNNs is that the model can process a sequence in parallel, making it more efficient compared to traditional RNNs. The main design components of the model include:\nA context drift formula that computes the embedding of sensory information with context.\nA decaying mechanism incorporated into the recurrent formula to compute a summary of the embedding.\nGating mechanisms used to extract the final output from the learned summary of the embedding.\n\nSTRENGTHS:\n1. Appropriate architecture may be suitable for a particular type of tasks. I agree that there might be some interesting special NN architecture that is particularly beneficial for RL tasks. Studying RL-specific NN structure is an important and practically useful research direction. \n\n2. The proposed NN architecture seems to be new.\n\nWEAKNESSES:\nThe main weaknesses are: clarity of presentation, lack of justifications of some design choices, experiments are not persuasive enough. \n\nClarification questions: \n\n1. What is the difference between subscript t and n? I found it to be confusing in eq(9). What should be the relation between n and t there? \n\n2. eq(4) already defined x_n as the embedding of observation-action pair, why eq(9) redefined this notation? \n\nAnd what is hat{x}? It is said that it is the raw sensory information, isn’t it part of observation variables? \n\nWhat is the context variable in RL? Do you simply define omega as a constant one vector? \n\nI strongly suggest the authors to use a particular RL environment to explain all these notations. I found it to be confusing when reading notations in the paper. \n\nJustify some designs: \n\n1. As a new NN architecture,  it is critical to study which component of the proposed methods matters. \nAt least, the paper should study the necessity of exponential decay mechanism, outer product design (eq 9), etc. One possible baseline in the experiments could be simply adding a decaying mechanism in the hidden state of a regular RNN so that it can somehow show the importance of the decay mechanism.  \n\nI have doubts on why decaying is good. Why it makes sense to set up an expiration for each feature unit? In existing literature, it is critical to always maintain a so-called coverage in the state space to learn an optimal policy (please refer to references [1][2][3]). If setting expiration for some feature units, doesn’t it mean manually discard some of the states? \n\n2. The paper should also include discussions with successor features, as the method of computing the hidden state (i.e. eq(7)) is highly similar to how a successor representation is learned. An in-depth study of successor representation may result in better understanding of the proposed architecture. \n\nExperiments. \n\nI found the experiments a bit disappointing. As a general NN architecture, why not also test it on those commonly used RL domains like mujoco, Atari, etc. as mentioned in the abstract, “it is a drop-in replacement for RNN.” There are already existing work using RNN in a variety of RL benchmark tasks. It is better to test the proposed architecture on those well-tested RL domains.  \n\nI would like to emphasize that i don’t think it is necessary to restrict the test on POMDP. In function approximation setting, a MDP can be always viewed as a POMDP. \n\n[1] An equivalence between loss functions and non-uniform sampling in experience replay by Scott Fujimoto et al. \n[2] Understanding and mitigating the limitations of prioritized experience replay by Pan et al. \n[3] regret minimization ER in off-policy RL et al.\n\nQUESTIONS:\nsee weaknesses.\n\nLIMITATIONS:\nna", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-15"}