Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation
Authors: Egor Cherepanov
Nikita Kachaev
Artem Zholus
Alexey Kovalev
Aleksandr Panov
Venue: ICLR 2026 Poster (2026)
Keywords: RL
POMDP
Memory
Classification
Primary area: reinforcement learning
TL;DR: A formal description of the memory types of RL agents and a methodology for conducting an experiment to test the memory.

Abstract: The incorporation of memory into agents is essential for numerous tasks within the domain of Reinforcement Learning (RL). In particular, memory is paramount for tasks that require the use of past information, adaptation to novel environments, and improved sample efficiency. However, the term ``memory'' encompasses a wide range of concepts, which, coupled with the lack of a unified methodology for validating an agent's memory, leads to erroneous judgments about agents' memory capabilities and prevents objective comparison with other memory-enhanced agents. This paper aims to streamline the concept of memory in RL by providing practical precise definitions of agent memory types, such as long-term vs. short-term memory and declarative vs. procedural memory, inspired by cognitive science. Using these definitions, we categorize different classes of agent memory, propose a robust experimental methodology for evaluating the memory capabilities of RL agents, and standardize evaluations. Furthermore, we empirically demonstrate the importance of adhering to the proposed methodology when evaluating different types of agent memory by conducting experiments with different RL agents and what its violation leads to.

Published as a conference paper at ICLR 2026
UNRAVELING THE COMPLEXITY OF MEMORY IN RL AGENTS: AN APPROACH FOR CLASSIFICATION AND EVALUATION
Egor Cherepanoy 1.2, Nikita Kachaev 1.3, Artem Zholus 4.5, Alexey K. Kovalevi Aleksandr I. Panoy1.2 AXXX, MIRIAI, SITMO University Al Talent Hub, "Mila Quebec AT Institute, Polytechhique Montréal cherepanov@axxx.ccm tech
ABSTRACT
The incorporation of memory into agents is essential for numerous tasks within the domain of Reinforcement Learning (RL). In particular, memory is paramount for tasks that require the use of past information, adaptation to novel environments, and improved sample efficiency. However, the term "memory" encompasses a wide range of concepts, which, coupled with the lack of unified methodology for validating agent's memory, leads erroneous judgments about agents' memory capabilities and prevents objective comparison with other memory-enhanced agents. This paper aims streamline the concept of memory in RL by providing practical precise definitions of agent memory types, such as long-term vs. short-term memory and declarative vs. procedural memory, inspired by cognitive science. Using these definitions, we categorize different classes of agent memory, propose a robust experimental methodology for evaluating the memory capabilities of RL agents, and standardize evaluations. Furthermore, we empirically demonstrate the importance of adhering to the proposed methodology when evaluating different types of agent memory by conducting experiments with different RL agents and what its violation leads to.
INTRODUCTION
Reinforcement Learning (RL) effectively addresses problems within the Markov Decision Process (MDP) framework. However, applying RL to tasks with partial observability remains challenging. requiring agents to efficiently process their interaction history (Esslinger et al., 2022; Hausknecht & Stone, 2015; Ni et al., 2021).
In complex environments with noisy observations and long episodes, storing and retrieving important information becomes crucial (Goyal et al., 2022; Graves al., 2016). Yet, the concept of "memory" in RL. literature lacks unified definition. Some works define it as the ability to handle dependencies within fixed context (Esslinger et al., 2022; Ni al., 2023), others the ability to out- of-context information (Parisotto et al., 2020), and in Meta-RL, as the ability to adapt to new environments (Team et al., 2023).
However, in the absence of clear definitions and standardized evaluation protocols, claims about memory capacity RL agents remain vague and often misleading. Memory is frequently attributed to architectural features like recurrence or attention, yet without proper isolation of memory effects, such assumptions can be incorrect. For instance, an agent might appear to exhibit long-term memory simply due to task configurations that allow shortcuts or overlap with short-term context. As result, many empirical evaluations risk conflating different memory mechanisms or failing detect architectural limitations. This hinders progress in developing truly memory-capaie agents and comparing models in a fair and reproducible manner.
In this work, we attempt to unify and clarify the concept of memory in RL agents by treating memory as an intrinsic attribute of memory-enhances agents, directly linking memory type classification to the agent's 's internal mechanisms. These specific memory types short-term declarative vs. procedural - can be rigorously assessed through experiments memory- memory-intensive intensive environments. Our classification, based temporal dependencies and the nature of the recalled infor-
long-term and

---

Published as a conference paper at ICLR 2026
mation, provides structured framework for distinguishing memory types, enabling fair comparisons, diagnosing architectural limitations, and guiding principled improvements.
It is important to clarify that our goal is not to replicate the full complexity of human memory. Rather, we selectively adapt well-established memory concepts from neuroscience such as short-term, long-term, declarative, and procedural memory that already informally used in RL, but lack precise definitions and formal grounding (Fortunato et al., 2020; Kang et al., 2024b; Ni et al., 2023).
In summary, main contributions
follows:
1. We provide formal definitions of key memory types in RL specifically, short-term (STM) vs. long-term (L.TM) and declarative vs. procedural memory grounded in neuroscience and formalized for RL settings Section 4.
2. We introduce a task-level decoupling of Memory Decision-Making (Memory DM) and Meta-RL, clarifying the behavioral role of memory in each category Section 4.
3. We propose a principled experimental methodology for evaluating STM and LTM in Memory DM tasks, including precise criteria for identifying memory boundaries Section 4.2.
4. We show that neglecting the proposed methodology can mislead conclusions about agent memory capabilities, highlighting the importance of proper experimental configuration Section 5.
2 BACKGROUND
2.1 MEMORY OF HUMANS AND AGENTS
Many RL studies reference memory types from cognitive science, such long-term (Lampinen et al., 2021; Ni et al., 2023), working (Graves et al., 2014), associative (Polson, 1975), and episodic memory (Pritzel et al., 2017), but often reduce them to vague temporal categories (e.g., short vs. long-term), with short-term spanning a few steps and long-term hundreds. This oversimplification, ignoring the relative nature of memory, complicates meaningful evaluation. To resolve this, formalize agent memory types and introduce a principled evaluation framework.
2.1.1 MEMORY IN COGNITIVE SCIENCE
Human adaptive behavior depends heavily on memory, which governs how knowledge and skills are acquired, retained, and reused (Parr et al., 2020; 2022). Memory exists in many forms, each of which relies on different neural mechanisms. Neuroscience and cognitive psychology distinguish memory by the temporal scales which information is stored and accessed, and by the type of information that is stored. Abstracting from this distinction, a high-level definition human memory is as follows: "memory- is the ability to retain information and recall
later time".
This definition aligns with how memory is typically understood in RL, and we adopt it to define types of RL agent memory. In neuroseience, memory is classified by timescale and behavioral function, distinguishing short-term memory, lasting seconds, from long-term memory, which can persist for a lifetime (Davis & Squire, 1984). It is also divided into declarative (explicit) and procedural (implicit) forms (Graf & Schacter. 1985): the former involves consciously recalled facts and events, while the latter relates to unconscious skills like riding a bike or skiing. Though these distinctions are well- estalisstbl in neuroscience, RI. requires precise, testable definitions. In what follows, we adapt these cognitive categories into a formal framework suitable for RL agents.
2.1.2 MEMORY IN RL.
Memory in RL encompasses diverse agent capabilities, but its definition varies across stud- ies. In many POMDPs, agents must retain key information to act effectively later within the same environment. This typically involves two kinds of temporal dependencies: 1) within bounded time window (e.g., transformer context (Esslinger et al., 2022; Ni et al., 2023; Yue al., 2024)); 2) beyond the current context, requiring persistent storage or recall (Parisotto et al., 2020; Sorokin et al., 2022). As noted in Section 2.1.1, STM and LTM correspond to different temporal scopes of declarative memory (see Figure 1). In contrast, Meta-RL
2

---

Published as a conference paper at ICLR 2026
Long-term memory &> K
involves procedural memory, enabling agents to reuse skills across tasks (Team et al., 2023) (see Figure 2). However, many works conflate these types, evaluat- ing "long-term memory" solely in Meta-RI. settings based on MDPs (Kang al., 2024a), without iso- lating decision-making from past information. To resolve this, we formalize RL memory types by task structure and temporal dependencies. In this work, we focus on declarative memory, guiding decisions from past observations in the same environment, em- phasizing its short- and long-term forms.
event
agent's context
4. A
t,
K+1
Short-term memory K
event
agent's context
1, - K- +1
te te At
2.1.3 MEMORY AND CREDIT ASSIGNMENT
Figure I: STM vs. LTM. be event start, f, recall time; K K-ek---onn context length, & correlation horizon. If the event lies beyond K, LTM is needed: if within, STM is enough.
Papers on agent memory, especially declarative mem- ory, often distinguish between two forms of temporal reasoning: memory and credit assignment (Mesnard et al., 2020; Ni et al., 2023; Osband et al., 2019). In Ni et al. (2023), memory is defined as recalling a past event the current time, while credit assignment is identifying when reward-relevant actions occurred. Though distinct, both concepts descrihe temporal dependencies between events. Here, we focus on the agent's ability to form such dependencies, treating memory and credit assignment as one. We adopt the general definition from Section 2.1.1, which applies to both, as it captures their shared temporal nature.
knowledge memorization
Declarative Memory
3 RELATED WORKS
event
Mp
Thyuse X Paga
Interest in memory-enhanced RL has led numer- ous architectures (Oh et al., 2016; Lampinen et al., 2021; Fortunato al., 2020) and benchmarks (Morad et al., 2023a; Cherepanov et al., 2025; Osband et al., 2019; Pleines al., 2023), yet the term "mem- ory" remains inconsistently defined and often mis- aligned with what experiments actually test. Some works define memory as retaining recent observa- tions within the same episode either via recurrent states (Hausknecht & Stone, 2015), transformer con- texts (Esslinger et al., 2022; Grigsby et al., 2024), or external stores (Lampinen et al., 2021; Le et al., 2024). Others expand it to long-range dependen- cies through learned state compression (Morad et al., 2023b), key-value recurrent updates (Pramanik et al., 2023; Cherepanov et al., 2024), or spatial memory maps (Parisotto & Salakhutdinov, 2017b). A separate line views memory as cross-episode knowledge transfer, e.g., in Meta-RL (Kang et al., 2024a; Bauer et al., 2023). This variety from within-episode recall to multi-task adaptation - reflects the lack of a shared definition. Our work addresses this gap by introducing a unified taxonomy grounded in temporal dependencies and task structure.
knowledge recall
Procedural Memory
skills memorization
Mi
M₂
skills recall
Mn
Therre X Tagu >
p(M)
Figure 2: Illustration of declarative and pro- cedural memory. Red arrows represent mem- orization steps, blue arrows indicate the recall of task-relevant information.
Among concrete instantiations, Esslinger al. (2022) introduced Deep Transformer O-eetworks (DTQN), which leverage transformer context for partially observable RI. Ni et al. (2023) extended this line with GPT-2 based agents, including DQN-GPT-2 and SAC-GPT-2, which apply attention- based memory online RL control. In the offline setting, Decision Transformer (DT) (Chen et al., 2021) uses a sequence model trained on trajectories for return-con ceturn-conditioned planning, while recurrent baselines such as BC-LSTM provide a contrast between attention-bant based and recurrent memory mechanisms. These models constitute the baselines we evaluate in our experiments.
Ni et al. (2023) further distinguish between memory the ability recall past events and credit assignment identifying when reward-relevant actions occurred. Kang et al. (2024b) build on reconstructive memory (Bartlet & Kintsch, 1995), emphasizing reflection grounded in interaction.
3

---

Published as a conference paper at ICLR 2026
These varied interpretations underscore the need for a unified definition of memory in RL. We address this by formalizing memory types via temporal dependencies and task structure. and proposing framework for empirical evaluation. Concurrently with our study, Yue et al. (2024) introduced memory dependency pairs (p, q) to model recall in demonstrations. While insightful for imitation learning, their approach lacks theoretical treatment of RL memory and does not address the broader laxonomy of agent memory types.
4 MEMORY IN RL
POMDP tasks involving memory fall into two categories: Meta-RL, focused on skill transfer across lasks, and Memory DM, where agents recall past information for future decisions. This distinction matters: Meta-RL relies on procedural memory for rapid adaptation, while Memory DM uses declarative memory to guide decisions within a single environment. Yet many works reduce memory to temporal range, ignoring the behavioral roles that distinguish these types. To formalize Memory DM tasks, we first define the agent's 's context length:
Definition 4.1. Agent context length (K N)-is is the maximum number of previous steps (triplets of (o, a. r)) that the agent can process at time t.
Memory
For example, an MLP-based agent processes one step at time (K = 1), while a transformer- based agent can process a sequence of up to
POMDPs
menorta Haatory ol interactions with
K = Katta triplets, where Katm is determined by attention. Looking ahead, RNNs also have K = 1, but using hidden states allows longer de- pendencies be handled. Using the introduced Definition 4.1 for agent context length, we can introduce a formal definition for the Memory DM framework we focus on in this paper:
Declarative
Procedural
Memory DM framework
Meta-RL framework runsrts stll,, exeriencee
mamertte Satt, localoon, exents
Long-term memory
Short-term memory
inner-loop
E K
K
MDPs
POMDPs
Definition 4.2. Memory Decision-Making (Memory DM) is a class of POMDPs in which the agents decision-makinn process at time f. is based on the history ha-t-1 = {(o, ds r=)) [-1
Long-term memory
Short-term memory
<> K
S K
Figure 3: Classification of memory types of RL agents. While the Memory DM framework con- trasts with Meta-RI., its formalism can also de- scribe inner-loop tasks when they are POMDPs. hu.t- of length t to action a,, maximizing the expected cumulative reward within single POMDP
if t > O otherwise = 2. The objective is to determine an optimal policy N² (at Ot, 1o.t-1) that maps the current observation Og and history
environment Mp: J = ET
where T episode duration, ? E [0, 1] discount factor. 7re
In the Memory DM framework (Definition 4.2), memory refers to the agent's ability to recall information from the past within single environment and episode. In contrast, in the Meta-RL framework (Definition 4.3), memory involves recalling information about the agent's behavior from other environments previous episodes:
Definition 4.3. Meta-RL- - is a class of POMDPs where the agent learns to learn from its past experiences across multiple tasks and memorize the common patterns and structures facilitate efficient aduptation to new lasks. Lei D = (T,Mi (TMMa 1==0 is all of the data of H episodes of length T collected in the MDP M; p(M). A Meta-RL algorithm is function fo that maps the
data D to a policy Th, where & = fo(D). The objective to determine an optimal fo: j
ED
G. for fo.
Σ
where Gz(T) - discounted return in the MDP M,
LEDIR
index of the first episode during the trial in which return counts towards the objective (Beck et al., 2024).
To operationalize the distinction between memory types in RL, we translate the neuroscience concepts of declarative and procedural memory (Section 2. into measurable task-level criteria: Definition 4.4 (Declarative and Procedural memory in RL). Let nenve be the number of training environments and Deps the number of episodes per environment. Then,
4

---

Published as a conference paper at ICLR 2026
1. Declarative Memory- rype of agent memory when agent transfers its knowledge within a single environment and across a single episode within that environment:
Declarative Memory
<
Nenve
neps
1.
(I)
2. Procedural Memory - a type of agent memory when an agent transfers its skills across multiple environments multiple episodes within single environment:
Procedural Memory
Reps
>>
In this formulation, knowledge refers to observable, environment--pecifie information such object locations or facts used within single episode. Skills, in contrast, are policies reused across tasks or trials. Accordingly, Memory DM primarily evaluates declarative memory, while Meta-RL settings test procedural memory (see Figure 3).
Having distinguished declarative and procedural memory, we now examine the temporal structure of memory in the Memory DM framework, focusing on its division into short-term and long-term forms. Definition 4.5 (Memory DM types of memory). Let K be the agent context length, ozf. for, 0g. rel relizte 1f++ an event of duration At that begins at = te and ends at t = te + Åt, and peecision-making point (recall) at time t- = by based on the current
Be, (a = observation and information about the event a Let also & = tt At+ 1 be the correlation horizon, i.e. the minimal time delay between the event a&t that supports the decision-making and the moment of recall of this event Be.- Then,
1. Short-term memory (STM) an agent's ability to use information about local correlations from the past within the context of length K at decision time:
Bt, = (or, V -
2. Long-term memory (LTM)- an agent ability to utilize information about global correlations from the past outside of the agent context of length K, during decision-making:
3e, (a At) = (or, o. =t,
t, - - A
An illustration for the definitions of classifying Memory DM tasks into LTM and STM from Defini- tion 4.5 is shown in Figure 1.
The two definitions of declarative memory encompass all work related to Memory DM tasks, where decisions are based on past information. Meta-RL consists of an inner-loop, where the agent interacts with the environment M ~ p(M), and an outer-loop for transferring knowledge between tasks. Typically, M is an MDP that doesn't require memory, serving only the outer-loop, which is what "memory" refers to in Meta-RI. studies.
The tasks which the agent makes decisions based on interaction histories in the inner-loop are not named separately, since the classification of Meta-RL task types (multi-task, multi-tast tast shot, and single-task) based solely on outer-loop parameters (Nenos and Neps) and does not consider inner-loop task types. However, we can classify the agent's memory for these tasks as declarative STM or LTM (Figure 3).
We introduce an additional decoupling of Meta-RI. task types into green (with POMDP inner-loop tasks) and blue (with MDP inner-loop tasks). In the green case, the agent's memory is required for both skill transfer in the outer-loop and decision-making from interaction histories in the inner-loop, and within the inner-loop can be considered a Memory DM. In the blue case, memory is needed only for skill transfer. While this paper focuses Memory DM tasks, the terminology enables further classification of Meta-RL tasks, with POMDP sub-classes highlighted in green. The proposed classification of tasks requiring agent memory is shown in Table 1.
4.1 MEMORY-INTENSIVE ENVIRONMENTS
To effectively test Memory DM agent's 's use of short-term and long-term memory, it is crucial to design appropriate experiments. Not all environments are suitable for assessing agent memory; for
5

---

Published as a conference paper at ICLR 2026
example, omnipresent Atari games (Bellemare et al., 2013) with frame stacking MuJoCo control tasks (Fu et al., 2021) may yield unrepresentative results. To facilitate the evaluation of agent memory capabilities, we formalize the definition of memory memory-intensive environments:
Table I: Classification of tasks requiring agent memory based on our definitions: green indicates tasks described by the proposed definitions of LTM and STM, while blue indicates those that are not. Meta-RI. tasks with a POMDP inner-loop marked green they can be classified as Memory DM tasks. POMDPI indicates a Memory DM task considered as an inner-loop task without outer-loop.
Theaara Nage
POMDP
Inner-lopp
task
Memory
Tasks that require agent memory
Definition 4.6 IMemory-Intensive Environments). Let .Mp be POMDP, and lel = =En
Memory DM
LTM <>
STM 5
((t,-te-At- - denote the set of correlation horizons for all event- recall pairs (a )t.)). Then Mr is a memory-intensive environment, de- noted Mp, if and only if: min, &n > 1.
Memory
POMDP
Dec.
Long-tern memory task
Short-term memory task
Meta-RL: Outer-loop
and inner-loop memory
LTM
STM
E> K
E K
Mota-RL
POMDP
Proc.
Single-task
Single-task
Multi-task
Multi-task
Corollary 1. A task corresponds to an MDP (i.e., is Markovian) if and only if all correlation horizons are trivial: max =
Meta-RL
POMDP
Proc.
2cro-sbot
zero-shot
>1
Meta-RL
POMDP
Proc.
Multi-rask
Multi-task
Meta-RL: Outer-loop
memoor only
memory
memory
{=
1
{=
>1
Meta-RL
MDP
Proc.
Single-task
Single-task
Proof. In an MDP, the optimal action depends only on the current state (or >1 observation), i.e., no past information
Meta-RL
MDP
Proc.
Multi-task
Multi-task
zero-sbot
zero-shol
>1
Meta-RL
MDP
Proc.
Multi-rask
Multi-task
is needed. This implies &n = 1 for all event-recall pairs, hence max = 1. Conversely, if
max &n = 1, then no decision depends on events beyond the current step, satisfying the Markov property.
Using the definitions of memory-intensive environments (Definition 4.6) and agent memory types (Definition 4.5), we can configure experiments to test short-term and long- long-term memory in the Memory DM framework. Notably, the same memory-intensive environment can validate both types of memory, as outlined in Theorem 2:
Theorem 2 (On the context memory border). Let Mp be a eeemory-intensiie environment and K be an agent's 's context length. Then there exists context memory border K 2 1 such that if K & K then the environment Mp is used to validate exclusively long-term memory in Memory DM framework:
k 2 1:VK E [1, K K min
Proof. Let K min E- 1. Then't V K < K guaranteed that no correlation horizon is in the agent history h-K+1:t- hence the context length K I min E - generates the LTM problem exclusively. Since context length cannot be negative or zero, it turns out that S K S was required to prove.
K min E 1, which
The following result, though intuitive, formalizes a practical criterion for isolating long-term memory evaluation by constraining the agent's context window. serves as the foundation for configuring experiments in the Memory DM framework. According to Theorem 2, in memory- intensive environment Mp, the value of the context memory border K can be found as
min {(t, -t.-ta At At+ 1t)+)))) -.
K = min
(4)
Using Theorem 2, we can establish the necessary conditions for validating short-term memory: 1) Weak condition to validate short-term memory: K K < max E, then the memory-intensine

---

Published as a conference paper at ICLR 2026
Algorithm Algorithm for setting up an experiment to test long-term and short-term memory in Memory DM framework.
Require: MM memory- intensive environment; (( memory mechanism.
1. Estimate the number event-recall pairs in the environment (Definition 4.6).
1. n = 0 - Environment is not suitable for testing long-ten long-term and short-tern term memory.
n 2 Environment suitable for testing long-term and short-lerm memory.
2. Estimate context memory border K (4).
A event-recall pair (B(a), a): find corresponding &,, [1...].
2. Determine K as K min =\t = min ho - min [t, - At 1)x) -1
3. Conduct an appropriate experiment (Definition 4.5).
1. To test short-term memory set K K.
2. To test long-term memory set K S K S Kest = H(K).
4. Analyze the results.
environment Mp is used to validate both short-term and long-term memory. 2) Strong condition to validate short-term memory: if max K, then the memory ientoy-intensive environment Mp is used to validate exclusively short-term memory.
According to Theorem 2, if K e K], none of the correlation horizons ç will be in the agent's context, validating only long-term memory. When K < K max S T 1, long-term memory can still be tested, but some correlation horizons & will fall within the agent's context and won't be used for long-term memory validation. In such a case it is not possible to estimate long-term memory explicitly. When K max all correlation horizons & are within the agent's context,
validating only short-term memory. Summarizing the obtained results, the final division of the required agent context lengths for short-term memory and long-term memory validation is as follows: (i) K E [] + validating LTM only; (ii) K E (K, max E) = validating both STM and LTM; (iii) K E [max E, oo) + validating STM only.
4.2 LONG-TERM MEMORY IN MEMORY DM
As defined in Definition 4.5, short-term Memory DM tasks arise when event-recall pairs in Mp fall within the agent's ' context (E S K), allowing decisions based on local correlations. This holds regardless of how large K is. Examples include (Esslinger et al., 2022; Grigsby et al., 2024; Ni et al., 2023). Validating STM is simple: increase K. In contrast, testing long-term memory requires more care and is typically more informative.
Memory DM tasks requiring long-term memory occur when event-recall pairs in the memory- intensive environment Mp are outside the agent's context ($ K). In this case, memory involves the agent's s ability to connect information beyond its context, necessitating memory mechanisms (Definition 4.7) that can manage interaction histories longer than the agent's 's base model can handle.
Definition 4.7 (Memory mechanisms). Let the agent process histories h4- h4-K+1:t of length K at the current time where K defined as a function that, for fixed K, allows the agent to process sequences of length Keff 2 K, ie., to establish global correlations out of context, where Keff the effective context.
agents context length. Then, memory mechanism μ(K) NN + N is
μ(K) Keff 2 K.
(5)
Memory mechanisms are key to LTM tasks by recalling out-of- of--te-onttett information in Memory DM.
Example of memory mechanism. Consider an agent based on RNN architecture that can process K = 1 triplets of tokens (o,, ae Tt) at all times f. By using memory mechanisms μ(K) (c.g.. in Hausknecht & Stone (2015)), the agent can increase the number of tokens processed in single step without expanding the context size of its RNN architecture. Therefore, if initially in memory- intensive environment Mp : K 1, it can now be represented as Mp < Kafs =(K). Here, the memory mechanism μ(K) refers the RNNs recurrent updates its hidden state.
7

---

Published as a conference paper at ICLR 2026
Thus, validating agent's ability solve long-tent tengttermin memory problems in the Memory DM framework reduces to validating the agent's 's memory mechanisms (K)). To design correct experiments in such case, the following condition must be met:
Mp KSKK <<S Kafl ((K)
(6)
According definitions, agents with memory mechanisms in the Memory DM framework that solve LTM tasks can also handle STM tasks, but not vice versa. The algorithm for setting up experiments to test an agent's STM or LTM is outlined in Algorithm
4.3 EXAMPLE E AND ESTIMATES
K
15,5 15
K 5.6 15
K 5.6
Following the proposed method- ology (Algorithm 1), we estimated the sets of correla- tion horizons and minimal recall distances & for range of popular memorry-ntenssee tasks (Table Table 2), including Passive T-Maze (Ni et al., 2023), Sinigrid-Memon (Chevalier- Boisvert et al., 2023), ViZDoom- al.,
aeuum
BTON DQN-5F73
DOM
EX
14 Namtop
-2.0g
Theestep
T-Maze
K
207. 207
103.
207
K
103. 103
Two-Colors (Sorokin 2022), Memory Maze (Pa- sukonis et al., 2022). Memory Cards (Esslinger et al., 2022), Mortar Mayhem and Mystery Path (Pleines et al., 2025), POP. Gym-Autoencou and POPGym- RepeatPrevious (Morad et al., 2023a).
25
-25
-35
64
Theashp
38 32
16
64
Theebee
26
6.4
14
POPGym-Autoencooe
K = 103, = 103
K 51, c 103
K = 51, 6 51
DTCH C0N-
5T-N
DTCM
304-2972 2072
0014 2272
-2
Theestop
Tlenstap
POPGym- RepeatPrevious
Figure 4: Performance of Online RI. agents "with memory" across different memory configurations. Each row shows a specific environment: T-Maze, POPGym-Autoeneode and POPGym- RepeatPrevious, with varying agent context length and cor- relation horizons &. The STM = LTM transitions reflect the relative nature of the settings to test memory, depending on both agent and environment parameters.
Example: Testing Memory in Passive T-Maze In Passive T- Maze, the agent sees cue at the start of a corridor and must turn correctly the junction. The episode lasts T = L+1. where L the corridor length. Using Al- gorithm 1: 1) There's one event- recall pair (n = 1), so the task suits both STM and LTM. 2) The event lasts one step (At = 0), so E T, and K T 1. 3) Varying T or context size K lets test STM (if K > K) LTM (if K & K S ((K)). While K K is enough in theory, choosing smaller K better reveals memory mechanism effects.
EXPERIMENTS
We evaluate memory-enhanced RL agents using the Memory DM framework distinguish short- vs. long-term memory. Our experiments stress the need for proper methodology (Algorithm 1) and show how poor setups can misrepresent memory use. We test four memory-intensive tasks: Passive T-Maze and SinigriddMemory (cue recall), and POPGym-Autoencode and RepeatPrevious (observation reconstruction and action repeat), all requiring recall over time. In the online setting, we evaluate DTQN (Esslinger et al., 2022), DQN-GPT-2, and SAC-GPT-2 (Ni et al., 2023) with attention- based memory. Offline, we test DT (Chen et al., 2021) and BC-LSTM to compare attention vs. recurrence. In all cases, we vary agent context K and task horizon to isolate memory types and reveal model limitations.
8

---

Published as a conference paper at ICLR 2026
5.1 PITFALLS OF NAIVE MEMORY TeSTS
Table 2: Correlation horizons & and LTM thresh- olds K for popular memory-intensive tasks. is corridor length, T is episode length. (f) and (v) denote fixed and variable setups. POPGym entries show values for the easy setting; for easy / medium / hard, becomes (2. 4, /04/208/33 /312) for Autoencode and (5/33/ 33/ /63 for RepeatPrevious.
Proper evaluation of memory in RL agents re- quires distinguishing STM from LTM by ac- counting for correlation horizons &. Without this, STM and LTM effects blur, misrepresent- ing agent capacity. We illustrate this with SAC- GPT-2 in Minigrid-Memory under two setups: (i) fixed L 21
Task
E
LIM
22), and (ii) variable L E E [7, 22),, testing STM (K = 22) and LTM (K = 14) settings. As shown in Figure 5, the variable setup gives high success for both set- tings, implying good memory.
Passive T-Maze
(L 1]
L+1
L+1
Minigrid Memory (f)
[1+1 1
L+1
L+1
Minigrid-Mo Memory (V)
7. L++1
ViZDoom- Two Colors
2,2055
Memory Maze 9x9
28, 1000)
28
28
Memory Maze 15x15
[4,,4000
45
45
Memory Cards Mortar Mayhem (finite)
2. 2.
But in the fixed case, LTM fails, revealing the agent's true limit. Mixed-horizon tasks can hide LTM deficits- only fixed & >> K setups expose them. Proper LTM evaluation requires control- ling the correlation horizon & relative to the agent's 's context K. Without this, STM effects may dominate and misrepresent the agent's memory type. Our methodology provides a principled way to avoid this confusion.
38, 218]
38
38
Mystery Puth (finite)
26]
JOPGym-Autoencee
2. 104]
JiPGyy--ReeentPrvvioe
(5)
5.2 THE RELATIVE NATURE OF AN AGENT'S MEMORY
According to Algorithm 1, the experimental setup for testing agent memory types (LTM and STM) depends on two parameters: the agent's 's context length K and the context memory border K, which in turn is determined by the environment's correlation horizon {. Verifying LTM or STM requires adjusting K or & while keeping the other fixed. This section outlines how these parameters interact in memory testing. An agent's 's memory cannot be defined isolation it arises from the interplay between its context K and the environment's horizons &. Thus, the same agent may exhibit either STM or LTM behavior depending on the task setup.
We test DTQN and DQN-GPT-2 in three memory- -intensie tasks: Passive T-Maze, POPGym Autoen- code, and RepeatPrevious - by varying K and & to simulate STM (E S K) and LTM (E > K); as shown in Figure 4, performance is high when ę s K but drops sharply for & K, confirming that long-range dependencies require explicit memory mechanisms. These results confirm memory is relative: LTM depends on both temporal distance and agent design. Without controlling K and E. memory claims are unreliable. Our K-C framework ensures consistent, interpretable evaluation.
5.3 GENERALIZATION ACROSS SEQUENCE LENGTHS
Evaluating memory in RL agents requires dis- tinguishing true long-term memory (LTM) from memorization within fixed context. To demon- strate this, we test DT and BC-LSTM on T- Maze: both are trained on specific corridor lengths and evaluated on both seen and longer, unseen ones. This setup tests whether agents can recall cue information beyond training range an LTM indicator. Figure shows success heatmaps across training and validation lengths. The diagonal indicates in-distribution perfor- mance; extrapolation lies to the right.
1.0
Success Rate
0.8
0.6
22 variable mode
0.4
K 14 variable mode
0.2
22 fixed mode
14 fixed mode
08.
0.2
0.4 Timestep
0.8
x10
Figure 5: SAC-GPT-2 in Minigrid-Memory (L 21) with short- (K = 22) and long-term (K
While both models process sequences and are labeled memory-enhanced, our framework reveals key differences. DT relies on a fixed attention window and operates with short-term memory, while LSTM uses a recurrent state, enabling true LTM. T-Maze results expose this gap: DT performs well when validation lengths
14) memory setups. Variable mode (green) masks memory limits; fixed mode (red) reveals failure at == 14, demonstrating lack of long-term memory -made evident by evaluation method.
9

---

Published as a conference paper at ICLR 2026
stay within context but fails for >> 90, whereas BC-LSTM generalizes to much longer sequences, demonstrating effective LTM. If evaluated only on shorter validation lengths, DT may appear stronger, masking memory limitations. Both perform well for lengths < 150, but at training length 300, DT scores 100%, while BC-LSTM drops to 0.87. For longer training (600, 900), BC-LSTM collapses, DT remains high- misleadingly favoring STM.
DT succeeds only when the correla- tion horizon satisfies whenever Ç > K, which demonstrates that DT lacks LTM. In contrast, BC- LSTM, on the horizons where it trains reliably, can exploit dependencies be- yond the training range and therefore exhibits LTM within those regimes. Our framework makes this distinc- tion precise by separating architec- tural limits from memory function, en-
Medel success Rats Possive TiMaot Incontsoni
K, but fails
116
32:9
(a) DT agent heatmap
Mude Succesu Nals Tasseen
Labtad
suring that DT is correctly identified as STM agent, while BC-LSTM is capable of LTM on the ranges where learns effectively (i.e. without van- ishing gradients problem (Trinh et al., 2018)).
EITAnOE
134
1009
(b) BC-LSTM agent heatmap
Figure 6: Generalization on Passive T-Maze. Each heatmap shows success rates for (a) DT and (b) BC-LSTM across training (vertical) and validation (horizontal) sequence lengths. DT succeeds only when validation S training, show- ing short-term memory limits. BC-LSTM generalizes beyond training, indicating strong long-term memory.
CONCLUSION
We propose a unified framework for classifying and evaluating memory in RL agents, grounded in neuroscience-
inspired definitions of short- and long-term declarative memory. By introducing the concept of correlation horizon and formalizing memory-intensive environments, we enable precise evaluation of agent memory. Our methodology reveals key differences between architectures: transformers like DTQN or DT rely mainly on short-term memory. while recurrent models such as BC-LSTM exhibit long-term memory. Experiments on T-Maze, MiniGrid, and POPGym confirm the need for proper setups to avoid misleading conclusions. The framework clarifies how memory mechanisms shape behavior and could be extended include additional systems from cognitive science, such as working or episodic memory, and to explore whether new types emerge in complex RL tasks.
As direction for future work, would be valuable to study adaptive dynamic updating of memory representations, since most existing work focuses primarily on memorization and retention rather than on how agents revise stored information over time.
REPRODUCIBILITY STATEMENT
We have taken several measures to ensure the reproducibility of our results. Model details: The formalization of our framework covering Memory DM, STM/LTM, and the correlation horizon provided in Section 4, with precise definitions in Definition 4.1, Definition 4.2, and Definition 4.5, Theoretical results: Assumptions
and the experimental configuration procedure in Algorithm and complete statements (including the definition of memorryintensive environments) are given in Definition 4.6, and key results with proofs appear in Theorem and its accompanying discussion. Experimental setup: Tasks, training procedures, and evaluation protocols are reported in Section 5, with validation protocol details in Section F.I, hyperparameters in Section F.I, and environment descriptions in Appendix F. Baselines: Baseline selections and configurations are documented in Section 5, with their hyperparameters listed in Section F.1. Code and data: An anonymous repository containing source code, training scripts, and configuration files submitted as supplementary material. Together, these resources allow for full replication of our theoretical analyses and empirical results.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Frederic C. Bartlett and Walter Kintsch. Remembering: A Study in Experimental and Social Psychol- ogy. Cambridge University Press, 2 edition, 1995.
Jakob Bauer, Kate Baumli, Feryal Behbahani, Avishkar Bhoopchand, Nathalie
Michael Chang, Natalic Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks- Sook-Thhmpson, Hannah Openshaw, Jack Parker- Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Satinder Singh, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei M Zhang. Hluman-timescall adaptation in open-ended task space. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 1887-1935. PMLR, 23-29 Jul 2023. URL https: papp:://prceediiin
mlr press v202/222222222 http//aaerrraa... .htmll...
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of mett-reeinforeement learning, 2024. URL https://aaxiv: org/ abss
2301. 08028.
Philipp Becker, Niklas Freymuth, and Gerhard Neumann. Kalmamba: Towards efficient probabilistic state space models for rl under uncertainty, 2024. URL htttps:/aaxiv..eo
/2406.
15131.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 50044110777 2021.
Egor Cherepanov, Alexey Staroverov, Dmitry Yudin, Alexey K. Kovalev, and Aleksandr Panov. Recurrent action transformer with memory. arXiv preprint arXiv:2306.09459, 2024. URL
htttp:://arxiv.org////
attgg//bb///// 2306. 09459.
Egor Cherepanov, Nikita Kachaev, Alexey Kovalev, and Aleksandr Panov. Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning. In 7th Robot Learning Workshop: Towards Robots with Human-Level Abilities, 2025.
Maxime Boissett Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid miniworld: Modular customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.13831, 2023.
Hasker Davis and Larry Squire. Davis hp, squire Ir. protein synthesis and memory: a review. psychol bull 96: 518-559. Psychological bulletin, 96:518-59, 11 1984. doi: 10. 00.1037/0033-2999.9.33 299..96....11
Ben Deverett, Ryan Faulkner, Meire Fortunato, Gregory Wayne, and Joel Leibo. Interval timing in deep reinforcement learning agents. Advances in Neural Information Processing Systems, 32, 2019.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI:: Fast reinforcement learning via slow reinforcement learning, 2016. URL htRlsttp:: ptttttpss//axxi..ooo
Kevin Esslinger, Robert Platt, and Christopher Amato. Deep transformer genetworks for partially observable reinforcement learning. arXiv preprint arXiv:2206.01073, 2022.
Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomènech Badia, Gavin Buttimore, Charlie Deck, Joel Z Leibo, and Charles Blundell. Generalization of reinforcement learners with working and episodic memory, 2020. URL https: /ttss///axxxi www.aawwwwwwaaaaaeaeaoaaooooeere abs /1910. 13406.
11

---

Published as a conference paper at ICLR 2026
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2021.
Anirudh Goyal, Abram L. Friesen, Andrea Banino, Theophane Weber, Nan Rosemary Ke, Adria Puig- domenech Badia, Arthur Guez, Mehdi Mirza, Peter C. Humphreys, Ksenia Konyushkova, Laurent Sifre, Michal Valko, Simon Osindero, Timothy Lillierap, Nicolas Heess, and Charles Blundell. Retrieval-augmenter reinforcement learning, 2022. URL https: /ttp/arxivvv.////////aa abs /2002. 08417.
P. Graf and D.L. Schacter. Implicit and explicit memory for new associations in normal and amnesic subjects. Journal of Experimental Psychology: Learning. Memory, & Cognition, 11:501-518, 1985.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines, 2014. URL https: / wawxiv.org/aaaaa...ooo wawpg///////////ee/oo///eeaaeee/ httbs///41.....
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwińska, Sergio Gómez, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Badia, Karl Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using neural network with dynamic external memory. Nature, 538, 10 2016. doi: 10. 00110///
Jake Grigsby, Linxi Fan, and Yuke Zhu. Amago: Scalable in-context reinforcement learning for adaptive agents, 2024. URL https: //wxivvvvvvvv//vv////v/////ea/aa wwwg/orggss.....o. 2390.09971..
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752, 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, arXiv preprint arXiv:2111.003900 2021.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution, 2018. URL
https: : / aaxii.ivvvvv///vv orgg wttg//bs////ooo.eoogog/g///ea /809 909. 01999.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamies for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2555-2565. PMLR, 09-15 Jun 2019. URL https: http///rcceennssseeeeeeep mlr presspsssss....................e httess/99//haa fnerl 9a html.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps, 2015.
Jikun Kang, Romain Laroche, Xingdi Yuan, Adam Trischler, Xue Liu, and Jie Fu. Think before you act: Decision transformers with working memory, 2024a. URL https: : /atxxaxvvvvoooo///a////a org/ abs/ 2305. 1638.
Yongxin Kang, Enmin Zhao, Yifan Zang, Lijuan Li, Kai Li, Pin Tao, and Junliang Xing. Sample efficient reinforcement learning using graph-based memory reconstruction. IEEE Transactions on Artificial Intelligence, 5(2):511772,, 2024b. doi: 10. 10.11099AA..222332266666
Andrew Lampinen, Stephanie Chan, Andrea Banino, and Felix Hill. Towards mental time travel: hierarchical memory for reinforcement learning agents. Advances in Neural Information Processing Systems, 34:28182-4 18-22229555. 28195, 2021.
Hung Le, Kien Do Dung Nguyen, Sunil Gupta, and Svetha Venkatesh. Stable hadamard memory: Re- vitalizing memory augmeeee agents for reinforcement learning. arXiv preprint a:X1v:2410.1013.. 2024.
Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning, 2023, URL https: //tasi///vvvv//// orgg alos/ 2303 03982.
Luckeciano C. Melo. Transformers are meta- meta-reeinforcemeet learners, 2022. URL https: //axiiv.
bTg//bbs2206. 06614.
12

---

Published as a conference paper at ICLR 2026
Thomas Mesnard, Théophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Arthur Guez, et al. Counterfactual credit assignment in model-free reinforcement learning. arXiv preprint 2011:2011.00444 2020.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner, 2018. URL https: // /awriiiiii...///i.//...aaaaaaav. wwwoorggsss... 707. 43141.
Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. Popgym: Benchmarking partially observable reinforcement learning, 2023a. URL https: //axiv.. org/ abs//abs//t 2303. 01859.
Steven Morad, Ryan Kortvelesy, Stephan Liwicki, and Amanda Prorok. Reinforcement learning with fast and forgetful memory. Advances in Neural Information Processing Systems, 36:72008-72029, 2023b.
Steven D. Morad, Stephan Liwicki, Ryan Kortvelesy, Roberto Mecca, and Amanda Prorok. Graph convolutional memory using topological priors, 2021. URL 2106. 14117.
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free can be a strong baseline for many pomdps. arXiv preprint urXiv:2//0005000 2021.
Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon. When do transformers shine in RL? decoupling memory from credit assignment. Thirty-sevetth Conference on Neural Information Processing Systems, 2023. URL https: htttp:://openreveew..t net http://www.e/touummmtteaea forum?idd APGXENKt 6h.
Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft, 2016. URL http://:////.xi...///a//b///////
0.05099228
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforce- ment learning. arXiv preprint arXiv: 908. 03568, 2019.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning, 2017a. URL httpp:://axxi....../// org/ abs/ 08360.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. arXiv preprint
2017b.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark. Seb Noury, al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp. 7487-7498. PMLR, 2020.
Thomas Parr, Rajeev Vijay Rikhye, Michael Halassa, and Karl J Friston. Prefrontal computation as active inference. Cerebral Cortex, 30(2):682-69) 2020.
Thomas Parr, Giovanni Pezzulo, and Karl J Friston. Active inference: the free energy principle in mind, brain, and behavior. MIT Press, 2022.
Jurgis Pasukonis, Timothy Lillicrap, and Danijar Hafner, Evaluating long-term memory in 3d mazes, 2022. URL https:
Mareo Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Memory gym: Partially observ- able challenges to memory-based agents in endless episodes. arXiv preprint arXiv:2309.17007 2023.
Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Memory gym: Towards endless tasks to benchmark memory capabilities of agents. Journal of Machine Learning Research, 26(6): 1-40, 2025.
Peter G. Polson. The American Journal of Psychology, 88(1):331-14, 1975. ISSN 00029556. URL http:/ / www. htw.wsttorrrrr.o//o//o/ooo/oo... wwtg///aa/e/eeeotoo/oooo 672.
13

---

Published as a conference paper at ICLR 2026
Subhojeet Pramanik, Esraa Elelimy, Marlos C Machado, and Adam White. Recurrent linear trans- formers. arXiv preprint arXiv:2310.15711 2023.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomênech, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control, 2017. URL https:
//arxiv.oro/ org.ii.rrraaaeee www.////..... 01988.
Jan Robine, Mare Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-bused world models are happy with 100k interactions. In The Eleventh International Conference on Learning Representations, 2023. URL https: /openrev///eeeee///e/eeeee
net/ wow.mmiiddeddaaaaaeao
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back- propagating errors. Nature, 323:533-536, 1986. URL htt.s:/tt///ttm //ppi semant icschol ar. org/Corpus 20500011
Mohammad Reza Samsami, Artem Zholus, Janarthanan Rajendran, and Sarath Chandar. Mastering memory tasks with world models, 2024. URL utLhttps:/ http://awwwvv..../.oo.ooo awaaab//400 22003. 04253.
Gresa Shala, André Biedenkapp. and Josif Grabocka. Hierarchical transfoomerss are efficient meta- reinforcement learners, 2024. URL httptts://aaxl...or///aa////////
06402.
Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling, 2023. URL https:: httpt::://axxxvooootee org/ www./os////a//eoeoooooooooeeee/s /2008 04933.
Doo Re Song, Chuanyu Yang, Christopher McGreavy, and Zhibin Li. Recurrent deterministic policy gradient method for bipedal locomotion on rough terrain challenge. November 2018. URL
http://dst doi. org/ 10 1109/ ICARCV. 2018. 8581309.
Artyom Sorokin, Nazar Buzun, Leonid Pugachev, and Mikhail Burtsev. Explain my surprise: Learning efficient long-term memory by predicting uncertain outcomes. Advances in Neural Information Processing Systems, 35:36875-3688 2022.
Ivan Sorokin, Alexey Seleznev, Mikhail Pavlov, Aleksandr Fedorov, and Anastasiaa Ignateva. Deep attention recurrent qentwwork, 2015. URL https: /awxiv..vvvv www.aiiww...a..eeeeooooooooooooe org/ abs 111.1.11....11 01
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmice Michael Chang. Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompsae Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei Zhang. Human-timesenn adaptation in an open-ended task space, 2023. URL https: / arxiv. org/ abs/ 201022311100 07608.
Trieu Trinh, Andrew Dai, Thang Luong, and Quoe Le. Learning longer-term dependencies mns with auxiliary losses. In International Conference Machine Learning, pp. 4965-4974, PMLR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Daan Wierstra, Alexander Förster, Jan Peters, and Jürgen Schmidhuber. Recurrent policy gradients. Logic Journal of the IGPL., 18:620-634, 10 2010. doi: 10. 10.10993/jgppaliix049.
William Yue, Bo Liu, and Peter Stone. Learning memory mechanisms for decision making through demonstrations. arXiv preprint arXiv:24/1.07954 2024.
Tony Duan YuXuan Liu and Wesley Hsich. Temporal convolutional policy networks, 2016. URL https: httpps//yuxuanll uu. htm.co/.coooooooooom.mmmo.... weeeee/tttte tttps/tpn.ppp.. .dd.
Wojciech Zaremba and Ilya Sutskever. Reinforcement learning neural turing machines - revised, 2016. URL htl.httss:///xxiv //t6:///..ttss///tttooooo//////2
wttp:////////./.....00.5./ 00521.
14

---

Published as a conference paper at ICLR 2026
Deyao Zhu, Li Erran Li, and Mohamed Elhoseiny. Value memory graph: A graph-structured world model for offline reinforcement learning, 2023. URL httpps//aaxxiv.crg//aa/aa 04384.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal. Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning 2020. URL https: //axiiv org abs / 910 08348.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX GLOSSARY
In this section, we provide a comprehensive glossary of key terms and concepts used throughout this paper. The definitions are intended to clarify the terminology proposed in our research and to ensure that readers have clear understanding of the main elements underpinning our work.
1. M MDP environment
2. MM POMDP environment
3. MM memory--ntensste environment
4. ho:-1 = {(o,, 04, rx)) agent history of interactions with environment
5. К agent base model context length
6. K context memory border of the agent, such that K [ strictly LTM problem
7. μ(K) memory mechanism that increases number of steps available to the agent to process
8. Keff = {u(K) the agent effective context after applying the memory mechanism
9. o {(o,, a,. +
+ should recall when making a decision the future
event starting time te and lasting At, which the agent
10. 3, (a..a = at (o,, a. the moment of decision making time t, according to
the event at
11. t to At + an event's correlation horizon
B APPENDIX ADDITIONAL NOTES ON THE MOTIVATION FOR THE ARTICLE
B.1 WHY USE DEFINITIONS FROM NEUROSCIENCE?
Definitions from neuroscience and cognitive science, such as short-term and long-term memory, as well as declarative and procedural memory, are already well-established in the RL community, but do not have common meanings and are interpreted in different ways. We strictly formalize these definitions to avoid possible confusion that may arise when introducing new concepts and redefine them with clear, quantitative meanings to specify the type of agent memory, since the performance of many algorithms depends on their type of memory.
In focusing exclusively memory within RI, we do not attempt to exhaustively replicate the full spectrum of human memory. Instead, our goal is leverage the intuitive understanding of neuroscience concepts already familiar to RL researchers. This approach avoids the unnccessary introduction of new terminology into the already complex Memory RL domain. By refining and aligning existing definitions, create robust framework that facilitates clear communication, rigorous evaluation, and practical application RL research.
B.2 ON PRACTICAL APPLICATIONS OF OUR FRAMEWORK
The primary goal our framework is address practical challenges in RL by providing robust classification of memory types based on temporal dependencies and the nature of memorized infor- mation. This classification is essential for standardizing memory testing and ensuring that RL agents are evaluated under conditions that accurately reflect their capabilities.
In RL, memory is interpreted in various ways, such as transformers with large context windows, recurrent networks, or models capable of skill transfer across tasks. However, these approaches often vary fundamentally design, making comparisons unreliable and leading to inconsistencies testing. Our framework resolves this by providing a clear structure to evaluate memory mechanisms under uniform and practical conditions.
The proposed definitions of declarative and procedural memory use two straightforward numerical parameters: the number of environments (Nenve) and episodes (Nepe). These parameters allow researchers to reliably determine the type of memory required for a task. This simplicity and alignment with numerical parameters make the framework practical and widely applicable across diverse RL problems.
16

---

Published as a conference paper at ICLR 2026
Moreover, the division of declarative memory into long-term and short-term memory, well as the need to use balance between the agent's 's context length K and the correlation horizons of the environment & when conducting the experiment, allows us to unambiguously determine which type of memory is present in the agent. This clarity ensures fair comparisons between agents with similar memory mechanisms and highlights specific limitations in an agent's design. By aligning memory definitions with practical testing requirements, the framework provides actionable insights to guide the development of memory-enhancee RL agents.
C APPENDIX - MEMORY MECHANISMS
In RL, memory has several meanings, each of which is related to specific class of different tasks. To solve these tasks, the authors use various memory mechanisms. The most prevalent approach to incor- porating memory into an agent is through the use of Recurrent Neural Networks (RNNs) (Rumelhart et al., 1986), which are capable of handling sequential dependencies by maintaining hidden state that captures information about previous time steps (Wierstra et al., 2010; Hausknecht & Stone, 2015; Sorokin et al., 2015; Duan et al., 2016; Song et al., 2018; Zintgraf et al., 2020) (pure LTM, according to our taxonomy). Another popular way to implement memory is to use Transformers (Vaswani al., 2017), which use self-attention mechanisms to capture dependencies inside the context win- dow (Parisotto et al., 2020; Lampinen et al., 2021; Esslinger et al., 2022; Melo, 2022; Team et al., 2023; Pramanik al., 2023; Robine et al., 2023; Ni al., 2023; Grigsby et al., 2024; Shala et al., 2024) (STM in case of classical transformers without additional memory mechanisms or LTM if we use recurrent memory, activation caching, etc.). space models (SSMs) (Gu et al., 2021; Smith et al., 2023; Gu & Dao, 2023) combine the strengths of RNNs and Transformers and can also serve to implement memory through preservation of system state (Hafner et al., 2019; Lu et al., 2023; Becker et al., 2024; Samsami et al., 2024) (LTM, according our taxonomy). Temporal convolutions may be regarded as an effective memory mechanism, whereby information is stored implicitly through the application of learnable filters across the time axis (YuXuan Liu & Hsieh, 2016; Mishra et al., 2018) (STM, since memory is represented as a fixed-size temporal convolution, analogous to an attention window). A world model (Ha & Schmidhuber, 2018) which builds internal environ- ment representation can also be considered as a form of memory. One method for organizing this internal representation is through the use of graph, where nodes represent observations within the environment and edges represent actions (Morad et al., 2021; Zhu et al., 2023; Kang et al., 2024b).
distinct natural realization of memory the utilization of an external memory buffer, which enables the agent to retrieve pertinent information. This approach can be classified into two categories: read- only (writeless) (Oh et al., 2016; Lampinen et al., 2021: Goyal al., 2022; Cherepanov al., 2024) and read/ wrrtt access (Graves et al., 2016; Zaremba & Sutskever, 2016; Parisotto & Salakhutdinov, 2017a).
Memory can also be implemented without architectural mechanisms, relying instead on agent policy. For instance, in the work of Deverett et (2019), the agent learns to encode temporal intervals by generating specific action patterns. This approach allows the agent to implicitly represent timing information within its behavior, showcasing that memory can emerge as a result of policy adaptations rather than being explicitly embedded in the underlying neural architecture
Using these memory mechanisms, both decision decision-making tasks based on information from the past within a single episode and tasks of fast adaptation new tasks solved. However, in works using the same underlying base architectures to solve the same class of problems, the concepts of memory may differ.
D APPENDIX POMDP
D.1 POMDP
The Partially Observable Markov Decision Process (POMDP) is a generalization of the Markov Decision Process (MDP) that models sequential decision-making problems where the agent has incomplete information about the environment's 's state. POMDP can be represented Mp = A, O,P, R, Z), where S denotes the set of states, AA is the set of actions, O is the set of observations and 2 P(o++1 81+1,0)) is observation function such that o++1 2(8411, at).
tuple
17

---

Published as a conference paper at ICLR 2026
An agent takes an action de AA based on the observed history hg.-1 {(os, at 1==o and receives a reward ry- = R(s,, a.). It is important to note that state s is not available to the agent at time t. In the case of POMDPs, policy is function T(ae 0e, h0=-11 that uses the agent history ha-e-1 to obtain the probability of the action or. Thus, in order to operate effectively in a POMDPs, an agent must have memory mechanisms to retrieve a history h Partial observability arises in variety of real-world situations, including robotic navigation and manipulation tasks, autonomous vehicle tasks, and complex decision- making problems.
E APPENDIX META REINFORCEMENT LEARNING
In this section, explore the concept of Meta- Reinforrennrr Learning (Meta-RL), specialized domain within POMDPs that focuses on equipping agents with the ability to learn from their past experiences across multiple tasks. This capability is particularly crucial in dynamic environments where agents must adapt quickly to new challenges. By recognizing and memorizing common patterns and structures from previous interactions, agents can enhance their efficiency and effectiveness when facing unseen tasks.
Meta-RL is characterized by the principle of "learning learn", where agents trained not only to excel at specific tasks but also to generalize their knowledge and rapidly adjust to new tasks with minimal additional training. This adaptability is achieved through a structured approach that involves mapping data collected from various tasks to policies that guide the agent's behavior.
Meta-RL algorithm is function fe parameterized with meta-parameters that maps the data D, obtained during the process of training of RL agent in MDPs (tasks) M; p(M), to policy typically referred as the outer-loop.
Ta Ø (D). The process of learning the function while the resulting function f is called the inner-loop. In this context, the parameters 0 are associated with the outer-loop, while the parameters @ are associated with the inner-loop. Meta-training proceeds by sampling task from the task distribution, running the inner-loop loop to improve the policies it produces. The interaction of the inner-loop with the task, during which the adaptation happens, is called lifetime be shared between all of the tasks and the tasks to only differ in the reward R(s, a) function, the dynamics P(s') s, a,. and initial state distributions (so) (Beck et al., 2024).
it, and optimizing the inner-
trial. In Meta-RL, it is common for S and A to
F APPENDIX - EXPERIMENT DETAILS
This section provides an extended description of the environments used in this work.
Passive-T-Maze (Ni et al., 2023). In this T-shaped maze environment, the agent's goal is to move from the starting point to the junction and make the correct turn based agent can select from four possible actions: a E fe,t, up, right, down. The signal, denoted by the variable clue, provided only at the beginning of the trajectory and indicates whether the agent should turn up (clue = 1) or down (clue = 1). The episode duration is constrained to T = L + 1, where is the length of the corridor leading to the junction, which adds complexity the task. To facilitate navigation, binary variable called flag is included in the observation vector. This variable equals 1 one step before reaching the junction and all other times, indicating the agent's proximity to the junction. Additionally, a noise channel introduces random integer values from the set -1,1, O, +1 into the observation vector, further complicating the task. The observation vector is defined as o [y, clue, flag, noise). where y represents the vertical coordinate.
initial signal. The
The agent receives a reward only at the end of the episode, which depends on whether it makes a correct turn at the junction. A correct turn yields reward of 1, while an incorrect turn results in reward of O. This configuration differs from the conventional Passive T-Maze environment (Ni et al., 2023) by featuring distinct observations and reward structures, thereby presenting a more intricate set of conditions for the agent to navigate and learn within a defined time constraint. To transition from a sparse reward function defined as penalty = チ-T. which imposes a penalty on the agent for each step taken within the environment. Thus, this environment has a ID vector space of observations, a discrete action space, and sparse and dense configurations of the reward function.
dense reward function, the environment is parameterized using penalty
18

---

Published as a conference paper at ICLR 2026
Minigrid-Memory (Chevalier-Boisvee et al., 2023). Minigrid-Memory is two-dimensional grid-based environment specifically crafted to evaluate an agent's long-term memory and credit assignment capabilities. The layout consists of T-shaped maze featuring small room at the corridor's outset, which contains an object. The agent is instantiated at a random position within the corridor. Its objective is navigate to the chamber, observe and memorize the object, then proceed to the junction at the maze's 's terminus and turn towards the direction where the object, identical to that in the initial chamber, is situated. A reward function defined successful completion, while failure results in reward of zero. The episode concludes when the agent either makes a turn a junction or exhausts a predefined time limit of 95 steps. To implement partial observability, observational constraints are imposed on the agent, limiting its view to a 3 x frame size. Thus, this environment has 2D space of image observations, discrete action space, and sparse reward function.
0.9 & is awarded upon
F.1 EXPERIMENTAL PROTOCOL
For each experiment, we conducted three runs of the agents with different initializations and performed validation during training using 100 random seeds ranging from to 99. The results are presented as the mean success rate (or reward) + the standard error of the mean (SEM).
Table 3: Online RI. baselines hyperparameters used the Minigrid-Memory and Passive T-Maze experiments.
Table 4: SAC-GPT-2
Table 5: DQN-GPT-2
Hyperparameter
Value
Hyperparameter
Value
Number of layers
Number of layers
Number of attention heads
2
Number of attention heads
Hidden dimension
256
Hidden dimension
256
Batch size
64
Batch size
64
Optimizer
Adam
Optimizer
Adam
Learning rate
3e-4
Learning rate
3e-4
Dropout
0.1
Dropout
0.1
Replay buffer size
le6
Replay buffer size
le6
Discount (-)
0.99
Discount (?)
0.99
Entropy temperature
0.1
Table 6: DTQN
Hyperparameter
Value
Number of layers
4
Number of attention heads
8
Hidden dimension
128
Batch size
32
Optimizer
Adam
Learning rate
3e-4
Dropout
0.1
Replay buffer size
5e5
Discount (7)
0.99
19

---

Published as a conference paper at ICLR 2026
Table 7: Offline RL baselines hyperparameters used for Decision Transformer and BC-LSTM in T-Maze experiments.
Table 8: Decision Transformer (DT)
Table 9: BC-LSTM
Hyperparameter
Value
Hyperparameter
Value
Number of layers
8
Number of layers
Number of attention heads
4
Hidden dimension (dmodel)
64
Hidden dimension (dmodel)
128
Bidirectional
False
Feedforward dimension (dinner)
128
Effective Context length (Kefs)
37
Head dimension (dhead)
128
Dropout
0.0
Context length (K)
31
Optimizer
AdamW
Dropout
0.0
Learning rate
3e-4
DropAttention
0.0
Weight decay
0.01
Optimizer
AdamW W
Adam betas
(0.9, 0.999)
Learning rate
le-4
Batch size
64
Weight decay
0.1
Warmup steps
100
Adam betas
(0.9, 0.999) 64
Epochs
100
Batch size
Warmup steps
1000
Epochs
200
20


Reviews:
[{"review_id": "rLPKjqgHaD", "reviewer": "Reviewer_jjn4", "summary": "This paper highlights the lack of a unified methodology for validating “memory’ for RL agents and presents a robust experimental methodology and standardize evaluations. In particular, this paper provides formal definitions to clarify the widely adopted concepts of memory in cognitive science.", "questions": "**Questions**\n\nQ1. Is it possible to present the outer loop in Figure 3?\n\nQ2. Could the authors provide a more detailed explanation of $n$ in $\\{\\zeta_n\\}_n$? Do we also have a set of $n$ here? This point is not very clear. $n$ should be properly defined first.\n\nQ3. Please use consistent notation for $\\min_n \\Theta$ and $\\max_n \\Theta$, instead of mixing $\\min_n \\Theta$ and $\\min \\Theta$.\n\nQ4. Some illustrations depicting cases that validate LTM and STM for different $K$ values (as explained in the lower part of Page 6) would help clarify the general characteristics.\n\n**Comments**\n\nC1. Please check the citation format in “skills across tasks Team et al. (2023)” on Page 2.\n\nC2. Figure 4 could be properly defined as a table rather than a figure. The term 0-shot seems a bit unnatural. Please use a more common term, such as 'zero-shot'.\n\nC3. Please check the citation format for Theorem 2 on Page 5 and for Algorithm 1 on Page 7.\n\nC4. $L$ is used before being properly defined, and its definition is placed within the script for Table 1. The reviewer strongly recommends defining the values before use.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper is generally well-written and well-motivated. This paper studies the relatively underexplored subject of “memory in RL agents” and presents formal definitions of various memory concepts, highlighting the importance of appropriate experimental configurations for evaluating them. Pictorial illustrations help clarify the concepts.", "weaknesses": "Some of the definitions require more explanation to fully understand the concept. Paper formatting could be improved. Please see the questions and comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bjmvvMi0L7", "reviewer": "Reviewer_vvf9", "summary": "This paper proposes a clear, systematic, and practically meaningful framework for classifying and evaluating “memory” capabilities in reinforcement learning (RL) agents. Drawing inspiration from cognitive science—particularly classic memory typologies such as short-term vs. long-term memory and declarative vs. procedural memory—the authors formalize these concepts into operational definitions suitable for RL settings. The paper further introduces key notions such as “memory-intensive environments,” “correlation horizon (ξ),” and “context memory border (K),” and builds upon them a rigorous experimental methodology.", "questions": "NO", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Conceptual Clarification and Formalization**\nThe paper successfully translates ambiguous cognitive science terms (e.g., STM/LTM, declarative/procedural memory) into precise, quantifiable, and verifiable definitions within RL (see Definitions 4.4–4.6). This formalization fills a significant void in current RL literature, where “memory” is often used loosely or inconsistently.\n\n**Proposal of a Unified Evaluation Framework**\nThe distinction between Memory Decision-Making (Memory DM) and Meta-RL is clearly articulated. The operational definitions of declarative memory (nenvs × neps = 1) and procedural memory (nenvs × neps > 1) provide a practical and measurable criterion for categorizing memory types.", "weaknesses": "**Abstract Treatment of Memory Mechanisms**\nWhile Definition 4.7 defines a memory mechanism as a mapping from base context K to effective context K_eff, it does not differentiate between implementation strategies (e.g., external memory, world models, state-space models). A deeper discussion of how different architectures realize µ(K) would strengthen the framework.\n\n**Limited Coverage of Other Memory Types**\nThe paper focuses primarily on declarative memory along the temporal axis. Although the authors explicitly state they do not aim to replicate all aspects of human memory, a brief discussion of how “working memory” or “episodic memory” might fit into this taxonomy would enhance its extensibility.\n\nThe paper reads more like a taxonomy of memory types and seems to lack sufficient novelty or creativity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nMvbXZzmuG", "reviewer": "Reviewer_RqbQ", "summary": "This paper proposes a standard \"K-$\\xi$\" definition of different kinds of \"memory\" in RL Agents. The majority of the paper is dedicated to establishing and defining the proposed classification system. Afterward, it explains how to use this classification system to evaluate LTM and STM separately and shows examples of the misleading conclusions of combined evaluations.", "questions": "1. The order of the figures and tables is disorganized.\n2. You claim that LSTM is better than Transformers in terms of LTM. What's your opinion on the following facts?\n    * The performance of LSTM falls low when the training sequence is larger than 300. Its low performance under both seen validation samples and longer, unseen validation samples seems to indicate that the LSTM's capacity to perform both LTM and STM tasks falls short when the training sequence becomes too long. This precisely reflects the gradient vanishing problem of LSTM and seems to contradict your claim that \"BC-LSTM retains LTM despite challenges like vanishing gradient.\"\n    * Current studies have designed many methods to introduce LTM into transformers, such as maintaining an external memory bank and summarizing previous steps. How well do these methods perform in mitigating the LTM issues inherent in Transformer models?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. They standardized the definitions of different types of memory, thereby providing a framework for fair evaluation of subsequent models.\n2. The definitions and formalizations are careful and rigorous.", "weaknesses": "The paper spends too much time on the basic theoretical and general analyses, while the experiments and analyses conducted under this theoretical framework are rather limited. Although the authors include several validation experiments demonstrating the necessity and importance of defining and distinguishing different types of memory, they do not provide any insightful conclusions or analyses derived from evaluations based on this distinction. In other words, what I expected to see was either a newly designed evaluation benchmark or tasks built upon this theoretical distinction of memory, or a comparative analysis of how the latest models or methods perform across different types of memory. But neither is included in the paper.\n\nFrankly speaking, I am not very familiar with the expected workload or contribution standards for theoretical analyses in the RL field. Therefore, I have set my confidence score relatively low. Overall, what I expected to see here was the proposal of a new theory accompanied by analyses of the limitations of current research or guidance for the design of new models and methods. However, this paper seems to have achieved only the first part. In addition, since it does not introduce any new concepts but rather provides standardized definitions for existing ones, I feel that the overall workload and contribution of this work are somewhat limited. Again, as I am not fully familiar with the typical expectations for such work in RL, I am open to discussion and would be willing to raise both the rating and confidence scores if convinced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z6y69Lw3rF", "reviewer": "Reviewer_Wrt1", "summary": "The overcome the limitation that the term “memory” in RL remains inconsistently defined and often misaligned with what experiments actually test, this paper introduces a unified taxonomy of memory in RL based on temporal dependencies and task structure. More specifically, they give a formal definition for the Memory Decision-Making with declarative long-term and short-term memory and Meta-RL with procedural memory. Then an experimental algorithm is proposed to test agent memory types based on context length and context memory border. Experiments on T-Maze, MiniGrid, and POPGym show that the proposed algorithm is efficient at distinguishing memory types.", "questions": "1.\tCould the authors classify more classical works in the domain of memory RL?\n2.\tCould this method be applied to agentic RL with LLMs?\n3.\tAre there any future research directions based on this work?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper is well-written and well-organized. It provides a clear taxonomy of memory in RL with clear definitions.\n2.\tThe classification of long-term and short-term memory with context length and context memory border is reasonable.\n3.\tThe experiments in Section 5.1 show that the variable correlation horizon could hide long-term memory’s deficits.", "weaknesses": "1.\tThe definition of memory in RL is not a novel thing. For example, long-term and short-term memory are already well-studied in the domain of RL. And skill transfer in Meta-RL is also a popular concept.\n2.\tMany works in memory RL such as [1,2] are not classified using the proposed experimental classification algorithm (Algorithm 1).\n3.\tThere are some typos such as “According to Theorem Theorem 2,”.\n\nReferences\n\n[1] Beck, Jacob, et al. \"Amrl: Aggregated memory for reinforcement learning.\" International Conference on Learning Representations. 2020.\n\n[2] Loynd, Ricky, et al. \"Working memory graphs.\" International conference on machine learning. PMLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)