Titans: Learning to Memorize at Test Time
Authors: Ali Behrouz
Peilin Zhong
Vahab Mirrokni
Venue: NeurIPS 2025 poster ()
Keywords: Attention
Associative Memory
Long-term Memory
Hybrid Models
Primary area: deep_learning

Abstract: Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context. We present a neural long-term memory module that learns to memorize historical context and helps attention to attend to the current context while utilizing long-past information. We show that this neural memory has the advantage of fast parallelizable training. From a memory perspective, we argue that attention due to its limited context but accurate dependency modeling performs as a short-term memory, while neural memory due to its ability to memorize the data, acts as a long-term, more persistent, memory. Based on these two modules, we introduce a new family of architectures, called Titans, and present three variants to address how one can effectively incorporate memory into this architecture. Our experimental results on language modeling, common-sense reasoning, and time series tasks show that Titans are effective compared to baselines, while they can effectively scale to larger context window in needle-in-haystack tasks.

Titans: Learning to Memorize at Test Time
Ali Behrouz
Peilin Zhong Google Research USA pe peilinz@google.com com
Vahab Mirrokni Google Research USA mirrokmi@google..o
Google Research
USA
alibehrouzzgoogle.ccom
Abstract
Over more than decade there has been extensive research effort on how effectively utilize recurrent models and attention. While recurrent models aim to compress the data into fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context. We present a neural long-term memory module that learns memorize historical context and helps attention to attend to the current context while utilizing long-past information. We show that this neural memory has the advantage of fast parallelizable training. From memory perspective, we argue that attention due to its limited context but accurate dependency modeling performs as a short-term memory, while neural memory due to its ability to memorize the data, acts as a long-term, more persistent, memory. Based on these two modules, we introduce a new family of architectures, called Titans, and present three variants to address how one can effectively incorporate memory into this architecture. Our experimental results on language modeling, common-sense reasoning, and time series tasks show that Titans are effective compared baselines, while they can effectively scale to larger context window needle-in in-haystack tasks.
Introduction
Transformers [1], have been firmly established as state-of-the-art models in sequence modeling, mainly due to their in-context learning and ability learn at scale [2]. The primary building blocks of Transformers-attennine
modules-function as associative memory blocks [3], where they learn store key-value associations and retrieve them by computing pairwise similarity between queries (i.e., search signals) and keys (i.c., contexts). Accordingly, by design, the output of a Transformer is exclusively conditioned on the direct dependencies of tokens the current context window. This accurate modeling of dependencies, however, comes with quadratic time and memory complexity in terms of the context length. In complex real-worlh tasks (e.g.. language modeling [4], video understanding [5], long-term time series forecasting [6]), the context window can become extremely large, making the applicability of Transformers challenging in these downstream tasks.
To overcome the scalability issue of Transformers, recent studies aim to design different variants of linear Transformers [7-9], where sofumax is replaced by a kernel function the attention (sce $A.1 1 for details), resulting in a significant drop in memory consumption. Despite efficiency and the ability scale longer context, linear Transformers do not show competitive performance compared to Transformers as the kernel trick makes the model a linear recurrent network, in which the data is compressed into a matrix-valued states [7]. This, however, brings contradictory fact about linear recurrent (or linear Transformers) models: On one hand, we use these linear models to enhance scalability and efficiency (linear vs. quadratic complexity), whose advantages is appeared
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

for very long context; On the other hand, a very long context cannot be properly compressed in small vector-valued or matrix-valued states [10].
Furthermore, beyond efficiency, most existing architectures-ranging from Hopfield Networks [11] to LSTMs [12] and Transformers [1]-face challenges when dealing with generalization, length extrapolation, and/or reasoning [13, 14], all of which are inseparable parts of many hard real-world tasks. Although these architectures draw inspiration from the human brain, each of which are missing: (1) a crucial component for learning process- such as short-term memory, long-term memory, meta- memory, attending to current context, etc. [15]; (2) how these components are interconnected systems that can operate independently; and/or (3) the ability to actively learn from data and memorize the abstraction of past history. We argue that in an effective learning paradigm, similar to human brain, there are distinct yet interconnected modules, each of which is responsible for a component crucial to the learning process.
Memory Perspective. Memory is fundamental mental process and is an inseparable component of human learning [16]. Without a properly functioning memory system, humans and animals would be restricted basic reflexes and stereotyped behaviors. Accordingly, memory has been the inspiration for many seminal research in machine learning literature; e.g., Hopfield Networks [11], LSTMs [12], and Transformers [1].
Taking inspiration from the common definitions of memory and learning in neuropsychology litera- ture [17], most existing architectures consider memory neural update caused by an input, and define learning as a process for acquiring effective and useful memory, given an objective. In this perspective, Recurrent Neural Networks (RNNs) [18] can be defined as models with vector-valued memory module M (also called hidden state) with two main steps: Given a new input 2t at time the model (1) updates the memory using function f(M,-1, (with compression); and (2) retrieves the corresponding memory of input using function g[M,, xz). Similarly. Transformers can be seen as architectures with a growing memory and two similar steps. That is, the pair of key and value matrices acts as the model's memory, and the model: (1) updates the memory by appending the key and value to the memory (without compression). and (2) retrieves query vectors' corresponding memory by finding the similarity of query and key vectors, which is then used to weight the value vectors for the output.
This perspective, can help us better understand existing paradigms, their critical differences, and design more effective architectures. For example, the main difference between Transformers [1] and linear Transformers [7] is the memory structure as well as the memory updating step, in which linear Transformers compress the historical data into a fixed-size matrix-valued memory while Transformers keep all historical data (within the context length) without any compression. Therefore, this perspective motivates us to ask: (Q1) What a proper memory update mechanism? and (Q2)
What is good memory retrieval process?
Human memory is neither a unitary process nor it serves single function [15]. In fact, memory is a confederation of systems-e.g. short-term, working, and long-term memory-each serving a different function with different neural structures, and each capable of operating independently [19]. This motivates us to ask: (Q3) How to design an efficient architecture that incorporates different interconnected memory modules. Finally, storing memory neural process that requires to encode and store the abstraction of the past. It can be over-simplification to assume a single vector or a matrix, whose parameters are encoding the data in a linear manner, are enough for storing long-term history. (Q4) Is a deep memory module needed to effectively storc/remember long past?
Neural Memory ($2). We present (deep) neural long-term memory that (as a meta in-context model) learns how to memorize/store the data into its parameters at test time. Inspired by human long- term memory system [20], we design this memory module so event that violates the expectations is more memorable. To better handle the limited memory, we present a decaying mechanism that consider the proportion of memory size and the amount of data surprise, resulting in better memory management.
Titans Architectures ($3). After designing the long-term neural memory, an important remaining question is how to effectively incorporate memory into an architecture. We present Titans, a family of deep models that consists of three hyper- heads: (1) Core: this module consists of the short-term memory, and is responsible for the main flow of processing the data (we use attention with limited window size); (2) Long-term Memory: this branch is neural long-ten long-term memory module that

---

is responsible to store/remember long past; (3) Persistent Memory: this is a set of learnable but mata-independent parameters that encodes the knowledge about task. Finally, as proof of concept. we present three variants of Titans, in which we incorporate memory as: (i) a context, (ii) layer, and (iii) a gated branch.
We provide an extensive list of related work and background concepts in Appendix A and Appendix B.
2 Learning to Memorize at Test Time
To overcome the lack of long-term memory and to enable the model to leam, forget, and retrieve information, in this section, inspired by the concept of Test Time Training [21-23], we present a neural long-term memory module, which is a meta models that learns memorize at test time. In Section 2.1, we first discuss the motivation and the design of the neural memory. In Section 2.2, we discuss how our architecture design can benefit from fast and parallelizable training. Finally, in Section 2.3, we augment our architecture using persistent memory module, in which we use learnable but data-independent parameters leamn meta information about the task.
2.1 Long-term Memory
To design a neural long-term memory, we need a model that can encode the abstraction of the past history into its parameters. An example of this can be deep large models that are shown to be memorizing their training data [24-26]. Therefore, a simple idea is to train a neural network and expect it to memorize its training data. Memorization, however, has almost always been known an undesirable phenomena in neural networks as it limits the model generalization [27], causes privacy concerns [24], and so results in poor performance test time. Moreover, the memorization of the training data might not be helpful at test time, in which the data might be dittof-distribution. We argue that, we need an online meta-model that learns how memorize/forget the data at test time. In this setup, the model is learning function that is capable of memorization, but it is not overfitting to the training data, resulting in a better generalization at test time.
Learning Process and Surprise Metric. The key idea to train a long-term memory is treat its training as an online learning problem, in which we aim to compress the past information Ji ri,..., T1-1 into the parameters of our long-term neural memory module Mr. As discussed earlier, an event that violates the expectations (i.e., is surprising) is more memorable for humans [20]. Inspired by this, simple definition of surprise for a model can be its gradient with respect to the input. The larger the gradient is, the more different the input data is from the past data. Accordingly, using this surprise score, we can update the memory as M, = M 0, (((M;--;; This surprise metric. however, can result in missing important information that comes after big surprising moment. That is, the gradient can become extremely small after several surprising steps, leading to stocking in a flat area (i.e., local minima), and missing information about some parts of the sequence. From the human memory perspective, an event might not consistently surprise us through a long-period of time although it is memorable. The reason is that the initial moment is surprising enough to get our attention through long time frame, leading to memorizing the entire time frame. To improve the above surprise metric, we break the surprise metric into (1) past surprise, which measures the surprise amount very recent past; and (2) momentary surprise, which measures the surprise of incoming data:
M S., S. = TJt St-1
0g (e
(1)
Past Surprise
Momentary Surprise
Interestingly, this formulation is similar to gradient descent with momentum, where S, is the mo- mentum element. Therefore, the momentum here act as a memory of surprise across time (sequence length). In this formulation, the term 77 a data-dependent surprise decay (a function of xx), controlling how surprise decays over time, and the term 0 is controlling how much of momentary surprise should be incorporated into the final surprise metric in a data- data-dependent manner. This dependeeyy is particularly important in this design: While surprise of previous tokens might be needed to affect the surprise of the next token, it is mostly valid if all tokens are relevant and are in the same context. Accordingly, a data-dependent 7) can control if memory needs to: (1) ignore the last surprise by setting π) - 0 (possibly due to the change of context), or (2) fully incorporate the last surprise by setting 7)t + 1 (possibly as the token is highly relevant to its recent past tokens).
3

---

Linear
Non: Linear
Momectum Calculation
Weight Decay
All gradienta pre-computed
W/o Decay
M
(Wi. X)X*
Or
w/ Decay G,B,(W6X XXX
Parallel Associative
via cumsum
via Gradient
Glabal Kernel
via Matmal
Figure 1: The illustration of how neural memory's training can be done in parallel and using matmuls.
Objective. Our above surprise metric is based on loss function ((:i.). which is the objective that our memory learning act as it at test time. That is, our memory module is meta model that learns function based on the loss function ((::-) In this work, we focus associative memory, in which we aim to store the past data as the pairs of keys and values. Given 24, similar to Transformers, use linear layers to project 2, into a key and value: k, WK andv, Wv, where Wx and Wy E I xdu, Next, we expect our memory module to Iearn the associations between keys and values. To this end, we define the loss follows:
((MM-1ix)) = (k,)
By optimizing the above loss function in the inner-loop of our meta model (memory), the model learns how to memorize the mapping between keys and values at test time. Note that, similar to meta-learning models [28, 29], training of the memory is the inner-loop, and so parameters WK and Wy are hyperparameters in the above loss function. Accordingly, in the inner loop, optimize M's weights, while in the outer-loop, we optimize other parameters of the entire architecture.
Forgetting Mechanism. When dealing with very large sequences, crucial to manage which past information should be forgotten-even with deep or very large matrix-valued memory. To this end, we use an adaptive forgetting mechanism that allows the memory to forget the information that not needed anymore, resulting in better managing the memory's limited capacity. That is, given the next token It, we modify the update rule as:
M, (1 M,) Se
- 0g VE (E(M,
where ag E [0, 1]dz is the gating mechanism that flexibly controls the memory; i.e., decides how much information should be forgotten. For example, can update the memory without affecting the past abstraction by letting a - O, and can clear the entire memory by letting a + Later this section, show that this weight decay mechanism is closely related to the gating mechanism in modern RNNs [30, 31].
Memory Architecture. In this paper, we focus on simple MLPs with LM 2 layers as architecture of our long-term memory. The main reason behind this choice is that we want to focus on better motivating the design of the long-term memory and ways that can incorporated into an architecture. Recently, there has been promising line of work to design architectures that are better memorizers [32-34]; incorporating them into our framework (i.c., replacing simple MLPs with such architectures) can be interesting future work.
Retrieving Memory. In the above, we discuss how one can design and train long-term memory module that learns to memorize test time. key remaining question is: How one can retrieve information from the memory? We simply use the forward pass without weight update (i.e., inference) to retrieve a memory correspond to query. Formally, given an input Tt, we use a linear layer Wq to project the input, i.e., = x, Wo and retrieve the corresponding information from the memory Y/ by Vt M(q.).
2.2 How Parallelize the Long-term Memory Training
As discussed above, the design of our long-term memory module is equivalent to training meta model by optimizing associative memory loss function {(Me-1; = ||M (k,) vell2 using gradient descent with momentum and weight decay. Therefore, theory, the training long-term memory module requires O (N) FLOPs, where N is the sequence length. However, in practice,

---

Teat Time
Conbextual Menory
Neural
Menoey
Uplate
Learning
Allectiinn
Learning
Core
Sequeace
Penintent
Memory
Pized
Leaznabie Date Independent Weighle
Figure 2: Memory Context (MAC) Architecture. This architecture includes three branches of (1) core, (2) contextual (long-term) memory, and (3) persistent memory. The core branch concatenates the corresponding long-term and persistent memories with the input sequence. Attention then performs on the sequence and decides what part of the information should be stored in the memory.
need to parallelize the training process and to fully take advantage hardware accelerators (e.g.. TPUs, GPUs), we need to tensorize the process and use more matmuls.
Similar Sun et al. [21] and contrary to recent modern linear recurrent models [35, 36, 30], our formulation of neural memory is not a linear recurrence. Therefore, it cannot be written as an associative operator, and be trained by parallel scan [37]. Accordingly, we use the idea of chunk- wise recurrence [38, 21], and instead of V (Mr-1; 22), we use V (M); in Equation 3, where is the last step the previous chunk (see Figure 1).
To fully take advantage of accelerators, we need to use matmuls and sum. We build upon the work of Sun al. [21] that shows forward pass of model optimizing with the mini-batch gradient descent (with constant learning rate) can be calculated using matmuls. As discussed above, we split the sequence into chunks of size b 1, and write the mini-batch gradient descent
Mi (1 (01-aa)MM M - W. = B.Mo Σ 0, Br
V((Me;a;).
1=1
where e' mod b), and B: IT (aj For the sake of simplicity, we focus on the first chunk, i.e., b and so 1' Also, we explain the process for the case that M, W, is linear. The process for MIPs with N, 2 is similar. Using our loss function, have:
P 0. VV -WoK V)XT,
ve(Wo:z:)) (Woks v.)=
Σ
(4)
where θ, = diag (l0 02 0b]) and Bü is defined analogously on B,
Note that, we do not need to store all Oso and Bag for k ,,...,,/4....... N/b, instead, we store these matrices for each chunk, and using less memory. Next, extend this representation so we can incorporate the momentum term. If we look at the momentum term, we have:
(5) Note that, we can compute all u, at the same time, and so Equation linear recurrence with u,s inputs, S, as the state, and 7), input-dependent transition value. Accordingly, we can use parallel associative sean [37] to calculate S,s in each chunk.
S. S 0g u4.
where u, =
2.3 Persistent Memory
Our long-term memory can also be seen as a contextual memory, meaning that the output is fully depend on the context. Therefore, in addition but input-independen parameters to act as task-related memory. This type of memory has been referred to as persistent or meta-memory in the literature [39, 40]. Given N, 1, we use Iearnable PN, and append it to the start of sequence: i.e., given context window size of N, we modify the input as:
long-term memory, also set of learnable
parameters [p1 P2
Inew [Pi
P2
PN.]1l
(6)
where is concatenation. We discuss the other motivations for persistent memory in Appendix C.
5

---

How to Incorporate Memory?
An important question that remained unanswered is: How one can effectively and efficiently incorpo- rate the designed neural memory into deep learning architecture? In this section, we aim to ans wer the above question by proposing three different variants of Titans, Later in our experiments, we show that each of these variants has its own (dis)advantages and also can show a trade-off between the efficiency and effectiveness in very long-contexts.
3.1 Memory Context
In the first architecture design (see Figure 2), we treat the memory as a context to the current information. That is, given long sequence x R Nxdu, first divide the sequence into fixed-size segments ş for i = 1,... ..... N/C. Given the incoming segment ( we consider it as the current context and its past segment as the historical information. Therefore, let M be the state of long-term memory before segment s(t), we use the input context as the query to the memory M,- to retrieve the corresponding information from the long-term memory. That is, we retrieve the past information that corresponds to g(t) as h = Mc-1(9:), where qe g WQQ Next, we use this historical information along with our persistent memory parameters as the input sequence to the attention module:
s = [Pi
P2 (((()
pN,
|| s(() ||
hes
(7)
V/x = Attn
(8)
The structure of the attention map over the entire sequence is shown in Figure 4a. We then use y/ to update the long-term memory module for the next segment and the final output:
M₁ M (Vr),
(9)
Ot yo 2 Mr (3r).
(10)
This architecture has two key advantages: (1) Attention by having both historical and current context, has the ability to decides whether given the current data, the long-term memory information is needed. (2) The attention module helps the long-term memory to store only useful information from the current context. That is. not all tokens in each segment are useful and memorizing all of them can result memory overflow. Therefore, attention helping the memory to understand which information is useful, better managing the memory capacity. At test time: (i) persistent memory parameters are fixed as they encodes the knowledge about the task, which should not be changed; (ii) the attention module weights are in-context learner; and (iii) the Long-term memory module is still learning (memorizing) the information at test time.
It is notable that we use chunk and segment for two different concepts. Chunks are subsequence that we use to accelerate the training process of the memory. In fact, for each chunk, take the gradient with respect to the last state of the previous chunk. On the other hand, we use segment to refer to the larger subsequence that we perform attention on (as discussed above).
3.2 Gated Memory
In the next variant (see Figure 5), in one branch, we directly use the long-term memory, and in the second branch, we use a sliding window attention (SWA):
P1
P2 ... PN,] || x
(11)
y Sw Attn" (8),
(12)
= & M()),
(13)
where SW-Attn' is sliding window attention with prefix (see Figure 4b). Note that, contrary to the previous design, we are not segmenting the input data. Also, we abuse the notation and use M(x) to refer to the final output of the memory after all recursion over the tokens of the sequence. In the above equation, @ can be any non-linear gating. In our experiments, we normalize the outputs and M(ī) using learnable vector-valued weights, followed by non-linearity. We provide additional motivaaioons/nneerretatiioss for this design as well as the visualization of attention mask in Appendix D.

---

The last variant is Memory as Layer (MAL), in which we sequentially use neural memory module and attention as the layer of the model. In our experiments, we use similar architecture as Samba [41], where we replace the the sequence model with our neural memory module (LMM). Additional details are provided in Appendix D.
3.3 Architectural Details
Memory Architecture. For the memory architecture, we use an MLP with LM layers (default is LM 2) with expansion factor of 4 and GELU activation function [42]. We also use residual connections and layer norm: M(x) = x +
Token Mixer. We follow previous studies [38, 35], and replace the attention with Titans in Llama's 's macro architecture with MLPs with SwiGLU() activation, rotary positional encodings (RoPE) [43], and RMSNorm [44]. Following the recent modern linear recurrent models [35, 45], we incorporate a ID convolution layer with size after each of the query, key, and value projections. For the sake of training stability, we also use & normalization to k. We also follow the recent architectures that use normalization and gating with linear layer before the final output projection [46].
Additional details about the architecture and dimensions of models are Appendix E.
Experiments
Next, we evaluate the performance of Titans and variants in language modeling, commonsense reasoning, needle in haystack, DNA modeling, and time series forecasting tasks. In more details, in this section, we answer the following empirical questions: (1) How do Titans perform compared to
baselines in downstream tasks? (see $4.2, $G.5, and 8G.6); (2) What is the actual context length of Titans? (see $4.3 and $G.1), I); (3) How do Titans scale with respect to context length? (see $G.4); (4) How the depth of memory can affect both performance and efficiency? (see $4.4); and (5) What is the contribution of each Titans' component in its performance? (see $4.5).
4.1 Experimental Setup
Models. In our experiments, we focus on the four variants of Titans, which we refer to as: Titans with Gate (MAG), and (3) Memory as Layer (MAL) as well as (4) neural memory module alone. For each of these models, we consider four scales with: (i) 170M, (ii) 340M, (iii) 400M, (iv) 760M, and (v) 1.3B parameters. While the first three are trained on 15B tokens sampled from Fine Web-Edh -du dataset [47], the last two are trained on 30B and 100B tokens from the same dataset.
(I) Memory as a Context (MAC), (2) Memory
Baselines. We compare models with the state-of-the-art linear recurrent models, Transformers, and hybrid models (recurrent + attention). More specifically in language tasks, we compare with Transformer++ [48], RetNet [49], Gated Linear Attention (GLA) [9]. Mamba [45], Mamba2 [30], DeltaNet [38], TTT [21], and Gated DeltaNet [35]. In needle in haystack tasks, we also compare with GPT4 [50], Llama3 with RAG [48], RecurrentGemmaa2998 [5]], and Mistral [52] models, all of
which are provided in the benchmark [53]. In time series tasks, we compare with hamba-based [54]. Transformer-ban based [55-57], and linear models [58-61].
Training. In the training, we follow the training procedure of Yang et al. [35], and use LLama tokenizer with a vocabulary size of 32K and use training length of 4K tokens (2K for SWA). We fixed the persistent memory size (# tokens) to 128, and 256 memory tokens encode the past data (i.e.. output of the long-term memory). We employ AdamW optimizer with learning rate of 4e-4 with cosine annealing schedule with batch size of 0.5M tokens, and weight decay of 0.
4.2 Language Modeling
We first focus on the perplexity in language modeling and also commonsense reasoning tasks. The results for Titans' variants and also baselines with three different sizes of 340M, 400M, and 760M reported in Table 1 (Table 6). Among non-hybrid models, including Transformer++, neural
7

---

Table 1: Performance of Titans and recurrent- and Transformer-basee baselines on language modeling and common-sense reasoning tasks. Hybrid models are marked with The best results among simple and hybrid models are highlighted.
Model
Wiki. ppl
LMB. ppl
LMB. acc 1
PIQA
Hella.
Wino. acc f
ARC- acc 1
ARC-c acc_n
SIQA
BoolQ acc
Avg.
acc
Wcc_n
acc 1
760M purams/
30B
tokens
Transformr+++
25.21
27.64
35.78
66.92
42.19
51.95
60.38
32.46
39.51
60.37
48.69
Mamha
28.12
23.96
32.80
66.04
39.15
52.38
61.49
30.34
37.96
57.62
47.22
DeltaNet
24.37
24.60
37.06
66.93
41.98
50.65
64.87
31.39
39.88
59.02
48.97
TTT
24.17
23.51
34.74
67.25
43.92
50,99
64.53
33.81
40.16
59.58
47.32
Gated DeltaNet
21.18
22.09
35.54
68.01
44.95
50.73
66.87
33.09
39.21
59.14
49.69
Samba"
20.63
39.72
69.19
47.35
52.01
66.92
33,20
38.98
61.24
51.08
Gated DeltaNet-H2
19.88
20.83
39.18
68.95
48.22
52.57
67.01
35.49
39.39
61.11
51.49
Titans (LMM)
20.04
21.96
37.40
69,28
48.46
52.27
66.31
35.84
40.13
62.76
51.56
Titans (MAC)
19.93
20.12
39.62
70.46
49.01
53.18
67.86
36.01
41.87
62.05
52.51
Titans (MAG)
18.61
19.86
40.98
70.25
48.94
52.89
68.23
36.19
40.38
62.11
52.50
Titans (MAL)
19.07
20.33
40.05
69.99
48.85
53.02
67.61
35.65
40.57
61.72
52.29
1,38 puramss
100B tokens
Transformer++
18.53
18.32
42.60
70.02
50.23
53.51
68.83
35.10
40.66
57.09
52.25
RetNet
19.08
17.27
40.52
70.07
49.16
54.14
67.34
33.78
40.78
60.39
52.02
Mamba2
16.56
12.56
45.66
71.87
55.67
55.24
72.47
37.88
40.20
DeltaNet
17.71
16.88
42.46
70.72
50.93
53.35
68.47
35.66
40.22
55.29
52.14
Gated DeltaNet
16.42
12.17
46.65
72.25
55.76
57.45
71.21
38.39
40.63
60.24
55.32
Samha
16.13
13.29
70.94
68.81
39.96
Gated DeltaNet-H2
15.91
12.55
48.76
72.19
56.88
57.77
71.33
39.07
41.91
61.55
56.18
Titans (LMM)
15.60
11.41
49.14
73.09
56.31
59.81
72.43
40.82
42.05
60.97
56.82
Table 2: Performance of Titans and baselines on S-NIAH task from RULER benchmark. The best results among simple and hybrid models are highlighted.
S-NIAH-PK
S-NIAH-N
S-NIAH-W
Model
2K
4K
SK
16K
4K
16K
2K
TIT
98.4
98.8
98.0
88.4
64).2
36.6
10.2
4.4
78.8
28.0
4.4
Mumba2
98.6
61.4
31,0
5.4
98.4
42.2
NettaNet
96.8
98.8
98.6
71.4
47.2
15.4
12.8
5.4
46.2
20.0
1.6
Titans (LMM)
99.8
98.4
98.2
96.2
100.0
99.8
93.4
80.2
90.4
89 4
85.8
Samba
98.8
98.0
97.4
97.2
98.8
98.6
96.2
95.6
96.8
90.0
84.0
Gated Del
99.2
97.8
97.4
98.4
98.0
97.8
96.2
95.8
97.4
96.8
88.4
Titans (MAC)
99.2
98.8
99.0
98.4
99.6
98.2
97.6
97.4
98.2
98.2
95.6
Titans (MAG)
99.4
98.0
97.4
97.4
Titans (MAL)
98.8
98.6
98.8
97.8
99.8
98.1
96.8
96.4
98.0
97.4
92.0
memory module achieves the best performance in both perplexity and accuracy measures. Comparing our neural memory module and TTT, which is also gradient- based recurrent model can show us the importance of our weight decay well as the momentum. As discussed earlier, the weight decay can be interpreted as a gating mechanism forget the past data, when it needed. Also, momentum can help better manage the memory by providing additional memory for the surprise metric. While some baselines also take advantage of gating mechanism, e.g., Mamba, Mamba2, and Gated DeltaNet, the superior performance of our neural memory module shows the importance of both our surprise mechanism and having deep and non- non-lincar memory. We further discuss the later in Section 4.4.
Comparing the hybrid models, we found that all three variants of Titans (MAC, MAG, and MAL) outperform both Samba (Mamba attention) and Gated DeltaNet-H2 (Gated DeltaNet atttention). We attribute the superior performance of Titans (MAL) to the power of neural memory module as the architecture design and used attention are all the same. Comparing Titans (MAG) and (MAC), we find that while their performance are close, MAC performs better when dealing with longer dependencies the data. Interestingly, both MAG and MAC outperform MAL variant, which due to using the same modules, we attribute this to the architecture design of these models. This finding is particularly important as the current hybrid models (except Hymba [40]) the literature are using MAL-style combination of recurrent models and attention.
8

---

Nambo
LMM B.
13.4
10.6
DMM
LMM m1)
LTM (La
17.0
13.0
16.5
16.0
80
13.8
12.4
8.6
2003
2020
2005
12000
Sequence
Sequince Length
Sequisce Lengtth
(a) 170M Parameters
(b) 360M Parameters
(c) 760M Parameters
Figure 3: The effect of memory depth on the perplexity. Deeper long-term memory results in better scaling in longer sequences.
Table 3: The results of RULER benchmark on hybrid or Transformer- models.
16K
32K
64K
128K
Avg.
Transformer
81.4
79.2
63.4
49.8
68.5
Samba
83.2
80.8
69.4
34.6
67.0
LongRoPE (Phi3-3.8B)
90.8
87.7
79.8
65.3
80.9
EM-LLM
89.4
86.2
80.4
76.8
83.2
Titans (MAC)
92.0
89.2
85.2
80.4
86.7
4.3 Needle in a Haystack
Scaling a model to longer context window is not always equivalent to being effective for very long sequences [62]. The context length of models. this task, we evaluate the model on retrieving a piece of information (i.e., the "needle") from long distractor texts (i.e., the "haystack"). (S-NIAH) task from RULER benchmark [62] and evaluate Titans and baselines on sequences with length 2K, 4K, 8K, and 16K. The results are reported in Table 2. Neural Memory module achieves the best results compare to baselines in all three tasks. We attribute this superior performance to three key differences of Titans with existing sequence models: (1) Compared TTT, our Neural Memory can better handle the memory capacity by using momentum and also the forgetting mechanism (i.c., weight decay). Therefore, with increasing the sequence length, the performance of Neural Memory does not drop and show a consistent trend; (2) Compared to Mamba2, which has the gating (forgetting) mechanism, Titans have deep non-linear memory, resulting better memory management. Also, contrary to our neural memory and DeltaNet, Mamba2 is not capable of removing a memory and so we can see a significant drop in performance when increasing the sequence length; (3) Compared to DeltaNet, although it is capable of removing memory using delta rule, it cannot erase the memory, lacking forgetting mechanism. Finally, As expected we can see on par or better results when using Titans variants, where the best results correspond to MAC.
(NIAH) task is designed to measure the actual effective
this part,
Single NIAH
Next, compare the larger and hybrid MAC variant of Titans with other hybrid long-context enhance models on harder setups of RULER benchmark. The results are reported in Table 3.
4.4 The Effect of Deep Memory
In this section, we evaluate the effect of deep memory in both wall-clock training time and model performance To this end, focus on different variants of our neural memory module, where LM = 1, 3, 4. We also use Mamba as a baseline for the model performance. For fair comparison, we use the same training process for models and train them on subset of the Pile dataset [63].
We report the perplexity of our models and baselines as the function of the sequence length in Figure 3. Interestingly, with the increase of memory depth. LM, the model can achieve better perplexity over all sequence length. Also, deeper memory modules are more robust to the sequence length when the model has less number of parameters. With the increase of the number of parameters, all models show better performance on longer sequences.
Note that, in this experiment, we only focus on the neural memory module to evaluate the effect of memory depth in the memorization process. Combining neural memory with attention as we do in Titans variants, can additionally euhance the performance of the model over long sequences.

---

Table 4: Ablation Study on Titans. All components of Titans are positively contributing to its performance.
Language Modeling
Reasoning
Long Context
Model
ppl
L.MM
27.01
47.83
92.68
+Attn (MAC)
26.67
48.65
97.95
+Attn (MAG)
25.70
48.60
96.70
+Attn (MAL)
25.91
47.87
96.91
Linear Memory
28.49
46.97
85.34
w/o Convolution
28.73
45.82
90.28
w/o Momentum
28.98
45.49
87.12
w/o Weight Decay
29.04
45.11
85.60
w/o Persistent Memory
27.63
46.35
92.49
4.5
Ablation Study
Finally, we perform ablation studies on the different architectural choices in Titans. We consider our neural memory module as base model and then changing one component time: (1) replacing deep memory with linear memory, removing (2) convolution, (3) momentum in the surprise measure, (4) weight decay (or forgot mechanism), and (5) persistent memory. The results are reported in Table 4. All components of neural memory design are positively contributing to its performance, where the greatest contribution comes from weight decay, momentum, convolution, and persistent memory, respectively.
The Effect of Architectural Design. To evaluate the effect of architecture design, we compare the performance of three represented variants of Titans in three aspects of (i) language modeling, (ii) commen-sense reasoning, and (iii) long context NIAH (BABIL tasks. The results are reported in Table 4. MAC and MAG have close performance language modeling and common-sent -ense reasoning tasks, while MAC achieve significantly better performance in long-context NIAH. Both achieve better performance than MAL. These results along with Figure 10, show trade-off' between fast training and more expressive design.
We provide additional experimental results, including efficiency evaluation, BABILong benchmark, the effect of deep memory as well as Titans performance on state tracking tasks in Appendix G.
Conclusion
In this paper, present neural long-term memory that, meta in-context learner, learns to memorize at test time. The neural memory module is a recurrent model in nature, and is adaptively memorizing tokens that are more surprising present Titans architectures, and its three variants, in which we suggest to incorporate the memory module (1) context, (2) gating, and (3) a layer. Our experimental evaluation on diverse tasks tasks validate that Titans are more effective than Transformers and recent modern linear recurrent models, specifically for long context.
close surprising tokens. Using this memory,
Limitations. Next, we highlight some limitations and potential future directions:
First, the long-term neural memory is a test time memorizer and so the in-context data is compressed into its parameters. In some cases, this memorization can lead to security, safety, and alignment challenges. Future study can to understand the long-term memory can cause security, safety, and alignment issues, and if so, how one can mitigate it.
The main focus of our paper has been on the design of long-tern memory system that can enhance the performance of the attention as short-term memory. There are, however, several important aspects that requires further study. For example, we leave the theoretical guarantee on the capacity and memory management of Titans as a future study. It is notable that after the initial public version of this work, Behrouz et al. [64] have provided theoretical results on the memory capacity and memory management of deep memory modules.
10

---

References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https://procceede neurips pttpppeeeeleeeepppeeeeee 2017/file 200772222228ee223333eeee111a6aa2
pdf.
[2] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint orXiv:2001.08361 2020.
[3] Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36, 2024.
[4] Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua. Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 157-177,,, 2024.
[5] Chao Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In Proceedings of the JEEE/CVE conference on computer vision and pattern recognition, pages 284-293, 2019.
[6] Haoyi Zhou, Shanghang Zhang. Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[7] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mas: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156-5165. PMLR, 2020.
[8] Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. Polysketchformer: Fast transformers via sketching polynomial kernlls. In Forty-first International Conference Machine Learning. 2024. URL
[9] Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Forty-first International Conference on Machine Learning, 2024. URL https::t//// /ttttps://opereviiii net/ferum i d=ia5XvxFUJT.
[10] Shida Wang. Longssm: On the length extension of state-space models in language modelling. arXiv preprint arXiv:2406.0208 2024.
[11] John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-22 2558, 1982.
[12] Jürgen Schmidhuber and Sepp Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
[13] Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowyez, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length general- ization in large language models. Advances in Neural Information Processing Systems, 35: 38546-38556, 2022.
[14] Zhen Qin, Yiran Zhong, and Hui Deng. Exploring transformer extrapolation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18897-18905, 2024.
[15] Nelson Cowan. What are the differences between long-term, short-term, and working memory? Progress brain research, 169:323-333, 338, 2008
[16] Scott Terry. Learning and memory: Basic principles, processes, and procedures. Routledge, 2017.

---

[17] Hideyuki Okano, Tomoo Hirano, and Evan Balaban. Learning and memory. Proceedings of the National Academy of Sciences, 97(23):12403-12200 2000.
[18] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
[19] Daniel Willingham. Systems memory in the human brain. Neuron, (811)5-8, 1997.
[20] George Mandler. The structure of value: Accounting for taste. In Affect and cognition, pages
3-36, Psychology Press, 2014.
[21] Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang. Sanmi Koyejo, al. Learning to (learn test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620, 2024.
[22] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems, 35:29374-29385, 2022.
[23] Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In International Conference on Machine Learning, pages 9639-9659, PMLR, 2022.
[24] Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. Beyond memorization: Violating privacy via inference with large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttps/////p/eeeee.e/ommom werperrrviee..ee/formm net/forum? 1d=kmmOBhQk7p.
[25] Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Re- thinking I1m memorization through the lens of adversarial compression. arXiv preprint wrXiv:240.115166, 2024.
[26] Danny Leybzon and Corentin Kervadec. Learning, forgetting, remembering: Insights from tracking IIm memorization during training. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 43-57, 2024.
[27] Reza Bayat, Mohammad Pezeshki, Elvis Dohmatob, David Lopez-Paz, and Paseal Vincent. The pitfalls of memorization: When memorization hurts generalization. arXiv preprint arXiv:2412.07684, 2024.
[28] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, pages 7693-7702. PMLR, 2019.
[29] A Nichol. On first-order meta-learning algorithms. arXiv preprint arxiv: 1803:0099399999 2018.
[30] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024.
[31] Antonio Orvieto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pages 26670-26698. PMLR, 2023.
[32] Jianyu Zhang, Niklas Nolte, Ranajoy Sadhukhan, Beidi Chen, and Léon Bottou. Memory mosaics. arXiv preprint arXiv:2405.0mm 4065000904444.. 2024.
[33] Edoardo Cetin, Qi Sun, Tianyu Zhao, and Yujin Tang. An evolved universal transformer memory. arXiv preprint arXiv:2410.13166. 2024.
[34] Vincent-Pierrs Berges, Barlas Oguz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, and Gargi Gosh. Memory layers scale. arXiv preprint arXiv:2412.09764 2024.
[35] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint 4rXiv:2412.066464,22 2024.
12

---

[36] Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. Longhorn: State space models are amortized online learners. arXiv preprint 1rXiv:2407.14207, 2024.
[37] Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations. 2023. URL.
wtwpeeewwwweeeeeonnenenneieeeeee Ad= AHAAA11HH3AAaas
[38] Songlin Yang, Bailin Wang, Yu Zhang. Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In The Thirty- Anmual Conference on Neural Information Processing Systems, 2024, URL URIhhtts:: wttphhtsss///ppeeee http:///ppeeeeeeeeeeeeee wtw.iiewwwe///ooum 1d=y@Rm4VMRRHH.
[39] Sainbayar Sukhhaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. Augmenting self-attention with persistent memory. arXiv preprint arXiv: 0/700, 2019.
[40] Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sumil Mahabalesh- warkar, Shih- Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, et al. Hymba: A hybrid-head architecture for small language models. arXiv preprint arXiv:2411.13676 2024.
[41] Badri N. Patro and Vijay S. Agneeswaran. Simba: Simplified mamba-based architecture for vision and multivariate time series, 2024.
[42] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:/606.084/ 2016
[43] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
[44] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[45] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL httpss://ppnttt http//ppnrviiv....mmmee/.. eview. www.ee.w.eeeeeeeeeee net/forum?
id=tEYskw1VY2.
[46] Harsh Mchta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. In The Eleventh International Conference on Learning Representations, 2023. URL aattps:/ae /ttettss://pprreeee //tp://v/ee..eeee/////mo weciiiiwwweeeeeeee
[47] Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thihty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreviae htt//orrmmminnnSScnmmmmm
[48] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini 2302. 2023.
[49] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang. and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.0862) 2023.
[50] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprini arXivv2303.0877 2023.
[51] Aleksandar Botev, Soham De, Samuel L. Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, et al. Recurrentgemma: Moving past transformers for efficient open language models. arXiv preprint arXiv:2404.07839 2024.
13

---

[52] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[53] Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Igorevich Sorokin, Artyom Sorokin, and Mikhail Burtsev. BABILong: Testing the limits of LLMs with long context reasoning-in saatiing-i----aeee aan---ayyttck. In The Thirty- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL http:tttoppppeeeeeieeee//////// wtt.rrenrrrieeeeeeeoooornnn net/forum? idmu7n2C084BQ.
[54] Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint 3xXi:22403.19885 2024.
[55] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211. 2022.
[56] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint wwXiv:2310.06666 2023.
[57] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross coos-dimension depen- deney for multivariate time series forecasting. In The eleventh international conference on learning representations, 2023.
[58] Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. Long-term forecasting with tiDE: Time-series dense encoder. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: https://opent http://///ppeeeee.eeeaee wtw.:e/wwwee/foommmmii pCbC3aQB5W.
[59] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023. URL https: /owpnevvieeeeeeeeeeeeeeeee wwwpww.weeeoooooo net/forum? Ad=ju_Uqu3840
[60] Ailing Zeng. Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
[61] Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation linear mapping. arXiv preprint arXiv:2305.1077 2023.
[62] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https: //oprr/viee/er.ee... net//formm?ice<koBBb66777
[63] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling, arXiv preprint arXiv:2101. 201.00027, 2020.
[64] Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, and Vahab Mirrokni. Atlas: Learning optimally memorize the context at test time. arXiv preprint 2rXiv:2505.2373: 2025.
[65] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural In- formation Processing Systems, volume 35, pages 16344-16359, Curran Associates, Inc., 2022. URL https: / httppe/e/////oooreeinn....me http://rrreeeeiiggeeeeieeep....e nwwrips. wwtprrppppccpppprrrpppppppppp hpwperpppr__aae files/ /ttes/ppppe/222//tieee http:///000000000200200000000000 Conereeoneee pdf.
14

---

[66] Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https: wowporevie..eeeeeeeeeeeeeeeeeeee
[67] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger. Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021. URL https: // //openreview...//////// nnt. www.orrmmmddduuuuu
[68] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Anna Korhonen, David R. Traum, and Lluís Mårquez, editors, ACL (1), pages 2978-2988. Association for Computational Linguisties, 2019. ISBN 978-1-950737-44-2.
[69] Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. Advances in Neural Information Processing Systems, 34: 34:17413-1143 17426, 2021.
[70] Simran Arora, Sabri Byuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Re. Simple linear attention language models balance the recall- throughput tradeoff. In Forty-first International Conference Machine Learning, 2024. URL
[71] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pages 9355-9366.
PMLR, 2021.
[72] Yaroslav Aksenov, Nikita Balagansky, Sofia Maria Lo Cicero Vaina, Boris Shaposhnikov, Alexey Gorbatovski, and Daniil Gavrilov. Linear transformers with learnable kernel functions are better in-context models. arXiv preprint arXiv:2402.10644, 2024.
[73] Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint
2024.
[74] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prud- nikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04511 2024.
[75] Bernard Widrow and Marcian Hoff. Adaptive switching circuits, 1988.
[76] JH Schmidhuber. Learning to control fast-weight memories: An alternative recurrent nets. accepted for publication in. Neural Computation, 1992.
[77] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines, 2014. URL https:
// /arxivv........ .rgg abs/1410........ 5401.
[78] Donald Olding Hebb, The organization of behavior: A neuropsychologiial theory, Psychology press, 2005.
[79] DL Prados and SC Kak. Neural network capacity using delta rule. Electronics Letters, 25(3): 197-199, 1989.
[80] Tsendsuren Munkhdalai and Hong Yu. Neural semantic encoders. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 1, page 397. NIH Public Access, 2017.
[81] Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalcarned neural memory. Advances in Neural Information Processing Systems, 32, 2019.
[82] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond lin- ear transformers with recurrent fast weight programmers. Advances in neural information processing systems, 34:7703-7717, 2021.
15

---

[83] Matteo Tiezzi, Michele Casoni, Alessandro Betti, Tommaso Guidi, Marco Gori, and Stefano Melacci. On the resurgence of recurrent models for long sequences: Survey and research opportunities in the transformer era. arXiv preprini UrXiv:2402.081322 2024.
[84]
Bo Peng, Eric Alcaide, Quentin Gregory Anthony, Alon Albalak, Samuel Areadinho, Stella Biderman, Huanqi Cao. Xin Cheng, Michael Nguyen Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartiomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan S. Wind, Stanislaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023, URL https: / www.oevieew..oeeoo wwt.:/etttoooommmonnomtememmmmr
[85] Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representutions, 2022. URL
[86] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with Istm. Neural computation, 12(10):2451-444 2000.
[87] Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, (6(10):2222-22333 2016.
[88] Jos Van Der Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate. arXiv preprint arXiv:/804.04840 2018.
[89] Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang. Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. In The Eleventh International Con- ference on Learning Representations, 2023. URL https::/ wwt.///mnneoooooo g40TKRKfS7R.
[90] Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892, 2024.
[91] Dzmitry Bahdanau. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv::449.0444 409.0473, 2014.
[92] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021.
[93] Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex atten- tion: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.054999 2024.
[94] Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:1 1111-10799999 2022.
[95] Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, and Mikhail Burtsev. Associative recurrent memory transformer. arXiv preprint urXiv:2407.0488 2024.
[96] Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: A memory-augmented transformer for sequence modeling. arXiv preprint arXiv:2010.0680 2020.
[97] Luca Zancato, Arjun Seshadri, Yonatan Dukler, Aditya Golatkar, Yantao Shen, Benjamin Bowman, Matthew Trager, Alessandro Achille, and Stefano Soatto. SOMOJO: Hybrid state space realizations of foundation models with eidetic and fading memory. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: // /popnrevvee..ooo net
16

---

[98] DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur, Block- recurrent transformers. Advances neural information processing systems, 35:33248-33261. 2022.
[99] Yukun Feng, Feng Li, Ziang Song. Boyuan Zheng, and Philipp Koehn. Learn remember: Transformer with recurrent memory for document-level machine translation. arXiv preprint arXiv:2205.0 2022.
[100] Aydar Bulatov, Yuri Kuratov, Yermek Kapushev, and Mikhail S Burtsev. Scaling transformer to tokens and beyond with rmt. arXiv preprint arXiv:2304. 1062, 2023.
[101] Zhiwei Wang, Yao Ma, Zitao Liu, and Jiliang Tang. R-transformer: Recurrent neural network enhanced transformer. arXiv preprint (0XXxi:907.0555 2019.
[102] Yinpeng Chen, DeLesley Hutchins, Aren Jansen, Andrey Zhmoginov, David Racz, and Jesper Andersen. MELODI: Exploring memory compression for long contexts. In The Thirteenth International Conference on Learning Representations, 2025. URL htt.hht::://ppeeeraaaate pet/forummid=TTGGRR
[103] Zexue He, Leonid Karlinsky, Donghyun Kim, Julian McAuley, Dmitry Krotov, and Rogerio Feris. Camelot: Towards large language models with training-free consolidated associative memory. arXiv preprint arXiv:2402. 13449, 2024.
[104] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020. URL https: hatp::://pperreee //pp//eveeeeeee.e///////ee wwwpeeeiieeeeeeefrreeeeee net/forum? id=Hk1BjCEKvH.
[105] Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang. Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang, and Julian McAuley. MEMORYLLM: Towards self-updatable large language models. In Forty-first International Conference on Machine Learning, 2024. URL httptttts//pmeeeee http:::://///eeeeeee/ee/eme wwwpreewwwwweeeeoouueenieeeee net//orrumid=poOKWeaa
[106] Yu Wang, Chi Han, Tongtong Wu, Xiaoxin He, Wangchunshu Zhou, Nafis Sadeq, Xiusi Chen, Zexue He, Wei Wang, Gholamreza Haffari, et al. Towards lifespan cognitive systems. arXiv preprint arXiv:2409. 13265, 2024.
[107] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free long-context extrapolation for Ilms with 2024.
efficient context memory. arXiv preprint
[108] Zafeirios Fountas, Martin A Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. Human-like episodic memory for infinite context llms. arXiv preprint arXiv:2407 240,00000 2024.
[109] Luca Zancato, Arjun Seshadri, Yonatan Dukler, Aditya Sharad Golatkar, Yantao Shen, Ben- jamin Bowman, Matthew Trager, Alessandro Achille, and Stefano Soatto. mojo: Hybrid state space realizations of foundation models with eidetic and fading memory. Advances in Neural Information Processing Systems, 37:130433-130466 2024.
[110] Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv preprint 24X.0:2404.07143, 2024.
[111] Hung Lc, Truyen Tran, and Svetha Venkatesh. Self-attentive associative memory. In Interna- tional conference machine learning, pages 5682-5691. PMLR, 2020.
[112] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916. 2014.
[113] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015.
17

---

[114] Jürgen Schmidhuber. Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets. In ICANN'93: Proceedings of the International Conference on Artificial Neural Networks Amsterdam, The Netherlands 13-16 September 1993 3. pages 460-463. Springer, 1993.
[115] Marcin Andrychowicz. Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
[116] Léon Bottou and Vladimir Vapnik. Local learning algorithms. Neural computation, 4(6): 888-900, 1992.
[117] Hao Zhang. Alexander C Berg, Michael Maire, and Jitendra Malik. Svm-knn: Discriminative nearest neighbor classification for visual category recognition. In 2006 IEEE Computer Society Conference on Computer Vision and Patlern Recognition (CVPR'OR) volume 2, pages 2126-2136. IEEE, 2006.
[118] Vidit Jain and Erik Searned-Mille Online domain adaptation of pre-trained cascade of classifiers. In CVPR 2011. pages 577-584. IEEE, 2011.
[119] Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian. Online model distillation for efficient video inference. In Proceedings of the IEEE/CVF International conference on computer vision, pages 3573-3582, 2019.
[120] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpee://pppeeveeeeeeeeee net wwwt/eeuunnn..ee odw.oruunnddn7755aa50 sS512VF.
[121] Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. LM-infinite: Zero-shot extreme length generalization for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Pupers), pages 3991-4008, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.1 1/201/1.2222.2..lllooolo 222. URL https: http:///ccaantthoggoy. www.oog///222 naacl-long 222.
[122] Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations, 2023. URL WtRLttt:////peeeeee /ttp/////eeeee.eee////eeee http/oorummidcOZZyyy00m0
[123] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL
https://openrevia
[124] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10..8863/v1/PP1-111 URL https: // ac aclantholoyy org/P16-1144/
[125] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
[126] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluí s Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, pages 4791 4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 18653/ 1/119- 1472. URL https: / aalaaaaannntooyyoooo wtw.oo/ooo...g/1-11777...
18

---

[127] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106, 2021.
[128] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 17X3vv8003005433 2018.
[129] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLLP), pages 4463-4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 1/D19- 19544.44. URL https: // /aclanthologyy org/D19- 1454/.
Empirical Methods
[130] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2924-2936, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 1181118611119/// 1/000- URL htt.htt://caaaatllllloo/ooo
[131] Bo Peng. RWKV-LM, August 2021. URL
[132] Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint arXiv:2406.0773 07522, 2024.
[133] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
[134] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intees intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459- 9474,, 2020.
[135] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint MrXiv:2412.15115 151 15, 2024.
[136] Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1):25, 2023.
[137] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch- Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024.
[138] Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, et al. Mechanistic design and scaling of hybrid architectures. arXiv preprint 1rXiv:2403.17844, 2024.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction both accurately state the main theoretical contri- butions of the paper and accurately discuss our experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our method in the main text, including its scope, assumptions, and potential areas for future improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification: All assumptions are stated in the body of the theorems directly. As much of the proofs as possible is provided in the main paper text. We provided all detailed proofs of lemmas and theorems introducing this work.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
21

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We plan to provide data and code via Github after the paper is made publicly available online.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nig) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wwtppblcc/guuiiicccccccccccoocat
wwwsssssssiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yee Yes]
Justification: Detailed descriptions of the experiment and the associated code are provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: To account for variability and support statistical validity, we report standard deviations computed over multiple runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: To facilitate reproducibility. we report accelerator type, memory capacity, and runtime for key experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:://eeeu/eeeeeeeeeeee httpp//s.c..pplbttee/cccccuaaua
Answer: [Yes]
Justification:
There are no violations of the NeurIPS Code of Ethics in this paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provided discussion in the Appendix about the impact of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit all external assets and explicitly acknowledge their licenses and terms of use.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thoode cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any user studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No usage of LLMs in the methodology or experiments.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //tp/r/eesppp//eeee ceturriss.ccCCofeeeeenece cc/Connferences/22251L.. for what should or should be described.
26

---

A Preliminaries
In this section, we discuss the notation and some background concepts that use through the paper. We let x E I xxd be the input, M be a neural network (neural memory module), Q K, V be the query, key and value of the attention mechanism, and M be the attention mask. When segmenting the sequence,
use ( to refer to the i-th segment. Through the paper, we abuse the notation and use subscripts to refer to a specific element of a matrix, vector, or segments, For example, we let { be the j-th token in the i-th segment. The only exception is subscripts with t, which we reserved to index recurrence over time, or the state of a neural network at time t. Given a neural network NV and a data sample x, we use (x)) (resp. (x)) to refer to the forward pass with (resp. without) weight adjustment. Also, we abuse the notation and use A/( to refer to the k-th layer of the neural network. In the following, we first, discuss the backgrounds for attention and its efficient variants followed by a review of modern linear RNNs. Finally, we discuss a memory perspective of these architectures that motivates us to design Titans.
A.1 Backgrounds
Attention. Transformers [1] as the de facto backbone for many deep learning models are based on attention mechanism. Given input x E RNxda, causal attention computes output y E R2 Nx based on softmax over input dependent key, value, and query matrices:
Q = wWQ
k=IWK = V=zVv
(14)
exp Q
K dm V
y E exp (Q, Kz/V
(15)
where WQ. WK, and Wv E RR are learnable parameters. Despite the power and effectiveness in recall, transformers need at least N x d operators to calculate the output, resulting in larger memory consumption and lower-throughput for longer sequences.
Efficient Attentions. To improve the memory consumption and throughput of softmax attention for longer sequences, various studies focused on I/O aware implementations of attention [65, 66], designing more efficient attention mechanisms by sparsifying the attention matrix [67-69], approxi- mating the softmax [70], or developing kernel-based (linear) attentions [8, 71, 9, 72j. In this part, we focus on the later, i.e., linear attentions, where the softmax in standard attention is replaced with an alternative kernel function ((,,), .), such that ((x, y) = ((x)((v). Accordingly, the attention can be written as:
(QQ Kÿ)
V;= Σ (Q.)) (K,,) J=1 1=11 (QQ)) ((₁,) V, Ej
y; Σ J=1 Σ (0 Kt)
(16)
((Q:) E ((Ke)
resulting in a tigher-throughput as terms Σn= Φ(K) ( and E/= oi K2) are re-using in each step. When choosing the kernel as identity matrix [49], the above formulation can also be written in recurrent format:
M. = MM K, Ve,
(17)
ye = Q.M,, which allows efficient inference for linear attentions.
(18)
Modern Linear Models and Their Memory Perspective. As discussed earlier, one can define learning as process for acquiring effective and useful memory. Building upon this, one can see the hidden state of Recurrent Neural Networks (RNNs) as a memory unit, which the model aims to compress the information into. Accordingly, in general form of recurrent neural network, the hidden state can be treated as a memory unit and the recurrence process can be split into the read and write operations in the memory unit. That is, we let E I. Nxda be the input, M E Idd is the memory unit, and y d is the output, then the general form of the recurrent neural network is defined as:
M = S(M+-1,
Write Operation
(19)
y: = z(Me.
Read Operation
(20)
27

---

where f(...) is the read and ((...) )) is the write corresponding functions. Note that here the subscript of M, shows the state of the memory at time t.
In this perspective, the recurrence formula of linear Transformers (see Equation 17) is equivalent to additively compress and write keys and values, (K,, Vi), V)), into a matrix-valued memory unit M, Therefore, when dealing with long context data, this additive nature of the process results in memory overflow, significantly damaging the performance of the model. To address this, studies have focused on two promising directions: (1) Adding forget mechanism: several studies have presented adaptive deetaddeenndntt forgetting gate mechanisms for linear models, where it can erase the memory when it is needed. As examples of such models, we refer to GLA [9]. LRU [31]. Griffin [73], xLSTM [74], and Mamba2 [30], which the later also connected the discretized version of traditional state space models [45].(2) Improving the write operation: To overcome the additive nature of memory write operation traditional recurrent models, Widrow and Hoff [75] presented Delta Rule, in which before adding memory (i.e., pair of key and value). the model first removes its past value. To enhance the parallelizable training and scaling, Yang et al. [38] present a fast paralellizable algorithm. Finally, very recently, Yang et al. [35] improved the DeltaNets by adding forget gate.
Memory Modules. Memory has always been one of the core parts of the neural network designs [76, 12, 77, 32]. The idea of seeing linear layers as the key-value (associative) memory system backs to fast weight programs, in which dynamic fast programs are incorporated into recurrent neural networks to serve as writable memory [76]. The two learning rules of Hebbian [78] and delta [79] are the most popular learning rules for fast weight programs, which have been extensively explored in various studies [80, 76, 81, 71, 82, 38, 35]. All these models, however, are based on momentary surprise, missing the token flow in the sequences (see Section 2.1), and most of them lacks forgetting gate, resulting in a poor memory management.
We further discuss the connection of our architectures with recent models in Appendix F. Additional related work are discussed in Appendix
B
Related Work
There are diverse perspectives that can independently lead to the design Titans or its components. Accordingly, to further situate our work in broader context, review three categories of studies:
B.1 Linear Recurrent Models
Recently, to address the computational cost of Transformers in both training and inference, linear recurrent models have attracted much attention [83], mainly due their fast inference and training. The first generation of models-suee such as RetNet [49], LRU [31], RWKV [84], S5 [37], and S4 [85]-uses data-independent transition matrix/decay mechanism. The second generation of such models started to incorporate gating mechanism, widely used techniques in traditional RNNS [86-88], into such linear architectures e.g., Griffin [73], SSMs [89, 54, 30, 45], RWKV6 [90]. The third generation of linear recurrent models are based more complex memory updating rule based meta-learning online learning, and/or delta-rule, resulting in more expressive and effective models such as: Longhorn [36], Gated DeltaNet [35], TTT [21], and DeltaNet [71]. Our LMM model can be seen as the next generation of such models, which we incorporate the token flow into the memory updating mechanism, having more powerful memory updating process. See Appendix F for a detailed discussion of different recurrent models and Titans.
B.2 Transformer-based Architectures
Transformers. Transformers [1] as the de facto backbone for many deep learning models are based on attention mechanism [91]. They, however, suffer from quadratic computational cost, limiting their ability to scale to long context window. To improve the memory consumption and throughput of softmax attention for longer sequences, various studies focused on I/O aware implementations of attention [65, 66), designing more efficient attention mechanisms by sparsifying the attention matrix [67-69, 92, 69, 93], approximating the softmax [70], or developing kernel-based (linear)
attentions [8, 71, 9, 72].
28

---

Segment-based Transformers. Another line of research to improve the efficiency of Transformers based or Chunk Transformers [68]. The main drawback of chunk Transformers that segments are fully separated and so the context window is limited to the length of the chunks. To address this issue, various studies discuss the importance of memory so can help the model to transfer information across chunks [94-99, 98, 100, 101, 96, 97]. The key differences of Titans with these models are: (1) The memory in such models are simple small size vectors, lacking expressive power to compress complex information; (2) The memory module lacks forget mechanism, leading to a fast memory overflow; (3) only focus on momentary surprise, missing the information flow. More specifically, recalling Recurrent Memory Transformers (RMT) [94, 95, 100], one treat Titans (MAC) as the generalization of RMT, where we use a neural memory module instead of vector- valuud small size memory. In addition to models with fixed-size vector-values memory, MELODI [102] also discusses more expressive memory system where the memory is updated through append operation over time. This model can be seen special case of Titans, where memory is static.
segment- sranentt-asedddeee
Memory for Large Language Models. Another interesting research direction has been to incorporate external memory modules to LLMs after training [103-105]. Such models are different from our approach as we incorporate the memory as a part of initial architecture and so we train it in an end-to-end manner. Also, most of these explicit memory modules suffer from the same limitations as chunk-basee Transformers (mentioned above). For a detailed discussion of such models, we refer to the recent study of Wang al. [106]. Several studies also discuss non-parametrie memory modules for Transformers [107, 108]. While these methods are fundamentally different from ours be used complementary components), we have provided experimental results to compare such methods with Titans in long context tasks. Recently, Zancato et al. [109] also present a hybrid model that uses state-space-moodee models as the long-term memory for attention module. This design is fundamentally different from ours as: (1) Memory: the fading memory is a Mamba, which uses vector-valued memory, but Titans use neural deep architecture as the memory, resulting in a potentially more powerful memory. (2) Token Selection: While we use a surprise metric based on gradient descent with momentum (measures the encoding of tokens). B' B'MOJO uses a dictionary and finds the shortest sequence that is currently unknown to assign. This selection is based on the actual sequence not the encoding (conceptual representation), and so might face issue in memorizing different words with the same concept. (3) In Titans, Memory helps attention to have access to relevant long past, and then attention helps the memory to filter important information of current context (inter-connected memory system), while in B NOMOO,, only fading memory is helping attention. (4) Forget gate, (5) Past surprise metric in Titans can consider the flow in the tokens.
(and potentially
B.3 Test Time Training and Fast Weight Programs
Memory Design and Augmentation with Memory. In the literature, substantial research effort have been toward designing memory modules that are capable of either memorizing the knowledge abstraction (e.g., persistent memory) [39], or memorizing the data-dependent information (also known as contextual memory), through recurrence [97, 94, 95], Transformers [110, 32-34, 111, 99], gradient [81, 23], or other learning paradigms [1 12, 113]. These memory models, however, either (1) are based on momentary surprise, missing the data flow and events, (2) lack forget mechanisms to remove the memory, leading to a fast memory overflow (3) are fixed-size shallow (matrix valued) memory, resulting in poor performance in long context, and (4) are based on fixed parameters at test time, lacking test time adaption.
Fast Weight Programs. The idea of seeing linear layers as the key-value (associative) memory system backs to fast weight programs, in which dynamic fast programs are incorporated into recurrent neural networks serve as writable memory [71, 76, 1 144]. The two learning rules of Hebbian [78] and delta [79] are the most popular learning rules for fast weight programs, which have been extensively explored in various studies [80, 76, 81, 71, 82, 38, 35]. All these models, however, are based on momentary surprise, missing the token flow in the sequences (see Section 2),., and most of them lacks a forgetting gate, resulting in a poor memory management. In this context, deep memory has also been discussed by Irie et al. [82], where they suggest 2-layer MI. as the memory. However, there are critical differences with Titans: (1) The update rule in Titans is based on both momentary and past surprise, (2) The deep memory in Titans uses input-dependent per-layer parameters, which allows
29

---

Dea Nowey)
Vewer Dbet: and Menwcy
(Toakirr Memser
Baa and Leaglers Menwey
Dbor Micsiry
Sungiees Mecvrr
Postiic Strneyy
Heavyy
Longleza Menuyy
Desttion Hemeyy
(a) Memory as a Context (MAC). We segment the sequence and use full causal attention in each window. Again, the first Np tokens are persistent memory and the next Ni are long-term memory tokens
(b) Memory as Gating (MAG). We use sliding win- dow attention (SWA) as short-term memory and our neural memory module as long- memory, com- bining by gating.
Figure 4: Attention masks for different variants of Titans.
each layer flexibly update its weights. Without this, the 2-layer memory do not show improvement in larger scale; (3) the desing of Titans is parallelizable and can be trained in larger scales.
Test Time Training. The key ideas of learning at test time or learning to learn (i.e., [115]) backs to very early studies on local learning [1 16), in which each test data sample is trained on its neighbors before making a prediction [117, 22]. This approach further has shown promising performance in vision tasks [1 [118, 119], mostly due to their ability to mitigate
samples. The most similar studies ours in this direction are MNM [81] and TTT-layer [21], which discussed the key differences in Appendix F.
C Persistent Memory
Next, we discuss the motivation of persistent memory from three perspective:
Memory Perspective. As discussed earlier, our neural long-term memory is contextual memory, in which all parameters are input-dependent. An effective memory system, however, also needs input-independent parameters to store the abstraction the task knowledge. That is, mastering a task requires the memorization of the knowledge that how the task can be done, and these parameters are responsible for storing such knowledge.
Feedforward Network Perspective. In the Transformer architectures, there are fully connected layers after the attention module, which are shown to be similar to attention weights but with tata-independent parameters. That is, Sukhbaatar et al. [39] showed that replacing the ReLU in fully connected layers with Softmax can results in an attention-like weights, in which weights are data-independent:
(21) In fact, WK and Wy acting similar to K and V matrices in attention module when they input-independent. The persistent memory weights are expected to have the same functionality, meaning that using them in the first part of the sequence leads having input-independent attention weights [39].
FFN(x) Wy Softmax (Wxz).
Technical Perspective. Attention with causal mask has implicit bias toward initial tokens in the sequence, and attention weights almost always highly active for initial tokens, resulting in performance damage. From the technical perspective, these learnable parameters at the start of the sequence can mitigate such effect by redistributing the attention weights more effectively [120, 121].
D How to Incorporate Memory
D.1 Memory as Gate
The overall attention mask of this design is shown in Figure 4b. In this design, sliding window attention acts precise short-term memory, while the neural memory module is acting as fading
30

---

Test Time
Neurai
Conbextual
Memory
Learning
Allenaiin
Learning
Core
Sequence
Persintent Menory
Fixedd
Learnable Indepeddetet Weights
Figure 5: Memory as a Gate (MAG) Architecture. This architecture, similarly, has the three branches of (1) core, (2) contextual memory, and (3) persistent memory. It, however, incorporates only persistent memory into the context and combine memory with the core branch using gating mechanism. test time, the behavior is the same as Figure 2.
Teet Time
Noural Memoey
Conbextual
Memory
Learning
In- Ce-Contert Learning
Atlestioo
Core
Sequencn
Persiatent
Memory
Learzable Dats indepenneeeodeett Weights
Figure 6: Memory Layer (MAL) Architecture. In this architecture, the memory layer is responsible to compress the past and current context before the attention module.
memory for the model. This architecture design can also be seen multi- architecture where the structure of heads are different [40].
D.2 Memory as Layer
The last variant uses the neural Memory As a Layer (MAL) of a deep neural network (see Figure 6). This architecture design is common in the literature, where the hybrid models stack recurrent models with full or sliding window attentions. Given input x, we have:
[p
P2
PN,] || x,
(22)
M()),
(23)
SW-Attm (y).
(24)
where SW-Attn is sliding window attention. The main drawback of this design that the power of the model is limited by each of the layers and so it cannot take advantage of the complementary data processing of attention and neural memory module. our experiments, for evaluating memory in this design, use similar architecture as H3 [122], where we replace the the sequence model with our neural memory module (LMM).
Memory Without Attention. Although in the above, we discussed MAL as the combination of LMMs and attention in sequential manner, one simple variant of MAL is to treat LMM as sequence model without any attention. From the memory perspective, as discussed in Section 1, we expect each part of the memory system to work independently, even if other components are disturbed. Therefore, long-term memory module should still powerful model even without short-term memory (i.e., attention). We refer to this variant LMM or Titans (LMM) in our experiments. We provide additional discussions on the connection of Titans and other modern recurrent models in Appendix F.
31

---

Table 5: Architectural Details.
Model
Block
Dim
Head
Peak LR
Token
170M
12
768
16
3e-3
15B
340M
24
1024
16
1.5e-3
15B
760M
24
1536
16
1.25e-3
30B
1.3B
18
2048
7c-4
100B
E Experimental Details
E.1 Architecture Details
We report the number of blocks, heads, size hidden dimension, and peak of learning rate in Table 5. In the training. we follow the training procedure of Yang al. [35], and use LLama tokenizer with a vocabulary size 32K and use training length of 4K tokens (2K for SWA). We fixed the persistent memory size (# tokens) to 128, and use 256 memory tokens encode the past data (i.e., output of the long-term memory). We employ AdamW optimizer with learning rate of 4e-4 with cosine annealing schedule with batch size of 0.5M tokens, and weight decay of O. 1. We use: (1) chunk size: which is 16. The smaller the chunk is the better performance we can get but with the cost of slower model; (2) segment size: in which, follow previous studies and use 2048 sliding window (when exists) and 512 as segment size.
E.2 Language Modeling and Common-sense Reasoning Datasets
Following recent studies on linear recurrent models [35, 30, 38], Wikitext [123]. LMB [124], PIQA [125], HellaSwag [126], WinoGrande [127]. ARC-easy (ARC-e) and ARC-challenge (ARC- <) [128], SIQA [129], and BoolQ [130]. Also, the baselines results for 400M models are from the reported results by Yang et al. [35].
F Long-term Memory Module (LMM) as a Sequence Model
In this section, we discuss how LMM as sequence model connected modern linear recurrent models. For the sake of simplicity, we start with a linear memory, where M₁ W, E du In this case, our objective function becomes {(M: 2;) ||M,k, vellz 2 in which we use gradient descent with momentum and weight decay for the optimization. Accordingly, revisiting the recurrent formula in Equation 3:
M, = diag (1 - a.) Mr St
(25)
S, diag (ne) S diag (f:) (M--ik v; kt)
(26)
LMM is Generalized Gated DeltaNet. As discussed by Yang al. [35]. DeltaNet [38] can alternatively be interpreted as online learning problem that optimizes the f || vell
resulting in:
S = S; 6,VL = S; 0,k,,, + d.v,k,
(27)
In this formulation, Gated DeltaNet the same as above but with an additional weight decay term [35]. Comparing Equation 25 and Equation 27, can that setting 7) 0 results in both formulations
to be equivalent. Accordingly, we can say LMM is generalizing the very recent study of Gated
DeltaNet [35] from three aspects:
Momentum-based Rule: The Delta Rule is based on momentary surprise, meaning that the flow of tokens cannot affect the memory update rule. LMM, however, is based momentum rule, which consider both past and momentary surprise.
Deep Memory: While Gated DeltaNet is limited to a linear (matrix-valued) memory as requires finding the closed recurrence form, LMM allows using deep memory module by using gradient-basee based formulation, resulting in higher expressive power.
32

---

Non-Linear Recurrence: While DeltaNet and Gated DeltaNet are based on linear recurrence, our LMM using inter-chunk non-linear recurrence and intra-chunk linear recurrence. This design allows L.MM having a higher expressive power.
Channel-wise Forget Gate: Contrary to Gated DeltaNet, which uses shared scalar forget gate across all channels, LMM uses channel-wise forget gate, which is more expressive.
Here, we discussed Gated DeltaNet as a sample of recent generation of recurrent models. Similar approaches such as RWKV-7 [131] are also using the same formulation and loss function, and so LMM is generalizing all such models. Note that RWKV-7 is also using channel-wise forget gate and so is similar to LMM in that manner.
LMM is Generalized Longhorn. Similar to DeltaNet, Longhorn [36] uses the same loss function but it derives the closed form using implicit online learning:
S = S; (I - 5,k,k,) + v,vkk,
(28)
where 6, 0, T+ It, however, lacks a forgetting gate, resulting in faster memory over- flow. Therefore, in addition two the abovementioned aspects of (1) Momentum boosentumbbased Rule, (2) Deep Memory, and (3) Non-Linear Recurrence, LMM has the advantage of using an additional (4) Forget Gate, leading a better memory management.
LMM is Generalized TTT Layer. To the best of our knowledge, TTT [21], is the only modern linear recurrent models with gradient- -aaeed updating rule. In addition to different architectural designs and also objective functions, our LMM has three key differences with presented TTT layers [21]]
1. Forgetting Mechanism: TTT layers updating memory at each time, without having the chance to forget the past data. Accordingly, when fixing the memory size, the model cannot manage the memory for long sequences. A forget mechanism, such as LMM's, allows clearing the memory when very past information is not needed anymore. We show that in general case, this forget mechanism is equivalent to weight decay and provide a fast method to incorporate it into the parallel training.
2. Momentum-based Update Rule: TTT layers are based on momentary surprise, meaning that the flow of tokens cannot affect the memory update rule. LMM, however, is based on momentum rule, which consider both past and momentary surprise. See Section 2. for the motivation of this design.
3. Deep Memory: While TTT-layers allows for deeper memory, the advantagess/disadvaaaage of such deeper memory modules have not been experimentally evaluated. Also, we observed that without our proposed layer-wise parameters, deep memory cannot show its advantages over shallow memory.
To the best of our knowledge, our neural long-tern memory module is the first linear recurrent model with momentum-based update rule.
Finally, as a key difference with all the above and other recent linear recurrent studies, note that the hybrid variants of modern linear models-such as Griffin [73], DeltaNet [38], Gated DeltaNet [35], H3 [122], Mamba2 [30], Samba [132], etc.-all are based on sequential layer-wise design. We present
Titans to show how effectively one can incorporate such memory modules into an architecture
G Additional Experimental Results
G.1 BABILong Benchmark
In this section, we use hard task from BABILong benchmark [53], in which the model needs to reason across facts distributed in extremely long documents. We follow the original experimental setup and training process in the benchmark. There are two settings: (1) Few-shot setting, in which we use large pre-trained models, and (2) fine-tuning setting, where we fine-tune the MAC variant of Titans to compare it with other fine-tuned baselines. The results for few-shot setting are reported in Figure 9a. In this setup, can Titans outperform all baselines-ic.. Mamba2.88 8B [45].
33

---

Nambo
LMM B.
13.4
10.6
DMM
LMM m:)
LTM (La=
17.0
13.0
16.5
16.0
80
13.8
12.4
8.6
2003
2020
2005
12000
Sequence
Sequince Length
Sequisee Length
(a) 170M Parameters
(b) 360M Parameters
(c) 760M Parameters
Figure 7: The effect of memory depth on the perplexity. Deeper long-term memory results in better scaling in longer sequences.
RWKV-6-7B [90]. RecurrentGemma-9B [51], Gemma-9B [133], Llama3.1-8B [48], GPT-4, and GPT4o-mini [50].
In the fine-tuning setup, we compare the small fine-tuned version of Titans (MAC) with: (i) the fine- tuned version of small models (almost the same number of parameters as Titans) such as Mamba [45], RMT [94], (ii) large models with sirieval--uumentee Generation (RAG) [134] such as Llama3. 8B [48], and (iii) extremely large models such as GPT-4 [50], GPT40-mini, Qwen2.5-72B [135], and Ilama3. 1-70B [48]. Baseline results are reported by [53]. The results of Titans and baselines are reported in Figure 9b. Titans outperform all models even extremely large models like GPT4. Also, compared to Transformer-base with memory models like RMT, Titans show better performance mainly due to their powerful memory.
notable that, in the above we aimed compare sequence modeling backbones the choice of different architectures can be orthogonal to this our work. However, there are other choices of architectures that can provide even better performance than Titans BABILong benchmark. For example, ARMT [95] achieves 98, 98, 98, 98, 98, 98, 97, 95, 93, 77 for 2K, 4K, 8K, 16K, 32K. 64K, 128K, 512K, 1M, 10M sequence length respectively. This performance better than Titans' results. However, one can simply incorporate our memory module into ARMT update and so achieve better performance. validate this claim, we also added new hybrid variant that uses GPT2 + MAC + MAL. The results are 99, 99, 98, 97, 98.96, 96, 95, 94, 79 for the above sequence length, respectively.
G.2 Language Modeling and Common-Sense Reasoning
G.3 The Effect of Deep Memory
In this section, we evaluate the effect of deep memory in both wall-clock training time and model performance?. To this end, we focus on different variants of our neural memory module, where LM = 1, 3, 4. We also use Mamba as a baseline for the model performance. For a fair comparison, we use the same training process for all models and train them on subset of the Pile dataset [63].
We report the perplexity of our models and baselines as the function of the sequence length in Figure 3. Interestingly. with the increase of memory depth, LM, the model can achieve better perplexity over all sequence length. Also, deeper memory modules are more robust to the sequence length when the model has less number of parameters. With the increase of the number of parameters, all models show better performance on longer sequences.
We also evaluate the effect of memory depth (IM = 1, 2, 3, 4) on the training throughput. We report the training throughput (the number of tokens per second) Figure 8. All models scale linearly with respect to the context length (i.e., constant trend in the number of tokens per second with respect to sequence length). Also, by increasing the memory depth, as expected, we can see a linear trend that a deeper memory results in a slower training. Therefore, not always efficient to use deeper memory modules, showing a trade-off between effectiveness and efficiency.
the function of sequence length in
Note that, in this experiment, we only focus on the neural memory module to evaluate the effect of memory depth in the memorization process. Combining neural memory with attention as we do in Titans variants, can additionally euhance the performance of the model over long sequences.
34

---

Table 6: Performance of Titans and recurrent- and Transformer-basee baselines on language modeling and common-sense reasoning tasks. Hybrid models are marked with The best results among simple and hybrid models are highlighted.
LMB. ppl
PIQA
Hella. acc n
Wino. acc 1
ARC-e acc
ARC-c acc n
SIQA acc
BoolQ ncc:
Avg.
ppl
acc
ace
340M params
1SB tokens
Transforme++
31.52
34.76
50.53
45.21
24.05
36.81
58.24
42,92
RetNet
32.50
49.73
28.
62.61
34.15
50.91
44,27
23.62
36.79
59.72
42.54
GLA
24.29
Mumba
30.83
40.21
29.94
63.79
35.88
49.82
49.24
24.56
35.41
60:07
43.59
DeltaNet
28.65
47.30
28.43
63.52
35.95
49.63
52.68
25.37
37.96
58.79
44.04
TTT
27.44
34.19
30.06
63.97
35.71
50.08
53.01
26.11
37.32
59.83
44.51
Gated DettaNet
27.01
30.94
34,11
63.08
38.12
51,60
55.28
26.77
34,89
59.54
45.42
Titans (LMM)
26.18
29.97
34.90
64.73
39.61
51.85
55.60
34.52
59.99
46.17
Titans (MAC)
25.43
28.1
36,00
65.32
40.35
51.21
58.17
29.00
38.63
60.
47.36
(MAG)"
25.07
28.72
52.49
57.72
60.01
47.54
Tians (MAL))
24.69
28.
35.74
64.97
39.44
51.97
56.58
28.21
38.14
57.32
46.55
400M params
15B
Transformer++
30.63
37.37
29.64
64.27
37.72
51.53
54.95
27.36
38.07
61.59
45.64
RetNet
29.92
46.83
29.16
36.97
51.85
56.01
27.55 55
37.30
59.66
45.47
HGRN2
32.33
47.14
26.12
64.52
35.45
52.24
55.97
25.51
37.35
59.02
44.52
GLA
27.96
36,66
27.86
65.94 94
37.41
49.56
56.01
26.36
38.94
59.84
45,24
Mamba
29.22
29.82
37.93
50.1
26.20
61.13
45.94
Mamba2
26.34
33.19
32.03
65.77
39.73
$2.48
59.00
27.64
37.92
60.72
46.91
DeltaNet
27.69
44.04
29.90
64.52
37.03
50.82
56.77
27.
38.22
60.09
45.57
TTT
26.11
31.52
33.25
65.70
39.11
$1.68
58.04
28.99
38.26
59.87
46.86
Gated DeliaNet
25.47
29.24
34.40
65.94
40.46
51.46
59.80
28.58
37.43
60..0
47.26
Samba"
25.32
29:47
36.86
66.09
39.24
$1.45
60.12
27.20
38. 68
58.22
47.23
Gated DeltaNet 112"
24.19
28.09
36.77
66.43
40.79
52.17
59.55
29.09
39.04
58.56
47.69
Titans (LMM)
25.03
28.99
35.21
40.91
52.19
59.97
29.20
60.85
47.83
Titans (MAC)"
25.61
27.73
36.92
66.39
41.18
$2,80
60.24
29.89 19.69
40.07
61.93
48.65
Titans (MAG)'
23.59
27.81
37.24
80
53.21
29.45
61,28
48.60
Tians (MAL.)
23.93
27.89
36.84
66.29
40.74
$2,26
59.85
29.71
38.92
58.40
47.87
760M params
30B tokens
Transformar++
25.
27.64
35.78
66.92
42.19
$1.95
60.38
32.46
39.51
60.37
48.69
RetNet
26.08
24.45
34.51
67.19
41.63
52,09
63.17
32.78
38.36
57.92
48.46
Mamba
28.12
23.96
32.80
66.04
39.15
$2.38
61.49
30.34
37.96
57.62
47.22
Mamba2
22.94
28.37
33.54
67.90
42.71
49.77
63.48
31.09
40.06
58. 15
48.34
DeltaNet
59.02
TTT
24.17
23.51
34.74
67.25
43.92
50.99
64.53
33.81
40.16
59.58
47.32
Gated DeltaNet
21.18
22.09
35.54
68.01
44.95
50.73
66,87
33.09
39.21
59.
49.69
Samba"
20.63
22.71
39.72
69.19
47.35
$2.01
66.92
33.20
38.98
61.24
$1.08
Gated DeliaNet-112
19.88
20.83
39.18
68.95
48.22
52.57
67.01
35.49
39.39
61.11
51.49
Titass (LMM)
20.04
21.96
37.40
69.25
48.46
52.27
66.31
35.84
40.13
62.76
51.56
Titans (MAC)
19.93
20.12
39.62
70.46
49.01
53,18
67.86
36.01
41.87
62.05
52.51
Titans (MAG)
18.61
48.94
52.89
40.35
52.50
Titans (MAL)
19.07
20.33
40.05
69.99
48.82
53.02
67.54
35.65
30.98
61.72
50,97
params
Transfomers++
18.53
18.32
42,60
70.02
50,23
$3.51
68.83
35.10
40.66
57.09
$2.25
RetNet
Mamba2
16.56
12.56
45.66
71.87
55.67
55.24
72.47
37.88
40.20
60.13
54.89
DeltaNet
17.71
16.88
42.46
70.72
50.93
53.35
68.47
35.66
40.22
55.29
52.14
Goted DeltaNet
16.42
12.17
46.65
72.25
55.76
$7.45
71.21
38.39
40.63
60.24
55.32
Samba'
16.13
13.29
44.94
70:94
53.42
55.56
68.81
36.
39.96
62.
54,00
DeltaNer-H2
57.77
71.33
39.07
61.55
56.18
Titans (LMM)
15.60
11.41
49.14
73.09
$6.31
$9.81
72.43
40.82
42.05
60.97
56.82
(MAC)
72.95
60.74
Titans (MA1.)
15.13
11.28
50.10
73.29
56.74
59.52
73.09
41.37
41.88
61.09
57.14
G.4 Efficiency
In this part, we compare the efficiency of our neural memory well as Titans with state-of-the-ar sequence models. The training throughput of models for different sequence length x batch size are reported in Figure 10. Comparing recurrent models, including our neural memory module, we can see our memory module is slightly slower than Mamba2 and Gated DeltaNet, mainly due to: (1) having deep memory and more expressive transition process (memory update), and (2) highly optimized kernel in the implementation of Mamba2. Interestingly, Titans (MAL.) are faster than baselines well as the memory module. The main reason for this better throughput is the highly optimized kernel of Flash-Attention [66], which used for implementing SWA and full attention module in Titans.
35

---

45
LMM =1x==1)
LMM =3)
Tokans/Second
40
LMM (h.t =
LMM (h.s 4)
35
30
10"
25
2000
4000
8D00
16000
32000
Sequence Length
Figure 8: The effect of memory depth on training throughput
GPL4
KMBETT
0wen2.5-220
Thaze OAACHTT
Uneaa RMG
Marite
Accuracy (%6
60
80
Accuracy
70
80
60
20
50
40
30
10
107
10
107
12A
10"
10
10
Sequence Length
Sequence Length
(a) Few-shot Setup
(b) Fine- Fine-Tuning Setup
Figure 9: Performance of Titans and baselines on BABILong benchmark. Titans (MAC) outperforms all baselines, including extremely large models, e.g., GPT4, We highlight the fact that ultra-large architectures (i.e., Llama, Qwen, GPT-4) are perform in zero-shot setting, while other small models (i.e.. Titans, Mamba, RMT) are fine-tuned. For this, we have followed the original setup in BABILong benchmark [53]
Table 7: Performance long-term forecasting, The best results are highlighted.
Neurra Memury
Stnta
ETranstrmer
PhiciTT
Creaftrmer
TimmsNet
Dlinsar
MAB
MSST
MSI
MAR
MSS:
0.357
3:400
0.513
0.406
ETTh1
0.421
3.447
0.445
0.323
0.450
FTTb3
8336
0.561
0.991
3/417
0.334
0.388
0.407
091
0681
0350
0.111
0.627
0.594
BC1.
0.362
01261
0.109
0.274
6.174
0.200
11318
01708
0336
0.200
0.344
10354
01251
16344
0.192
0.39
0.213
0/30)
Triffic
0.415
0.289
0.005
0.291
0.428
0.392
0.655
0.338
0.481
0.306
0.390
0.305
0.360
0.333
0.600
0.356
Bx22
0383
Weather
0.231
0.255
0.55
0.200
0.238
0
0.212
0.231
0.239
0.21
0229
03315
0.271
0.299
0.27
026
0317
DeltaNet
Titans (MAL)
Mambaa
Titans (MUN))
Mumbs
45
Transformere
1MM
Teless//eoon
335
30
25
2k
4K
8K
16K
Sequence Length
Figure 10: Training throughput comparison of Titans and baselines.
36

---

Table 8: Downstream evaluation of pre-trained DNA models GenomiesBenchmarks [136]. We
report top-1 classification accuracy (%).
Human Reg.
CNN
93.3
68.0
DNABERT
74.0
85.7
88.
85.6
75.1
GPT
70.5
$3.5
91.5
$7.7
73.0
HyenaDNN
74.2
89.2
93.8
96.6
80.9
Transformet++
73.4
89.5
89.9
91.4
79.5
Memory Module
89.3
96.6
79.9
Table 9: The performance our model compared to baselines.
SWDE
NQ
DROP
FDA
SQUAD
TQA
Transformers
84.9
23,0
28.4
72.5
48.1
64.4
Gated DeltaNet
63.2
19.1
26.7
33.4
39.6
59.7
Titans (LMM)
65.1
20.7
27.2
37.3
42.6
61.0
G.5 Time Series Forecasting
To show the effectiveness of our memory module broader tasks, we also evaluate its performance in time series forecasting tasks. To this end, use Simba framework [41] for time series forecasting, and replace its Mamba module with our neural memory. We report the results on common time series forecasting benchmark datasets-EI ETT, ECL, Traffic, and Weather [6]. The results reported in Table Our neural memory module outperforming all baselines, including Mamba-based, linear-based, and Transformer-based architectures.
G.6 DNA Modeling
In order understand the capability of Titans beyond natural language, we further evaluate the performance of our neural memory module DNA modeling tasks. To this end, evaluate pre-trained models on the downstream tasks GenomiesBenchmarks [136]. We follow the same experimental setups from Nguyen al. [137], and re-use the reported results of baselines by Arora et al. [70]. The performance of Titans (LMM) and baselines are reported Table 8. We find that LMM is competitive with state-of-the-art architectures across different downstream genomics tasks.
G.7 In-context Recall Tasks
In-context recall is one of the challenging benchmarks for recurrent neural networks. this section, we show that our Titan due its design of surprise performs favorably compared to other recurrent neural networks (note that Gated DeltaNet itself outperforms other modern recurrent neural networks), achieving closer performance to Transformers.
G.8 Synthetic Benchmark of MAD
We evaluate the performance of Titans (LMM) on MAD benchmark, synthetic benchmark that evaluate the performance of models in recall, memorization, compression, and copying tasks [138]. The results are reported in Table 10.
37

---

Table 10: Performance of Titan and baselines the synthetic benchmark of MAD [138]. Titan outperforms all the baselines, including Transformers.
Selective Copying
Compression
(Noisy) ICR
Fuzzy ICR
Memorization
Transformers
49.4
100
48.2
95.9
83.8
Gated DeltaNet
44.8
100
32.5
96.2
81.7
Titans
49.6
100
49.7
99.4
83.5
38


Reviews:
[{"review_id": "FNyPIzmxWH", "reviewer": "Reviewer_Uxn4", "summary": "This paper proposes the novel architecture for sequence models called Titans. Titans contain test-time trainable neural long-term memory, which allows the model to surpass other sequence models like transformers and RNNs, especially on tasks with longer sequence lengths. Authors describe the mathematical framework they use to train such models, provide a comprehensive study on the architecture design, and compare their approach with a wide range of baselines on several benchmarks like language modeling, reasoning, and long-range tasks.", "questions": "- How do we control the number of retrieved memories (N_l)? Our memory module maps hidden states of each token to some recalled value, so the number of memories equals the number of tokens. Do we cut the last few vectors (256 in this case)? How do we handle the case when the input is shorter than 1 chunk (which is 2k if I understood correctly?)\n- Please clarify your chunk-segmentation procedure. What is the size of the chunk? Is it equal to the size of the segment in MAC? Does your memory retrieve 256 tokens only for MAC, while for MAG and MAL you basically retrieve the number of vectors equal to the chunk size (= number of input tokens at the current step)?\n- Could authors add comparison of results of ARMT and Titans on BABILong and discuss key differences between those two methods?\n- Why don’t we just use sliding window attention in MAC just like in MAG? It is possible to keep the attention on the last tokens of the previous segment.\n\n\nTypos: \n- L94: date->data\n- L1111: though -> through", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- This work proposes a unique approach of thinking about memory as an explicit test-time learner of associations between keys and values. It offers an elegant general solution to the majority of sequence tasks by delegating some computation and training to the test time, making a model to learn more general patterns during training.\n- In this work, the authors comprehensively explore the capabilities of the proposed architecture on a diverse set of tasks.\n- Authors provide a reasonable ablation study, showing the importance of their architectural decisions.\n\nWeaknesses:\n- It’s unclear in MAC description: how we control the number of retrieved memories (N_l). Titans memory module maps hidden states of each token to some recalled value, so the number of memories equals the number of tokens. Maybe the authors mean that we cut the last few vectors (256 in this case), but it is not explicitly stated. It’s unclear how we handle the case when the input is shorter than 1 chunk (which is 2k).\n- The majority of used common-sense tasks are short (Table 1), and Titans model is basically full-attention + memory module (because sequence fits within one segment). So these tasks don’t cover sliding-window or segment-wise text processing, which is essential for long contexts. For wikitext, which requires more than 1 segment = chunk (4k tokens = 2 chunks?) MAG and MAL surpass the MAC architecture. I believe you need more comprehensive investigation on what architecture is best, especially on real-world long-context tasks (it’s unclear what long-context tasks are mentioned in Table 3).\n- Results provided on BABILong on figure 8, do not include results of approaches that perform better than Titans. For example, the paper authors refer, “Associative Recurrent Memory Transformer,” as well as the very BABILong paper, includes ARMT results on BABILong, which are higher than Titan's.", "was_revised": true, "final_justification": "The authors presented complete results for BABILong and an additional MAC+MAL variant. This increased transparency and addressed previous concerns about these results. The authors also provided the requested details during rebuttal and discussed key architectural differences between ARMT and Titans. Based on this, I've raised the quality to 4, and I hope to see updates in the camera-ready version.", "rebuttal": "**We thank the reviewer for their time and valuable comments.** We respond to your comments below:\n\n> It’s unclear in MAC description: how we control the number of retrieved memories (N_l).\n\n**Response**: Thank you for bringing this to our attention. As you mentioned, we fix the number of memory tokens in the context and consider the last N_l tokens. When input is shorter, we pad the sequence until it matches the length of the segments/chunks.\n\nFollowing your suggestion, we will add these details to the final version of the paper and will make sure to properly discuss it.\n\n> The majority of used common-sense tasks are short (Table 1), and Titans model is basically full-attention + memory module (because sequence fits within one segment)\n \n**Response:** Thank you for mentioning this. Please note that we use 512 tokens as our segment size. We will make sure to make this clear in the final version of the paper. Also, please note that even our attention-free model (Titans (LMM)) outperforms all the baselines including hybrid models. Therefore, we believe these results show the strengths of Titans memory module. While we also agree with the reviewer that better understanding of the advantages of each architecture and different methods that one can use to combine attention with memory modules requires further evaluations, we want to kindly bring to your consideration that this has not been the main focus of our submission. For our experimental setups, we aimed to follow the literature (including those with hybrid architectures) and so used the same common-sense reasoning and language modeling tasks that were previously used in similar studies. However, to fully address your concerns, if you believe it is necessary, we would be happy to evaluate the performance of Titans on any downstream tasks. \n\nThank you very much again for this valuable comment.\n\n\n>  Results provided on BABILong on figure 8\n\n**Response:** Thank you for bringing this to our attention. Please note that our main goal has been to compare sequence modeling backbones as the choice of different architectures can be orthogonal to this direction. That is, ARMT uses delta-rule to update its associative memory and so it can be replaced by our Titans (LMM) module, further enhancing the performance of ARMT. However, to fully address your concern, we provide the results of ARMT and compare it with Titans:\n\n| Model   | 2K | 4K | 8K | 16K | 32K | 64K | 128K  | 512K | 1M | \n| - | - | - | - | - | - | - | - | - | - | \n| Titans | 99 | 99  | 99 | 97  | 98  | 96  | 98  | 95  | 94 | \n| ARMT |  98 |  98 | 98  | 98  | 98  | 98  | 97  |  95  | 93 | \n\n\nOn average, Titans provide a better accuracy than ARMT. \n\n\n> Please clarify your chunk-segmentation procedure. \n\n**Response:** Please note that we use chunk and segment for two different concepts. Chunks are subsequence that we use to accelerate the training process of the memory and our chunk size is 16. In fact, for each chunk, we take the gradient with respect to the last state of the previous chunk. On the other hand, we use segment to refer to the larger subsequence that we perform attention on. We will make sure to make this clear in the final version of the paper. \n\nAlso, regarding the different architectures, yes, you are right. For MAG and MAL, we have the same number of outputs as the input size. While on MAC, we use a fix size memory tokens. We will make sure to discuss this in the final version of the paper.\n\n> Why don’t we just use sliding window attention in MAC just like in MAG? It is possible to keep the attention on the last tokens of the previous segment.\n\n**Response:** Thank you for mentioning this. That is totally possible and is a very interesting extension for the MAC architecture and a promising direction for future work. We will make sure to discuss it in the final paper. \n\n> Typos:   \n\n**Response:** Thank you very much for bringing this to our attention. We will make sure to proofread the paper and fix all typos. \n\n**We hope that our above responses have addressed your concerns, and we would be more than happy to answer any remaining concern.**"}, {"review_id": "BBhB8C1Eqb", "reviewer": "Reviewer_CPKN", "summary": "The authors introduce an interesting concept for test-time memory updates that can be thought of as a learnable “long-term” memory for transformers. Four variants of this are presented in the form of Context, Layer, Gate, and Solely Memory which balance tradeoffs between efficiency and performance on different context lengths. By explaining how to implement these methods in a highly parallelizable way the authors open the door to significant scaling and efficient use.", "questions": "Q1. Is there a sense from any of the tasks, either by probing or qualitative testing, what data actually gets stored in practice and what gets forgotten as a result of the surprise mechanism?\n\nQ2. Could the memory module be shared across modalities in combined multi-modal architectures? \n\nQ3. How sensitive is Titans’ performance to the size of its long-term memory? Are there additional ablation studies that vary the memory capacity to see whether gains saturate or degrade beyond a certain point or tradeoffs that could be made depending on the tasks?", "limitations": "Further discussion of the limitations of the approach could be included in the main body of the paper, for example w.r.t. scaling limits.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1. The authors demonstrate a promising new architecture to scale to extremely long context windows with linear-time inference. Evaluations including both the needle-in-a-haystack benchmarks and the language modeling and other tasks \n\nS2. Providing a way to update the memory weights while leaving the core model untouched is very useful. It is a strength of the method that GPU throughput can be maintained with the added benefit of the longer context window provided by the memory module.\n\nS3. The proposal of using the two surprise metrics (past and momentary) to generate the memory update seems useful, though it may be somewhat brittle if it requires tuning of task-specific hyperparameters.\n\nS4. The paper is very clear: it is helpful to provide the human memory lens to frame the work and then to represent the different architecture tradeoffs as implementation options (Context/Gate/Layer) with their various efficiencies. The callbacks to human memory throughout help ground the discussion.\n\nW1. There are some downsides to allowing the memory to be updated during inference-time: for example it might expose further security, safety, and alignment issues by giving an attacker access to a computable attack vector. While these don’t take away from the results presented it would be good to address them as potential limitations.\n\nW2. Currently the experiments scale up to 1.3B parameter models, which show promising results. Further experimentation would be needed to validate whether this will continue to scale or if other issues might arise at larger 8B+ scale.", "was_revised": true, "final_justification": "I believe the authors have addressed the reviewer concerns well, providing additional data where needed and clarifying points and modifications that need to be made to the final manuscript. While I believe there are some issues that are not fully addressed in the current work around potential risks, given the current scale of the experimentation I do not think they are pressing to introduce before acceptance and can be followed-up in future work.", "rebuttal": "**We thank the reviewer for their time and valuable comments.** We respond to your comments below:\n\n> There are some downsides to allowing the memory to be updated during inference-time: for example it might expose further security, safety, and alignment issues by giving an attacker access to a computable attack vector. While these don’t take away from the results presented it would be good to address them as potential limitations.\n\n**Response**: Thank you very much for mentioning this. We understand the importance of potential safety and alignment risks and it would be a very important direction to understand such risks in the future studies. Following your suggestion, we will make sure to discuss this as the potential limitations and advocate for further study in the future in this direction.  \n\n\n> Currently the experiments scale up to 1.3B parameter models, which show promising results. Further experimentation would be needed to validate whether this will continue to scale or if other issues might arise at larger 8B+ scale.\n\n**Response**: Thank you for your comment. We understand the importance of scaling to larger scales. However, please note that training large models is extremely costly and challenging. Therefore, most academic research papers (such as Gated DeltaNet (ICLR 2025), Longhorn (NeurIPS 2024), TTT (ICML 2025), etc.) in this area focus on similar size models as ours. We followed the literature and reported the results on these scales. \n\n\n\n> Is there a sense from any of the tasks, either by probing or qualitative testing, what data actually gets stored in practice and what gets forgotten as a result of the surprise mechanism?\n\n**Response**: Please note that as an illustrative example, we have already provided Figure 10 in Appendix that shows the surprise value of different tokens in a specific prompt from the FineWeb dataset. In this example, we can see that: the surprise metric is effective to capture important tokens that are needed for the context. Words such as “Attention!” are getting higher scores, while words such as “Currently,” “Only”, etc., are mainly ignored and forgotten. \n\n\n> Could the memory module be shared across modalities in combined multi-modal architectures?\n\n**Response**: Thank you for asking this very interesting question. In this work our focus has been on the language modeling tasks and so there has been only a single modality. However, it would be a very interesting future direction to design multi-modal Titans. In theory, the memory can be shared across different modalities as the memory only memorizes tokens. If different modalities are already tokenized, then the memory module should be able to learn what tokens are more important from data (and so can be shared across different modalities).  \n\n\n\n> How sensitive is Titans’ performance to the size of its long-term memory? Are there additional ablation studies that vary the memory capacity to see whether gains saturate or degrade beyond a certain point or tradeoffs that could be made depending on the tasks?\n\n**Response**: Please note that in all variants of MAC, MAG and MAL the long-term memory size is tied to the number of parameters of the model (given the key and value features, d, the memory size is d^2). Therefore, increasing or decreasing the memory size will result in #parameter change in the entire architecture. However, in Figure 6, we have ablated the performance of the memory module on different sequence lengths and memory sizes (proportional to memory size). Based on the results across model sizes, we can see that increasing the capacity of the memory enhances its effectiveness for longer context length as the memory has more parameters to store the context in. On the other hand, in each size of memory, we can see that deeper memory modules perform better at longer context length, which further validate the importance of deeper memory. Furthermore, as you might have noticed, for each model size (please see Figure 6 (a) and (b)), increasing the context beyond a point can damage the performance. The reason for this is that memory does not have the capacity to store and compress all necessary tokens. Again, we can see that deeper memories are more robust to this pattern.   \n\n**We hope that our above responses have addressed your concerns, and we would be more than happy to answer any remaining concern.**"}, {"review_id": "kK7P0EjL0D", "reviewer": "Reviewer_ajQb", "summary": "The paper introduces Titans, a new family of neural architectures that incorporate a long-term memory module capable of learning to memorize at test time. The core idea is to augment attention mechanisms - which effectively model short-range dependencies but are limited by fixed context lengths - with a trainable neural memory system that resembles human long-term memory. This memory is updated using a \"surprise\" signal based on gradients during inference and includes a gating mechanism for adaptive forgetting. Three Titans variants are proposed to integrate memory in different ways: as additional context (MAC), through gating (MAG), or as a sequential layer (MAL). These models show improved results across tasks such as language modeling, commonsense reasoning, and long-context benchmarks. The architecture supports parallelizable training and achieves better performance and scalability than several recent recurrent and hybrid Transformer-based models. Titans demonstrate that dynamically updating neural memory at test time can enhance long-range sequence modeling.", "questions": "1. Can you clarify how the surprise-based memory update behaves over long sequences? It's unclear how the model avoids drift or forgetting. Some visualization or stability analysis would help.\n\n2. Why use a deep MLP for memory? Have you compared against shallower versions, recurrent cells, or Hopfield-style mechanisms? Ablations would make this design choice more convincing.\n\n3. The paper's relation to test-time training and meta-learning models like TTT or fast weight programmers is underdeveloped. What is the core difference beyond architecture?\n\n4. The role of persistent memory is vague. How is it trained, and how sensitive is the model to its size? More detail would clarify whether it's essential or just another form of prefix tuning.", "limitations": "No. The paper does not adequately discuss limitations or societal impacts. The authors should address possible risks related to test-time adaptation, such as instability, data leakage, or unintended memorization of sensitive information. It would also help to discuss potential computational overhead at inference due to memory updates, and any fairness or privacy concerns when memory is updated dynamically from user data.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: The paper presents a technically solid and well-executed approach with a clear motivation for combining attention with a test-time trainable long-term memory. The proposed memory module is grounded in both neuroscience-inspired ideas and practical algorithmic design, and is evaluated across a diverse set of benchmarks. The experiments are thorough, with meaningful ablations and comparisons to relevant baselines. However, some architectural choices, such as the specific structure of the memory MLP or hyperparameter tuning strategies, could have been better justified or explored. While the surprise-based update is interesting, it lacks deeper theoretical analysis or guarantees on memory stability or convergence.\n\nClarity: The paper is generally well written and structured, but at times suffers from dense exposition and terminology that could challenge readers unfamiliar with meta-learning or memory systems. While figures are helpful, some architectural descriptions are overly abstract or spread across too many sections. The treatment of related work is extensive but might benefit from a clearer contrast with specific prior methods, especially regarding what exactly distinguishes Titans from prior hybrid models. More intuitive explanation of the gating mechanisms or test-time dynamics would improve accessibility.\n\nSignificance: The work addresses a central challenge in sequence modeling - how to scale models to long contexts without compromising performance or efficiency. The results demonstrate that Titans outperform strong baselines across tasks, including those focused on long-range reasoning. This suggests potential for practical impact and further adoption. That said, the gains over recent hybrid models like Gated DeltaNet or Samba are sometimes modest, and it remains to be seen how Titans generalize beyond the tasks shown. The design space for combining memory and attention is still broad, and the paper only explores three specific configurations.\n\nOriginality: The idea of learning memory at test time via surprise-driven updates is novel in the context of modern sequence models, and the framing of memory as a dynamic, test-time learning module is fresh. While components like gating, persistent memory, and associative loss are not individually new, their combination and positioning within a scalable architecture are meaningfully different. Still, the paper could more explicitly position itself relative to test-time training literature and clarify how it advances those ideas in the long-context modeling setting.", "was_revised": true, "final_justification": "I found the author's response convincing and have made a small upward adjustment to my evaluation.", "rebuttal": "**We thank the reviewer for their time and valuable comments**. We respond to your comments below:\n\n> However, some architectural choices, such as the specific structure of the memory MLP or hyperparameter tuning strategies, could have been better justified or explored.\n> Why use a deep MLP for memory? Have you compared against shallower versions, recurrent cells, or Hopfield-style mechanisms? Ablations would make this design choice more convincing.\n\n**Response:** Thank you for mentioning this. However, please note that we already have ablations on shallow models as the memory. In Table 3 (5th row), we replace our deep memory with a linear shallow memory and the results show the effectiveness of the deep memory module. Furthermore, in Figure 6, we ablate the effect of the depth of the memory (including shallow version) on the performance of the model in different model sizes and sequence lengths. While it is interesting, exploring other architectures as the memory is out of the scope of our paper and we leave it for future studies. \n\nAlso, we would be happy to clarify any unclear point about the hyperparameters. We do not extensively tune any hyerperparameter and there are only two extra hyperparameters that we use: (1) chunk size: which is 16. The smaller the chunk is the better performance we can get but with the cost of slower model. (2) segment size: in which, we follow previous studies and use 2048 as sliding window (when exists) and 512 as segment size. We however, would be happy to clarify any specific part that has not been clear. \n\n\n> While the surprise-based update is interesting, it lacks deeper theoretical analysis or guarantees on memory stability or convergence.\n\n**Response:** Please note that while we understand the reviewer's interests and curiosity about the theoretical side of memory modules, we want to clarify that none of the peer studies provide such theoretical results about the capacity of the memory or why one-shot memorization works (e.g., GLA (ICML 2023), Mamba2 (ICML 2024), Mamba (COLM 2024, best paper award), Gated DeltaNet (ICLR 2025), GSA (NeurIPS 2024), Simba (NeurIPS 2024), Longhorn (NeurIPS 2024), TTT (ICML 2025, spotlight), etc.). These are indeed interesting questions that one can ask and explore in future studies. Our submission is a 9-page research paper that aims to provide **enough** evidence for its usefulness and effect. Even the initial Transformers paper, after 8 years, is still being studied, improved, and evaluated. Therefore, we believe there are indeed many interesting questions that one can study in future work.       \n\n\n\n> The treatment of related work is extensive but might benefit from a clearer contrast with specific prior methods, especially regarding what exactly distinguishes Titans from prior hybrid models.\n\n**Response:** We kindly refer the reviewer to Appendix F, in which we directly compare our architectural design with most close studies, including test time training models. Also, for hybrid models, we dedicate the entire Appendix B2 to this. Similarly, we extensively discuss all TTT and fast-weight programs study in a separate sub-section (Appendix B3). However, we would be happy to provide more details if there is a specific part that the reviewer believes require more discussion.\n\n\n> The design space for combining memory and attention is still broad, and the paper only explores three specific configurations.\n\n**Response:** We agree with the reviewer that there are a broad range of ideas that one use to combine memory and attention in different configurations. Please note that this is a 9-page research paper, and we cannot discuss all possible cases. We hope that our work motivates future studies to further study the combination of memory modules and attention. \n\n\n> Can you clarify how the surprise-based memory update behaves over long sequences? It's unclear how the model avoids drift or forgetting. Some visualization or stability analysis would help.\n\n**Response:** Please note that defiantly forgetting and drift is possible over long sequences as the memory has a fixed size. This has always been a limitations for all recurrent models. (In our RULER benchmark you can see that with increasing the sequence length, the performance drops). However, in several experiments, we show that our memory module is more robust to forgetting and drift (Please see RULER results). \n\n\n\n> The role of persistent memory is vague. How is it trained, and how sensitive is the model to its size? More detail would clarify whether it's essential or just another form of prefix tuning.\n\n**Response:** Please note that persistent memory are just learnable tokens (which can also be seen as a form of prefix tuning). Such tokens are proven to be effective from different perspectives, which we have discussed in Appendix C. Looking at our ablation study in Table 3, we can see that while persistent memories are improving the performance of the model, the performance gain is not very significant and so removing them does not affect the performance of the Titans that much. Therefore, it is fair to say that Titans' performance does not rely on them and so it is not sensitive to it at all. We would be happy to discuss this further in the final version of the paper.\n\n\n\n**We hope that our above responses have addressed your concerns, and we would be more than happy to answer any remaining concern.**"}, {"review_id": "6LMb8c3FWS", "reviewer": "Reviewer_WL32", "summary": "I previously reviewed this paper for ICML 2025, and this review takes into account both the original submission and how the authors addressed reviewer concerns. The paper introduces Titans, a family of architectures incorporating a neural long-term memory module that learns to memorise at test time through gradient-based updates. The key innovation is a surprise-based memory mechanism that combines \"momentary surprise\" (gradient updates) with \"past surprise\" (momentum) to determine what information to store. Unlike traditional recurrent models that compress everything into fixed-size states, Titans uses a deep MLP as memory that can adaptively learn and forget information. The authors present three architectural variants that incorporate this memory differently: Memory as Context (MAC) treats memory as additional context for attention, Memory as Gate (MAG) combines memory with sliding-window attention through gating, and Memory as Layer (MAL) uses memory and attention sequentially. The paper demonstrates improvements over baseline models on language modelling, common-sense reasoning, and needle-in-a-haystack tasks, with the authors claiming effectiveness on long contexts.\n\nThe authors have made significant efforts to address reviewer feedback, including adding comparisons with hybrid models like Samba, providing more architectural details, expanding the related work section, and adding state-tracking evaluations. However, many critical concerns remain unaddressed as discussed below.", "questions": "Since I have many things to say for this paper, I tried to group my most important questions into the following 4 categories about theoretical issues, fair evaluation, positioning and reproducibility:\n\n1. **Theoretical foundations**: Could you provide the theoretical analysis of memory capacity and convergence that was promised in the ICML rebuttal? Additionally, the analysis of 2-3 updates improving performance by 0.5% that you promised to include is missing from the current version - could you add it?\n2. **Fair comparative evaluation**: Can you provide (a) BABILong results where all models are either fine-tuned or zero-shot, (b) a unified evaluation where all methods appear on the same benchmarks, and (c) throughput comparisons including Samba in Figure 9? Finally, critical comparisons with key memory baselines are currently relegated to the appendix. To allow for a fair assessment, it is important that competitive results are presented and discussed in the main text. Can you please provide a unified summary table in your rebuttal, and explain how you will restructure the main paper to foreground these essential results in a final version?\n3. **Positioning and advantages**: Given that existing surprise-based approaches achieve 100% accuracy at 10M tokens without fine-tuning, what are the specific advantages and use cases for Titans? Why does memory-constrained Titans outperform full Transformers on retrieval tasks? \n4. **Reproducibility**: Can you provide clear implementation guidance, including memory size selection criteria, hyperparameter settings, and details about the \"highly optimised kernel\" mentioned for MAL?", "limitations": "The authors briefly mention limitations in the appendices but don't adequately address:\n\n- The gap between claimed capabilities (\"larger context windows\") and actual evaluation (16K tokens)\n- Why their approach requires fine-tuning while other surprise-based methods work zero-shot\n- Computational overhead of test-time gradient updates\n- When their complex architecture is justified over simpler alternatives\n\nThe paper would benefit from a dedicated limitations section in the main text.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "#### Strengths:\n- *Novelty*: Although this work builds on top of a lot of other studies on state-space models as Transformer alternatives, it has some interesting novelties. I found treating these models just as a neural network with gradient updates and momentum very interesting. It is an intuitive connection but no other model I am aware of makes it. Also, the three variants of Titans (MAC, MAG, MAL) provide interesting ways to incorporate memory into sequence models, with MAC showing particularly strong results.\n- *Evaluation*: The paper evaluates across multiple domains (language, time series, DNA) showing versatility.\n- *Parallelisation*: The chunk-wise training method (Section 2.2) enables practical training despite gradient-based updates, which is crucial for scaling up these models.\n- *Performance*: Titans show consistent gains over baselines in multiple benchmarks.\n- *ICML feedback*: Authors added Samba comparisons and some other hybrid models in accuracy benchmarks, state-tracking experiments (MAD benchmark), memory interpretability analysis, and expanded related work as requested.\n\n#### Weaknesses:\n* Overall quality:\n\t* Despite promising formal proofs in the ICML rebuttal, the revised paper lacks the theoretical analysis of memory capacity and convergence properties.\n\t* Some important model comparisons are incomplete. For example, the throughput comparison with Samba is still missing from figures despite specific reviewer requests in ICML.\n\t* BABILong evaluation remains flawed, comparing fine-tuned small models against zero-shot large models without acknowledging this unfair advantage\n\n* Clarity:\n\t* \"Momentary surprise\" is presented as novel when it's identical to the well-established delta rule. I find this misleading as it stands.\n\t* Different methods are evaluated on different benchmarks without clear justification. Why are some baselines missing from certain benchmarks (e.g. only RULER and not BABILong)? This makes it difficult to form a coherent picture of relative performance.\n\t* On a related point, in retrieval tasks, methods with no memory constraints (full Transformers, RAG-based or retrieval-based approaches) somehow perform worse than Titans which has explicit memory limitations, despite the fact that in the literature are solving much longer NIAH tasks with 100% accuracy. This surprising result deserves some discussion. Is it due to training differences, evaluation setup, or fundamental advantages of the current approach?\n\t* While the connection to momentum is mentioned, its framing could be clearer in the main text. The detailed clarification in Appendix F is welcome, but the main narrative would benefit from being more upfront about this connection to a standard optimisation technique\n\t* There is no clear guidance on hyperparameter selection, memory size choices, or when to use which variant of Titans.\n\n* Significance of claims:\n\t* Despite claims about \"larger context windows,\" rigorous evaluation (RULER) only extends to 16K tokens.\n\t* The paper doesn't clearly establish when/why one would use Titans over simpler alternatives or more successful approaches.\n\t* \"Past surprise\" is standard momentum; the main contribution appears to be combining existing techniques\n\t* The paper doesn't adequately distinguish itself from recent work using surprise for memory in LLMs (e.g. EM-LLM).\n\n\n**Overall,** the paper presents some very interesting architectural ideas and shows empirical improvements. The authors made good-faith efforts to address ICML feedback. However, the presentation remains problematic with key details missing or relegated to appendices without explanation. The revisions give the impression of a checklist being completed rather than a deep re-evaluation of the paper's weaknesses. Critical comparisons lack proper methodology, reproducibility information is scattered, and the theoretical foundations are absent despite being promised. While the core ideas merit publication, the paper needs substantial revision to meet publication standards. My '*Borderline Accept*' score reflects this tension: the core ideas have high potential, but the unaddressed flaws are significant. I believe these issues are addressable, and I look forward to the authors' rebuttal, after which I will re-evaluate my final assessment.", "was_revised": true, "final_justification": "The authors have constructively engaged with the review and committed to addressing my major concerns: adding a theoretical limitations discussion, clarifying the relationship to standard optimisation techniques, moving key long-context results to the main paper, and improving figure labelling for methodological clarity. These changes will significantly strengthen the paper's presentation and contribution. I appreciate the authors' responsiveness and look forward to seeing these improvements in the final version.", "rebuttal": "**We thank the reviewer for their time and valuable comments**. We respond to your comments below:\n\n> Despite promising formal proofs in the ICML rebuttal, the revised paper lacks the theoretical analysis of memory capacity and convergence properties.\n\n**Response**: We first want to kindly bring to your consideration that providing theoretical results is not a **must** for publications. While we understand the reviewer's interests and curiosity about the theoretical side of memory modules, we want to clarify that none of the peer studies provide such theoretical results about the capacity of the memory or why one-shot memorization works (e.g., GLA (ICML 2023), Mamba2 (ICML 2024), Mamba (COLM 2024, best paper award), Gated DeltaNet (ICLR 2025), GSA (NeurIPS 2024), Simba (NeurIPS 2024), Longhorn (NeurIPS 2024), TTT (ICML 2025, spotlight), etc.). These are indeed interesting questions that one can ask and explore in future studies. Our submission is a 9-page research paper that aims to provide **enough** evidence for its usefulness and effect. Even the initial Transformers paper, after 8 years, is still being studied, improved, and evaluated.       \n\nIn ICML you mentioned you are willing to increase your initial rating of “weak accept”, if we provide theoretical results. **Despite providing theoretical results and addressing your concerns, your initial rating remained the same**. Therefore, it was not clear for us if our theoretical results are what you expected. In this submission, we decided to make our work more focused, and so leave such exploration for future work, mainly due to the fact that **the goal of this work is not on one-shot memory update nor understanding the capacity of recurrent models**. \n\n\n> Some important model comparisons are incomplete. For example, the throughput comparison with Samba is still missing from figures despite specific reviewer requests in ICML.\n\n**Response**: Please note that as we discussed earlier in the ICML review process, Samba is Attention + Mamba. We already have a throughput comparison with Mamba, which indicates that our Titans (LMM) is faster. Therefore, we can conclude that Samba = Attention + Mamba is slower than our MAL = Attention + LMM. However, following your suggestion, here we provide the Samba throughput and compare it with our models: \n\n| Model | 2K | 4K | 8K | 16K |  \n|-| -| -| - |-|  \n|Samba|\t~39|\t~38|\t~36|\t~35|    \n|MAG| ~42|~41|~41|~40|  \n|MAL |~43|~43|~41|\t~41|\n\n\nWe will add these results in the camera ready version of the paper. \n\n\n> BABILong evaluation remains flawed, comparing fine-tuned small models against zero-shot large models without acknowledging this unfair advantage.\n\n**Response**: First, in lines 1374 - 1383 we clearly acknowledged and explained that small models are fine-tuned and large (commercial) models are zero-shot. Therefore, saying ``without acknowledging’’ is a misunderstanding. Second, we have followed the benchmark setups (please see Figure 1 in **Peer-reviewed** BABILong original paper in NeurIPS 2024). Therefore, we believe if there is a setup in this peer-reviewed paper that the reviewer found flawed, we should not be penalized for that. Please note that it is not expected nor we have the budget to re-run all the benchmark results for all the baselines and change the **original** setup of the benchmark. If the reviewer believes that it can solve their concern, we can remove the results of large (commercial) models, which definitely are not expected to compare with.\n\nMoreover, in Table 10, we have provided the results for even 128K sequence length in RULER, which we believe should be enough to show the good performance of Titans in longer contexts.   \n\n\n> \"Momentary surprise\" is presented as novel when it's identical to the well-established delta rule. I find this misleading as it stands.\n\n**Response**: Could you please **specify** the part that we claim momentary surprise as a novel contribution of our work so we can fix it? \n\nIn line 1168, we have clearly mentioned that delta-rule is equivalent to momentary surprise and this is the difference of our update rule, because we consider ``past surprise`` as well. Again in line 1328 we clearly mentioned that `The Delta Rule is based on momentary surprise`. \n\n\n> Different methods are evaluated on different benchmarks without clear justification. Why are some baselines missing from certain benchmarks (e.g. only RULER and not BABILong)? This makes it difficult to form a coherent picture of relative performance.\n\n**Response**: Please note that the **only** benchmark with different baselines is the BABILong benchmark. The state-of-the-art recurrent models are considered as the baseline for all language modeling, common-sense reasoning, and RULER tasks. The BABILong benchmark, however, has its own baselines and it has not been considered in any previous similar studies so we can borrow their results. Therefore, it is hard to adapt such models for this downstream task, is out of the scope of our paper, and further requires more compute resources. Once again, please note that we have already shown the good performance of our model until 128K (which has not been evaluated in any similar previous studies on recurrent and linear models). Therefore, if you believe removing BABILong benchmark results can make the evaluation more clear, we would be happy to do it in the camera ready version of the paper.    \n\n\n> Despite claims about \"larger context windows,\" rigorous evaluation (RULER) only extends to 16K tokens.\n\n**Response:** This seems to be a misunderstanding. In Table 10, we have already provided the results for the RULER benchmark until 128K sequence length. \n\nPlease note that this is a research paper and more than these sequence lengths are often a goal for commercial LLMs, or research papers that are built on large pre-trained models. Furthermore, our goal is to close the gap between the effective context length of recurrent models and they already perform near zero accuracy in 16K context length of RULER. \n\n> \"Past surprise\" is standard momentum; the main contribution appears to be combining existing techniques\n\n**Response:** We respectfully disagree with the reviewer that this connection affect the novelty of our method. Momentum term has never been used in this context. With the same logic, one can argue that most of recent architecture studies, including the work of DeltaNet (Schlag et al., 2021) and linear attention (Katharopoulos et al., 2020), are not novel because all use gradient descent, which is a well-known algorithm. \n\n> Additionally, the analysis of 2-3 updates improving performance by 0.5% that you promised to include is missing from the current version - could you add it?\n\n**Response:** Yes, we will make sure to add these results in the final version of the paper. \n\n> Fair comparative evaluation: Can you provide (a) BABILong results where all models are either fine-tuned or zero-shot\n\n**Response:** Please note that all small-models are fine-tuned. If the reviewer believes that the original setup of the peer-reviewed benchmark is not proper, we would be happy to remove the results of zero-shot models. Please once again note that all recent studies in this line has focused on RULER, and we have already provided its results until 128K.\n\n\n> Why does memory-constrained Titans outperform full Transformers on retrieval tasks?\n\n**Response:** Please note that, the unbounded memory in Transformers is not always the best for all the tasks. The reason is data can be compressible and so the task does not require direct retrieval. Also, it is notable that unbounded memory in Transformers does not mean that they can utilize it in all tasks. As an example, several papers discuss limitations like ''attention sink’’, which can cause Transformers to not scale to a large context. Also, in Table 8, we have provided the results for retrieval tasks that Transformers achieve better results. Please note that this is one the studies in this direction and requires several follow up works to fully address all the aspects. \n\nMoreover, all Transformer and retrieval-based models consider the dependencies of tokens as linear mapping between keys and values. Even given direct access to all past tokens (having perfect memory) the KV cache directly maps keys to values. This might not always the perfect assumption. On the other hand, Titans uses a deep memory and so can learn non-linear mapping between keys and values.  \n\n\n> There is no clear guidance on hyperparameter selection, memory size choices, or when to use which variant of Titans.\n\n**Response:** Please note that memory size is tied to the number of parameters. Given $d$ as the dimension of keys and values, the memory has size $L \\times d^2$, where $L$ is the depth of the memory.  Therefore, it is not a new hyperparameter, and can be determined based on the computational resources that are available. Also, in our ablation study, we show that for long-context tasks MAC performs better, while for language modeling MAG achieves the best results. We will emphasize more on these in the final version of the paper. Please let us know if there are specific parts that are not clear. \n\n> Details \"highly optimised kernel\" mentioned for MAL?\n\n**Response:** Please note that as we mentioned in the paper, this is Flash attention kernel and it is not our contribution. The implementation of attention is based on the highly-optimized kernel of Flash attention.\n\n> Limitations:\n\n**Response:** Following your suggestion, we would be happy to discuss them as the limitations and motivated future work to address them.\n\n> Details of Implementation:\n\n**Response:** Following your suggestion, we adhere to provide all the details upon acceptance of the paper.\n\n**We hope that our above responses have addressed your concerns, and we would be more than happy to answer any remaining concern.**"}]

Author rebuttal:

Meta-review: The paper at hand proposes a new intriguing perspective on recurrent memory and new connections between RNNs and attention. All reviewers agree that the paper deserves to be accepted, though some concerns were raised regarding additional experimental results and potential directions to improve clarity.

I suggest acceptance, though I recommend that the authors work towards improving clarity in their work, particularly in terms of transitioning from "the idea" to "the implementation".
Decision: Accept (poster)