MoM: Linear Sequence Modeling with Mixture-of-Memories
Authors: Jusen Du
Weigao Sun
Disen Lan
Jiaxi Hu
Tao Zhang
Yu Cheng
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient/Low-Resource Methods for NLP
Linear Sequence Modeling
Machine Learning for NLP
Primary area: foundation or frontier models, including LLMs

Abstract: Linear sequence modeling methods, such as linear attention, state space modeling, and linear RNNs, offer significant efficiency improvements by reducing the complexity of training and inference. However, these methods typically compress the entire input sequence into a single fixed-size memory state, which leads to suboptimal performance on recall-intensive tasks. To address this limitation, we introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes multiple independent memory states, with a router network directing input tokens to specific memory states. This approach greatly enhances the overall memory capacity while minimizing memory interference. MoM serves as a general framework that can be seamlessly combined with diverse memory update mechanisms across linear models. As a result, MoM performs exceptionally well on recall-intensive tasks, surpassing existing linear sequence modeling techniques. Despite incorporating multiple memory states, the computation of each memory state remains linear in complexity, allowing MoM to retain the linear-complexity advantage during training, while constant-complexity during inference. Our experimental results show that MoM outperforms current linear sequence models on downstream language tasks, particularly recall-intensive tasks, and even achieves performance comparable to Transformer models.

Published as a conference paper at ICLR 2026
MoM: LINEAR SEQUENCE MODELING WITH MIXTURE-OF-MEMORIE
Jusen Du Weigao Sun Disen Lan Jiaxi Hu', Tao Zhang", Yu Cheng
Tsinggua University, "Shanghai Al Laboratory. Fuudnn University, The Hong Kong University of Science and Technology (Guangzhou), The Chinese University of Hong Kong
ABSTRACT
Linear sequence modeling methods, such linear attention, state space model- ing, and linear RNNs, offer significant efficiency improvements by reducing the complexity of training and inference. However, these methods typically compress the entire input sequence into a single fixed-size memory state, which leads to suboptimal performance on recall-intensive tasks. To address this limitation, we introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes multiple independent memory states, with a router network directing input to- kens to specific memory states. This approach greatly enhances the overall mem- ory capacity while minimizing memory interference. MoM serves as general framework that can be seamlessly combined with diverse memory update mech- anisms across linear models. As a result, MoM performs exceptionally well on recall-intensive tasks, surpassing existing linear sequence modeling techniques. Despite incorporating multiple memory states, the computation of each memory state remains linear in complexity, allowing MoM to retain the linear-complexity advantage during training. while constant-cone complextyy during inference. Our ex- perimental results show that MoM outperforms current linear sequence models downstream language tasks, particularly recall-intensive tasks, and even achieves performance comparable to Transformer models.
INTRODUCTION
Attention mechanisms have made significant contributions to the field of artificial intelligence, ad- vancing various modalities such language, vision, audio, video, graphs, and even time series (Achiam et al., 2023; Team, 2023). The Transformer (Vaswani, 2017), known for its ability to capture long-range dependencies, has become foundational architecture in this space. However, traditional Transformers encounter computational challenges due to their quadratic time complex- ity, O(?²), with respect sequence length n, making it difficult to scale to long sequences. To overcome this limitation, several linear sequence modeling methods have been proposed, including linear attention (Katharopoulos et al., 2020; Qin al., 2023a; Li al., 2025), state space modeling (Gu & Dao, 2024; Dao & Gu, 2024), and linear RNNs (Peng et al., 2024; Qin et al., 2024d), which offer (n) training complexity and (1) inference complexity. These approaches often reduce the input sequence to a fixed-size hidden space, collapsing the information into single "memory state". While these methods enhance efficiency, they face two main challenges: limited memory capacity and memory interference. When new information overwrites the single fixed-size memory state, previously stored representations may degrade, which negatively impacts its long-term memory per- formance on recall-intensive tasks.
We argue that the strong performance of Transformer models on recall-intensive tasks arises from their ability to avoid memory interference by maintaining independent key-value caches for each token, thus offering virtually unlimited memory capacity. In contrast, linear sequence modeling relies extreme compression, consolidating all the input information into single fixed-size mem- ory state (Katharopoulos et al., 2020; Dao & Gu, 2024). This approach results in limited memory capacity and inherentl leads to memory interference issues.
"nterss Shanghai AT Laboratory; CCorresponding Authors; *Project Lead.

---

Published as a conference paper at ICLR 2026
Interestingly, the human brain has developed mechanisms that enable large memory capacity while reducing memory interference. Neuroscience studies show that in the hippocampus, theta oscil- lations (4~8 Hz) and gamma oscillations (30~100 Hz) work together to support neural coding mechanism for multi-item memory (Buzsáki, 2002; Lisman & Jensen, 2013). Specifically, each theta cycle is subdivided into multiple gamma subcycles, and within each gamma subeycle, dis- tinct group of neurons is activated following the "E%-max" mechanism (de Almeida et al., 2009). This sequential activation temporally separates different memory items, thus preventing interference.
Inspired by these biological insights, we propose a new architecture called Mixture-of-Memorrie (MoM), which aims strike a balance between the explicit token representations Transform- ers and the extreme compression found in earlier linear sequence modeling methods. MoM employs multiple independent memory states, with router network that directs input tokens specific mem- ory states. The input sequence is divided into predefined number of subsequences (phase-specific neural assemblies), which are processed parallel and fed into the corresponding memory projec- tions (dentate microcircuits) generate key-value pairs. As the linear sequence modeling layer pro- cesses each subsequence using an RNN-like update mechanism, it produces multiple memory states that capture different aspects of the input sequence. The final output computed as a weighted sum of these memories, which we refer to as the fixture-of-memorrt capacity and eliminates memory interference, enabling MoM significantly outperform existing linear sequence models that rely on single fixed-size memory state.
This approach expands memory
Our contributions can be summarized as follows:
We present MoM, an architecture that incorporates multiple independent memory states, significantly enhancing memory capacity and eliminating memory interference, while taining the efficiency benefits of linear-time training and constant-memory inference.
Distinct with existing gating mechanisms, MoM is a new paradigm reduce memory interference by separating the memory states. The overall design is broadly compatible with diverse linear sequence modeling methods, making it straightforward and effective approach to boost task performance.
Through empirical evaluation, we show that MoM outperforms strong linear sequence modeling baselines across variety of language tasks, particularly recall-intensive tasks. MoM even achieves performance on par with Transformer models, a feat that current linear sequence modeling methods struggle to match.
PRELIMINARY
For notations in this work, we use bold lower-case letters for row vectors (e.g., It, kt), bold upper- case letters for matrices (e.g.. Q, K) and the identical letters represent row in the matrix, c.g., q: the f-th row of Q.
LINEAR ATTENTION
To reduce the time complexity of Transformer attention, various optimization techniques have been proposed. Linear Transformers (Katharopoulos al., 2020) replace the softmax attention mecha- nism with dot-product of feature maps Φ(-):
E Σn (o
o,
where qı, kr, U Rd. While the presence of the denominator may lead numerical instability (Qin et al., 2024b) and the feature map can utilize an identity function, which we omit for simplicity. In perspective of memory, the formulation also be written in recurrent format:
M, = M, - k. vc. o = Mr
(2)
This indicates that linear attention can function as a linear recurrent layer with matrix-valued hidden state M which refer to memory sate and the output is generated by querying the memory state M. This represents the ultimate compression of sequence information, condensing the entire sequence into single memory state.
2

---

Published as a conference paper at ICLR 2026
Building the foundational concepts of linear attention and memory perspective, recent advancements have focused on optimizing memory structure, including gated updates (Yang et al,, 2023; Qin et al., 2024e;d) and memory capacity expansion (Peng et al., 2024; Qin et al., 2024d).
METHOD
3.1 MOTIVATION
Linear sequence models compress the entire sequence data into a fixed-size memory state. Despite numerous efforts to minimize information loss, such introducing gating mechanisms and employ- ing more precise control over memory modifications (Orvieto et al., 2023; De et al., 2024; Beck al., 2024; Yang et al., 2023; Zhang al., 2024), some degradation in this compression process is inevitable. Expanding the memory capacity has been shown to mitigate this issue to some extent, with studies indicating that increasing memory capacity can enhance model performance (Qin al., 2024d; Peng et al., 2024).
However, previous approaches that simply increased the size of the RNN state, essentially expand- ing single memory state, struggled to capture the full spectrum of information within an entire sequence. We propose that this difficulty arises because sequence information is often multifaceted, and single, expanded memory may not be capable of simultaneously capturing multiple aspects of the data. Inputs that introduce new or orthogonal information may interfere with existing memory content when using shared memory. Rather than discarding these inputs through gating mech- anisms or overwriting the existing memory state, it may be more effective to consider alternative strategies that allow for the preservation of diverse information without interference.
3.2 MoM: MIXTURE-OF-MEEMORES
To address the challenge outlined above, we propose novel approach for encoding multi- item memory such as theta-gamma oscilla- tions (Lisman & Jensen, 2013), and concepts from Mfxture-of--xpprts (MoE) (Shazeer et al., 2017), where different experts handle specific tokens. In this approach, we leverage mul- tiple memory states, each of which is selec- tively updated by different inputs. This in- creases the memory capacity and enables the model to retain diverse pieces of information by storing various types of inputs in separate memory states.
Outuu
Mutrrl
Mixture of Memories
X
Memory
Memory
Memoryy
Mamory N
Malmul
Matmul
Natmul
Pmoj
Proj
Proj
Proj
In our framework, the memory states function similarly to the experts in MoE. However, in- stead of relying on completely separate net- works, these modules individual RNN states embedded within a linear recurrent mechanism. This design allows for the isolation of mem- updates while concurrently managing dis- tinct types of information. It important to note that MoM essentially differs from tradi- tional MoE, as we will discuss Appendix B. Figure provides overview of the MoM ar- chitecture. Below, we introduce the structure of the MoM layer and explain how this multi- memory architecture is implemented in the con- text of linear sequence modeling.
lnval
topk-22
Local KV Nomony
Shared
Figure 1: MoM Architecture. Each input ken selectively activates and updates K memory states, leaving non-activated memory states un- changed to avoid interference from current input. Additionally. we introduce continuously acti- vated shared memory. This figure presents the basic memory update mechanism; other mecha- nisms involving gating more complex updates follow a similar approach.
3

---

Published as a conference paper at ICLR 2026
3.2.1 ROUTER
We use router assign imputs different memory states. Utilizing the top-k concept, each token is routed to the top-k memories based on its importance scores, Specifically, we use a simple linear layer to generate these scores for each input token, After applying a softmax function, we select the top-k scores and normalize them. scores; (sof(aa))) W₁)) R
(3)
scores; TR.
9r Scorcss
(4)
where 2, E Rd, the top-k number, W Rax M learnable weight, g. the normalized importance scores of the input Tt.
3.2.2 LINEAR RECURRENT MEMORY MODULE
After the router network, the input & is directed to top-k linear recurrent modules, meaning that the top-k memories are activated while the others remain inactive.
Each Memory. For each activated memory, indexed by m, we perform the following operation:
1. Key and Value Projections: We project the input 2t to kt and oz" using W and w
ke F. th e. æ. W E Rd,
(5)
where W W are learnable projection weights for kv of the m-th memory module.
Memory Update: We update the activated memory state using kt" ot"
M. M (k,")Te" v." gaxd
(6)
The equation above represents the simplest form of memory update for clarity. Our approach is flexible and does not rely on specific memory update mechanism. To enhance performance, we can incorporate mechanisms such as forget gates (Sun et al., 2023).
More generally, method be adapted incorporate various memory update methods pro- posed in previous work. Detailed descriptions of these methods are provided in Table
Memory Mixing. After updating the activated memory states, we perform weighted of these memory states using the importance scores obtained from Equation(4).
Table 1: Memory Update Rules. We demonstrate that several linear sequence models can be viewed as recurrent models in terms of memory updates, where a,, b, E (0, 1) are data-dependent scaler, a; data-dependent vector, and y is data-independent constant.
M
Σ
gt (m)
M Rdd
(7)
where M. is one activated memory and gi (m) is the im- portance score of M.".
Method
Memory Update Rule
We then obtain the output of the MoM by applying query vector to the mixed memory My:
Linear Alta
M. M.
RetNet
M. 7M. -1 +AA o.
GLA
M. (a] 1)M, 1 1.A
qi M, Idd.
(8)
M.
G-DeltaNet
a,(I N/ R.) 6
Finally, the output of the MoM layer is computed by ap- plying an activation function, normalization, and a linear transformation.
TTT
M. M. M.V.
Titans
M. a.M A.V
Mumba2
M. a.M. b.Ad
HGRN2
M. (nl 1)M + mn
Throughout the recurrent process, only subset of mem- ory states is activated and updated at each time step, while memory states that are not routed remain inactive and un- changed. When the input passes through the key-value projection layer, it generates multiple sets of keys and values that are fed into different memory modules. This design enables the model to maintain multiple memory states, each preserving dis- tinct pieces of information. By aggregating the activated memories into comprehensive mixed
RWKV6
M. = a.M. + b.
RWKV7
M. = (af 1)M + +.
4

---

Published as a conference paper at ICLR 2026
memory by weighted summation, the query effectively retrieve information from this mixed memory, and generate attention output followed by other layers.
Shared Memory. To enhance model's ability to capture long-term dependencies, introduce a shared memory mechanism. This shared memory has access to the entire sequence information, allowing it to effectively store and retrieve long-term information. By integrating shared memory into our model, we ensure that it can leverage the complete historical context, resulting in significant improvements in performance and robustness.
3.3 MARDWARE-EFFICHEE IMPLEMENTATION
Memory
Momory
Memory 3
Memory
Memory's
Triton Kernel
Memory
Memory
Memory 4
Original Sequence
Reorganized According to Memorias
Varlen Input Sequence
Figure 2: Hardware-efficient Implementation of MoM. Tokens sharing the same color are routed the same memory. ① Tokens are first split into groups according to memory routing results, @ then concatenated into a varlen input sequence, @ processed by the Triton kernel, @ the outputs are returned, @ split back into their respective memories, and @ finally restored the original sequence order. For clarity, the illustration shows the top-1 routing case, and the gleu projection is omitted.
In the implementation of MoM, mixing memories before query multiplication is equivalent to mul- tiplying each memory by the query and then mixing the results, allowing to reuse efficient Triton- based operators from prior linear sequence models. We first reorder the sequence tokens according the routing results that they follow the memory layout. The reordered tokens are then catenated with varlen for operator computation, after which the results are aggregated via weighted be effectively reduced varlen operations,
summation. In this way, MoM's computation abling efficient execution. We elaborate on this process below.
Given input tokens Fb, Rd for batch b {ly..., B} and time step E {l,..., T), each token routed to one or more memories m E M} with routing weights Obb,t.m 2 0 satisfying M
Em=1 Olb,t,m
For each (b, m), define the ordered index set
Ia.m (to.m (1), to (Lb,))),
where t (/) is the original sequence index of the j-th token assigned memory m. and T4,m ml We index buckets lexicographically by p = (b 1)M + m and define cumulative boundaries
0,
Sp
(p = 1. 1........ BM). Ln
The flattened sequence X is obtained by
+ sp-itj hh t))
1...., Lb,ms
with varlen representation (X, s). where (so. (so..... SBM).
For each bucket = (6 - M)M + m, queries share a projection matrix Wo, while keys and values memory-specifie projections W/() (m) (mm)
d. Wo k
w(m) (m)
Tus
6_ w(()) (m)...... R... E {8p-1 + : 8p))
memory-specifie kernel Fm with parameters ((m) is applied independently to each segment:
sy tis Fnn(as, thss h tlis 6 1:s 0 @lm)).
5

---

Published as a conference paper at ICLR 2026
Mapping outputs back to the original sequence, the j-th token in Ta,m has per-memory output
6b mm) sp-1
Finally, token-level representations are reconstructed by weighted summation:
M
Ao,t
ttm Ób,t,
m=1
EXPERIMENTS
4.1 EXPERIMENTAL SETUPP
Models. In our experiments, we employ the Gated DeltaNet (Yang et al., 2024) as the memory update mechanism in MoM. The model is configured with four memory states, two of which activated at each time step, along with a shared memory.
Baselines. We evaluate MoM against several linear recurrent models and Transformers, including RetNet (Sun et al., 2023), GLA (Yang et al., 2023), Gated DeltaNet (Yang et al., 2024), and Trans- former++ (Touvron al., 2023), which incorporates Rotary Position Embeddings (Su et al., 2024) and GLU (Shazeer, 2020) into the Transformer architecture. To ensure a fair comparison, we train all baseline models from scratch using the exact same number of tokens.
Training. We follow the training procedure described by Yang et al. (2023), utilizing the SlimPa- jama dataset (Soboleva et al., 2023) sampled with 100B tokens and tokenized using the Mistral tokenizer (Jiang et al., 2023). We train models from scratch with parameter sizes of 380M and 1.3B, respectively. For the 380M models, we train on 15B tokens with a batch size of 0.5M tokens. More detailed training configuration is provided in Appendix C. We utilized publicly available pretrained weights from Zhang al. (2024) with exactly same configuration !
Parameter Explanation. We report model sizes using the common shorthand, where "380M" denotes a configuration with 24 layers and hidden size 1024, and "1.3B" denotes 24 layers with hidden size 2048. The main goal of MoM is to expand the memory capacity of linear sequence models through sparse activation. To this end, we apply sparse activation only to the key and value projections, which results in small increase in activated parameters that is well justified by the performance gains. A detailed discussion on fairness is provided in Appendix G.
4.2
MAIN RESULTS
4.2.1
RECALL-INTENSIVE TASKS
Linear sequence models, due to their limited memory capacity, often exhibit a significant perfor- mance gap compared to Transformer models, especially in recall-intensive tasks where extensive context is crucial. These tasks highlight notable performance differences among various linear mod- els, making them more accurate benchmark for evaluating linear model's capabilities in handling contextual information.
To thoroughly assess model's proficiency in such scenarios, test six recall-intensive tasks following Arora et al. (2024): FDA (Arora et al., 2023). SWDE (Arora et al., 2023; Lockard et al., 2019), SQuAD (Rajpurkar et al., 2018), NQ (Kwiatkowski al., 2019), TriviaQA (Joshi et al., 2017) and Drop (Dua et al., 2019). These tasks are designed to challenge a model's ability to perform context-based retrieval and comprehension.
As shown in Table 2, our proposed approach, benefiting from increased memory capacity and memory mixing mechanism, achieves significant improvements over other linear sequence mod- els. Specifically, our model effectively narrows the performance gap with Transformer models. This improvement underscores the advantage of our method in capturing and utilizing long-range depen- dencies, thereby enhancing performance on tasks that require extensive contextual understanding.
4.2.2 LONG CONTEXT TASKS
"Mdels marked with asterisk use open-source pretrained weights with identical training configurations.

---

Published as a conference paper at ICLR 2026
Table 2: Results on Recall-Intensive Tasks. All inputs are truncated maximum length of 2K tokens. MoM significantly outperforms all other linear models across both model sizes. In the 1.3B model, MoM even achieves performance very close to that of Transformer models.
Avg. (us FIA)
Scale
Model
FDA
SWDE
SQUAD
TriviaQA
Drop
Avg.
380M Params
Transformer++
46.14
25.87
33.22
18.94
45.97
20.03
31.70
28.81
15B Tokens
RetNet
5.90
9.28
22.41
6.91
40.05
18.59
17.19
19.45
L=24, d=1024
HGRN2
11.53
17.34
24.08
12.67
17.35
21.14
23.06
GLA
11.26
16.78
27.85
12.77
43.90
17.68
21.71
23.80
GSA
6.36
16.87
21.90
14.60
42.18
16.72
19.77
22.45
Gated DeltaNet
20.53
23.24
28.55
14.98
44.91
16.48
24.78
25.63
MoM
22.98
29.90
29.69
16.60
48.82
20,99
28.16
29.20
1.3B Params
Transformer+++
44.32
32.43
42.59
24.49
58.47
21.56
37.31
35,91
1008 Tokens
RetNet
13.62
22.59
33,46
15.43
53.79
19.79
26.45
29.01
1,224, d=2048
HGRN2
12.35
23.24
33.19
19.10
55.27
19.65
27.13
30.09
GLAT
27.61
30.93
35.04
22.27
56.28
19.45
31,93
32.79
GSA
23.25
32.80
35.57
22.96
57.05
20.65
32.05
33.81
Gated DeltaNet
30.25
27.65
34.06
23.22
58.23
20.36
32.30
32,70
MoM
41.14
34,30
37,08
24.11
58.59
21.03
36.04
35.02
Table LongRench Benchmark Re- sults. Note: Sum Summarization, FS Few-shot, Syn = Synthetic.
Assessing performance on long-context tasks is crucial for linear models, it reflects their ability handle long-range dependencies effectively. We evaluated our model's comprehension of long contexts using the Long- Bench benchmark (Bai et al., 2024: Contributors, 2023). In Table we present the average results across various categories, including summarization, few-shot learning, synthetic tasks, and code completion, along with the over- all mean across all tasks. The complete detailed results are provided Appendix I.
Model
Sum
Syn
Code
RetNer HGRN2!
6,30
15.76
2.64
40.52
13.61
6,51
15.50
2.61
40.11
13.02
GSA!
7.75
20.29
1.92
42.83
14.61
Gated DeltaNet
7.14
18.00
2.10
41.52
13.98
MoM
Table 4: Comparison of Mixture of Memories and Single Memory Expanded. We constructed MoM models using different memory update mechanisms. Separate memory segments yielded bet- ter performance compared to simply increasing the memory capacity of single memory.
ARC-e acc?
ARC-c ace,
Hella. acc,1
Lamb. acct
PIQA accf
Wino. acef
Model
Params
Avg.
GLA expanded GLA MoM
425M
42.34
22.95
34.56
20.45
63.00
50.12
38 38.90
395M
42.85
24.15
36.60
23.23
63.22
49.88
39.99
Gated DeltaNet expanded
550M
43.60
24.66
37.80
26.90
64.47
50.51
41.32
Gated DeltaNet MoM
444M
44.65
24.74
36.54
27.93
66.16
51.78
41.97
Model
Params
FDA
SWDE
SQUAD
NQ
TriviaQA
Drop
Avg.
GLA expanded GL.A MoM
425M
15.08
20.15
28.28
13.30
41.65
18.74
22.87
395M
9.90
21.65
29.36
14.16
45.20
20.89
23.53
Gated DeltaNet expanded
550M
18.26
24.27
30.03
17.74
48.34
19.26
26.32
Gated DeltaNet MoM
444M
22.98
29.90
29.69
16.60
48,82
20.99
28.16
4.2.3 MIXED MEMORY VS. SINGLE MEMORY
To validate the effectiveness of our mixed memory mechanism, we compare our MoM model with mixed memories to a baseline model that uses an expanded single memory with the same activated memory capacity. We adopt the same memory update method as existing linear models and extend within our MoM framework. For comparison, we employed the commonly used method of ex- panding the single memory by expanding the dimension of match the total size of all activated memories in the MoM model. We evaluate their performance on common-seet sense reasoning tasks and recall-intensive tasks in Table 4.

---

Published as a conference paper at ICLR 2026
Table Comparison with the Same Activated Parameters. MoM and Gated DeltaNet with 400M activated parameters are tested.
ARC-e acct
ARC-c acca
Hella. acca
Lamb. accf
PIQA accf
Wino. acc1
Model
Params
Avg.
Gated DeltaNet
400M 400M
46.04 47.10
23.55
35.18
27.01 26.88
66,05 64.64
50.83
41.44
MoM
23.72
35,43
51.22
41.50
Model
Params
FDA
SWDE
SQUAD
NQ
TriviaQA
Drop
Avg.
Galed DeltaNet
400M
20.53
23.24
28.55
14.98 15.36
44.91
16.48
24.78
MoM
400M
24.16
25.59
29.46
46.15
18.35
26.51
20
morrsfomrre
800
Transfomeras
60
70
GSA
GLA
700
MaN Hemory
perplexity
Gateed De Dalte
600
Transftrmees Nemory
GPFD Marmary
Transfomeres DDM
400
300
20
100
Sequence Length
Context Length
Figure Inference Efficiency of MoM. We Figure 4: Length Extrapolation. We extrap- demonstrate the inference time and GPU mem- olated models trained on 2K sequences to ory consumption required to generate IK tokens length of 32K for perplexity (ppl) evaluation. specific sequence lengths.
The experimental results demonstrated that using multiple mixed memories leads to greater im- provement than simply expanding the capacity of a single memory with less parameters. This con- firms that mixed memory effectively reduce interference from different inputs. Assigning inputs specifically to different memories, combined with the use of forget gate, proves to be a more effective approach for reducing interference than relying solely on forget gate.
4.2.4 EFFICIENCY
We compare the inference speed and memory usage of MoM and Transformer++ with flash attention Fig 3. Our analysis demonstrates that MoM exhibits linear complexity, showcasing significant ad- vantages over the Transformer model when handling long sequences. Specifically, MoM's efficient memory update mechanisms allow to process longer inputs with reduced computational overhead, scalable solution for large-scale natural language processing tasks.
positioning it
4.2.5 LENGTH EXTRAPOLATION
We pretrained the models on the Slimpajama dataset with a 2K context length and conducted ex- trapolation experiments on various lengths using the Fineweb (Penedo al., 2024) dataset. We extended the length to 32K to calculate perplexity (ppl). As shown in Fig 4, the Transformer model experienced significant increase in ppl due its poor extrapolation capability. Among the linear models, MoM achieved the best results.
4.2.6 MEMORY ANALYSIS
Memory Load Balance Analysis. To evaluate whether each memory segment in MoM is effectively balanced during inference downstream tasks, we analyzed the number of tokens routed to each layer using around 300k tokens from the ARC-easy benchmark. We visualized the results with auxiliary loss (following the formulation introduced in Switch Transformer (Fedus al., 2022)) in Fig with heatmaps and we also visualized results with auxiliary loss in Fig 10. Due to the adoption of auxiliary loss, the memory segments in each layer are almost uniformly routed and activated.
8

---

Published as a conference paper at ICLR 2026
Memory Specialization Analysis. To quantitatively investigate whether the router guides memories toward specialized roles, we analyzed the routing decisions within the model's deep layers during inference the eRC- benchmark. We sampled the input hidden states (æ,) for large number of tokens and used UMAP (MeInnes et al., 2018) to project these high-dimensional states into a 21) space. Each point in the visualization color-coded by its Top-1 routed memory index.
The results are presented Figure 5. We observe a clear and distinct clustering phenomenon, which confirms that the router has learned meaningful specialization. Since our model employs top-k=2 routing, each token is routed to two memories. For visualization clarity, the figure only plots the top-1 memory destination. Consequently, some overlap at the cluster boundaries is visible, which is an expected outcome of this design.
This clustering analysis also provides new perspective for understanding MoM from Test-Time Training (TTT) point of view. The memory update mechanism we use, Gated DeltaNet, adopts Delta Rule learning style, dynamically fitting k - mapping test time (with the optimization goal &M = v). When test data is highly discrete or widely distributed in the feature space, single memory network M struggles to fit all the data quickly and accurately. Our UMAP analysis demon- strates that the MoM router acts as dynamic clustering mechanism, automatically partitioning the broad input data stream during inference into multiple, more concentrated, and cohesive subsets. then assigns each subset to specialized memory for processing. This is equivalent implementing a form of TTT Ensemble Learning: each memory Mn. no longer needs to fit the entire complex data distribution, but only simpler sub-distribution, thereby reducing the learning difficulty.
UMAP Visualization (MoM Layer 22)
UMAP Visualization (MoM Layer 23)
(rdeg countp atbing serrg
30
Mamaay
UPAP Dimerkkn
Vanaay
Vanay
Mamary'
-20
J940 Dimention
UMAP Oimersion
Figure 5: UMAP visualization of memory specialization. Token hidden states are colored by their top-1 memory destination. The distinct clustering suggests a learned specialization. The partial overlap is an expected artifact of plotting the top-1 destination for top-k=2 router.
Memm1 0.6 0.25 0.28 0.23
0.24 0.21 6.22
0.22
0.25..25...2
022
023
0.33
0.37
0.28
0.22
0.24
0.31
0.27
0.14 0.25
Memonies
Mem-2
0.19
0.21
0.27
0.36
60.26
0.24
6.25
0.18
014
0.21
0.34
0.28
0.25
0.26
0.30
0.25
0.21
Mami4
0.26
0.30
0.25
3
3
L144
L15
L16
L17
L18
6119
L20
121
1:23
Layers
Figure 6: Memory Load Balance Analysis. Token Routing Distribution Across Layers and Mem- ories with Aux Loss.

---

Published as a conference paper at ICLR 2026
Recall
Common
Table 6: Ablation memory count and shared memory, showing average accuracy across recall-intensive tasks.
29.14
42.5
29
27:
42.0
27
Avg
41.5
41.34
Recall f Common 1
25
24.49
40.87
Aux Loss Scale
41.0
1e-2
27.59
42,10
23.27
5e-3
26.55
41.71
23
40.5
1o-3
28.16
41.97
2
8
27.23
41.58
Number Memories
Number Memories
Shared Memory
w/ shared memory
28.16
41.97
Figure 7: Scaling performance with increasing number of memories with a fixed activation ratio of 0.5.
w/o shared memory
26.06
40.38
4.2.7 MoM SCALING UP ABLATION STUDY
We examine the effect of scaling both the number of memory states and the number of top-k tivations in MoM. To ensure comparability, Fig. reports results with a fixed activation ratio of 0.5. Increasing the number of memories from 8 consistently improves performance across both recall-intensive and commonsense benchmarks. These results indicate that enlarging the memory pool effectively mitigates interference and enhances capacity. More comprehensive results covering other activation ratios and activation settings are provided in Appendix H.
We further study the influence of auxiliary loss and shared memory in MoM, using 380M- parameter model trained on 15B tokens. As shown in Table 6, auxiliary loss improves stability and performance when applied with suitable weight. In addition, shared memory consistently benefits performance with global information. These results highlight the complementary roles of auxiliary loss and shared memory in stabilizing and enhancing MoM.
5 CONCLUSION
In this paper, we propose Mixture-of-Memoria (MoM), a novel architecture that enhances memory capacity and eliminates memory interference. By leveraging multiple independent memory states, MoM significantly improves performance on recall-intensive tasks while maintaining the efficiency advantages of linear models. Instead of simply discarding tokens done in gating mechanisms, memory separation paradigm provides a more effective way to preserve sequence information. Our experimental results demonstrate that MoM outperforms existing linear sequence modeling meth- ods, particularly on tasks requiring strong recall, and achieves performance comparable to Trans- former models. This makes MoM promising approach for applications need strong efficiency and recall-in ieall--ntennsve performance, paving the way for efficient sequence modeling.
ETHICS STATEMENT
This work does not involve human subjects, sensitive data, or high-risk applications. All experi- ments are conducted on publicly available datasets. We encourage responsible and ethical use of the proposed methods in line with community standards.
REPRODUCIBILITY STATEMENT
Our code is released https: //ttt///ghhub com htmm/OpenSparseee.... /MoM and all the models we trained from scratch will be released at huggingface. We provide the training and evaluation scripts to make that all the results in the paper be easily reproduced.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et. GPT-4 technical 2023.
report. arXiv preprint
Simran Arora, Brandon Yang. Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trum- mer, and Christopher Ré. Language models enable simple systems for generating structured views of heterogeneous data lakes, 2023.
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughe tradeoff. arXiv preprint arXiv:2402. 2024.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, mul- titask benchmark for long context understanding, 2024. URL https:// /ttp///aavvvvv//vv org/abs/ 2308. 14508.
Maximilian Beck, Korbinian Pöppel, Markus Spanring. Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint wrXi:2245.005117 2024.
Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00665 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty- AAAI Conference Artificial Intelligence, 2020.
György Buzsáki. Theta oscillations in the hippocampus. Neuron, 33(3):325-340) 2002.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, Jbs/1803.05457, 2018.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. com hww.aa/.ocoooopeccoooppaaaaa http///ppppcpssseeeeeee/ee 2023.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specializaaionnnnn in mixture- of-experts language models. arXiv preprint S11i:22001060666 2024.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint 4rX1v22405.21060, 2024.
Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Al- bert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, al. Griffin: Mix- ing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19422 2024.
Licurgo de Almeida, Marco Idiart, and John E Lisman. A second function of gamma frequency oscillations: an e--max winner-take-all mechanism selects which cells fire. Journal of Neuro- science, 29(23)
2009.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning paragraphs. arXiv preprint 1rXiv:1903.0016,, 2019.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
11

---

Published as a conference paper at ICLR 2026
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa. Jason Phang. Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. 108062..
org/ rrcgrrrcddd////
Albert Gu and Tri Dao. Mamba: Linear- sequence modeling with selective state spaces, 2024. URL htt.tt:t//axxiiii org/ abs wtw.////... 2.12..2222 00752.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured
state spaces, 2022. URL https://axxii...ttt.tm htt.:orgg/bbb////aa 00396..
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effec- live as structured state spaces. In S. Koyejo, S. Mohamed, A. Agarwal, D. Bel- grave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 22982-22994. Curran Associates, Inc., 2022. URL
9156b0 -agerreeeeeeeeecttoeeeooo------- Copeeeereeeeeee-eCeeeereeen filee/
ct http:pps.ccpppper.leeee////epppp wwweesieeepaaaeeee fttpss/apppe/22/////. Bcw9cc4599
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint urXiv:1705.03355 2017.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns:: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165, PMLR, 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint ar arXiv:2006. 16668, 2020.
Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.088
2025.
John E Lisman and Ole Jensen. The theta-gamma neural code. Neuron, 7766) 2013.
Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. OpenCeres: When open information extrac- tion meets the semi-structured web. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 3047-3055 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-1
URL https: ac http:////aaaaaaaaaaoocacc oow.holooy..org//11111133
Ilya Loshchilov, Frank Hutter, al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.011013 5, 2017.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: (eoo:0002200 2018.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
12

---

Published as a conference paper at ICLR 2026
Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pas- canu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pp. 26670- 266700.2668... PMLR, 2023.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoe Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv: 1606.666366633 2016.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirt Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL BtLhttts:/ httptt:://ppeeeeeeeeeeeee http//ppneeeeeeeeeeeeeeeee net forum?? id=n6SCkn22es QaG.
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, et al. Eagle and finch: Rwky with matrix- valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892 2024.
Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Bao- hong Lv, Xiao Luo, Yu Qiao, et al. Transnormerllm: A faster and better large language model with improved transnormer. 2023a.
Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Bao- hong Lv, Fei Yuan, Xiao Luo, al. Scaling transnormer 175 billion parameters. arXiv preprint arXiv:2307.14990 2023b.
Zhen Qin, Xuyang Shen, Dong Li, Weigao Sun, Stan Birchfield, Richard Hartley, and Yiran Zhong. Unlocking the secrets of linear complexity sequence model from a unified perspective. arXiv preprint arXiv:2405.1 17383, 2024a.
Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models. arXiv preprint arXiv:2401 04658, 2024b.
Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. arXiv preprint arXiv:2405. 2024c.
Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgm2: Gated linear rnss with state expansion. arXiv preprint WrXiv:2404.07904 2024d.
Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for se- quence modeling. Advances in Neural Information Processing Systems, 36, 2024e.
Xiaoye Qu, Daize Dong, Xuyang Hu, Tong Zhu, Weigao Sun, and Yu Cheng. Llama-moe v2: Ex- ploring sparsity of llama from perspective of cexture-of-expers with post-training. arXiv preprint arXiv:2411.15700 2024.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad, 2018.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. arXiv preprint arXiv: /907:///7.../// 2019.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The pparsely-gated mixture-of-experr layer. arXiv preprint /701 00638, 2017.
13

---

Published as a conference paper at ICLR 2026
Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, and Yiran Zhong. Scaling laws for linear complexity language models. arXiv preprint orXiv:2406.16680 2024.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, 2023. URL https: //huggingface. co datasetssaaaaaaooa cerebrrsserebbreaeaeeee /S1 5limPs a ama- 627B.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Weigao Sun, Zhen Qin, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. Linear attention se- quence parallelism. arXiv preprint arXiv:2404.m 02882, 2024a.
Weigao Sun, Zhen Qin, Weixuan Sun, Shidi Li, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. Co2: Efficient distributed training with full communicationn-ompuaatiii overlap. arXiv preprint arXiv:2401.16265 2024b.
Weigao Sun, Disen Lan, Yiran Zhong, Xiaoye Qu, and Yu Cheng. Lasp-2: Rethinking sequence parallelism for linear attention and its hybrid. arXiv preprint arXiv:2502.0756: 2025.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (leam test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04622 2024c.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuging Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
Xiaqiang Tang, Weigao Sun, Siyuan Hu, Yiyang Sun, and Yafeng Guo. Ms-net: A multi-path sparse model for motion prediction in multi-scenes. IEEE Robotics and Automation Letters, 2023.
InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
Qwen Team. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024. URL https::/ httpps///wwenmm... github. ic ht/blo/////ooeooocccecoooeoooeea httpllog/wwwwmmmm///..//// qwen-moe/.
Hugo Touvron, Thibaut Lavril Gautier Izacard, Xavier Martinet, Marie-Anee Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23Xiv:2002.13977 2023.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:23/2.06665, 2023.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang. Wei Bi, et al. Gated slot attention for efficient linear-time sequence modeling. arXiv preprint arXiv:2409.074
2024.
Beitong Zhou, Jun Liu, Weigao Sun, Ruijuan Chen, Claire J Tomlin, and Ye Yuan. pbsgd: Powered stochastic gradient descent methods for accelerated non-convex optimization. In IJCAL, pp. 3258- 3266, 2020.
Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-man moe: Building mixture-of from llama with continual pre-training In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15913- 15993, 2024.
14

---

Published as a conference paper at ICLR 2026
A
RELATED WORK
LINEAR RECURRENT MODELS
Linear recurrent models, comprising linear attention, linear RNNs, state-space models (SSMs), have garnered significant research interests (Qin et al., 2023b). The advancement of SSMs began with the pioneering work on S4 (Gu et al., 2022), which was later optimized through a diagonalized version (Gupta et al., 2022). Despite their strong performance on the LRA benchmark, these models have faced challenges in language modeling mainly because they rely solely on data-independent processes, As research progressed, constant forgetting gates were introduced, helping to alleviate some interference issues by uniformly managing memory decay (Sun et al., 2023; Gu & Dao, 2024).
The next breakthrough involved data-dependent forget gates. These allowed models to dynamically adjust memory updates based on the input data, significantly enhancing performance across various tasks (Qin et al.. 2024c;d; Yang et al., 2023; Zhang al.. 2024; Yang et al., 2024; Qin et al., 2024a). Sequence parallelism techniques are well adapted on linear recurrent models (Sun et al., 2024a; 2025) for efficient long context training. There have also been recent advancements in scaling law and test-time regression optimization (Shen et al., 2024; Sun et al., 2024c; Behrouz et al., 2024).
Building on these advancements, our MoM model incorporates data-dependent mechanisms that selectively update memory. By efficiently managing interference through tailored memory updates and leveraging increased memory capacity, MoM represents a further evolution, improving model expressiveness and performance.
MIXTURE-OF-EXPERTT
Mixture-of- of--eertss (MoE) is technique designed to enhance the capacity of deep neural networks while maintaining computational efficiency (Fedus al., 2022: Rajbhandari al., 2020; Lepikhin et al., 2020; Tang et al., 2023; Zhu et al., 2024; Qu et al., 2024). MoE achieves this by activating a subset of parameters, known as "experts", for each input, which reduces the computational costs. Shazeer first integrated MoE into LSTM layers (Shazeer et al., 2017). The Switch Transformer (Fedus et al., 2022) refined this approach by simplifying the gating mechanism to select only one expert per input. Gshard (Lepikhin et al., 2020) further advanced this by using top-2 expert routing strategy to improve performance. Recent MoE models, such as Deepseek-MoF (Dai et al., 2024), introduce shared experts capture and consolidate common knowledge across different contexts, while designing fine-grained experts to increase combinatorial flexibility.
B COMPARISON BETWEEN MoM AND MOE
While our approach implementing the Mixture (MoM) draws inspiration from the Mixture-of- (MoE) framework, there are notable differences that distinguish our method from traditional MoE implementations.
Purpose:
The MoE. was introduced to scale up the number of parameters without signif- icantly increasing computational resources. It address the limitations of dense models in scaling both parameters and computational demands through sparse activation. However, MoM is designed to expand the memory capacity of linear attention models while pre- serving their linear time complexity. By sparsely activating memories and using weighed summation create mixed memory, MoM effectively address the challenge of forgetting historical information in linear attention. Moreover, by separating the memory into distinct states, MoM reduces interference between different pieces of information.
Structure: In conventional MoE, each expert is a separate neural network within the feed- forward network (FFN) layer such as Mwen-MoE (Team, 2024) and Linear-MoB Sun et al. (2024a). In contrast, in MoM, each memory is RNN state with distinct key-value pro- jection weights to generate different key-value pairs. MoE operates during the channel mixing phase, where each token is processed independently by selected experts. On the other hand, MoM functions during the token mixing phase, where each memory processes different segments of the sequence, preserving inter-token relationships.
15

---

Published as a conference paper at ICLR 2026
Table 7: Results on Common-Sense Reasoning Tasks. The performance of linear models and Transformer models is comparable; however, MoM consistently achieves the best average perfor- mance across all model sizes.
Wiki. ppl4
Lamb, ppl]
ARC-e acc?
ARC-c acta
Hella. acca
Lamb, accf
PIQA ace?
Wino. ace!
Scale
Model
Avg.
380M Panams
Transformer++
26.88
76.46
44,91
25.94
34.95
26.90
64.31
$1.07
41.35
Tokens
RetNet
31.07
44,49
23.93
52.33
40.19
1.=24, d=1024
HGRN2
27.90
77.40
45.24
23.63
35.61
24.74
65.45
54,06
41.46
GLA
28.78
79.95
44.53
22.27
34.84
24.94
63.93
51.38
40.32
GSA
28.17
82.50
45.50
24.23
35.00
24.02
64.85
50.43
40.67
Gated DeltaNet
26.47
46.04
35.18
50.83
MoM
25.86
55.41
44,65
24.74
36.54
27.93
66.16
51.78
41.97
1.38 Params
Transfomer+++
17.61
19.29
55.01
28.07
49.21
40.95
70.08
56.27
49.93
100B Tokens
RetNer!
18.18
21.97
57.49
26.88
48.09
37.75
69.37
53.28
48.81
L=24, d=2048
HGRN2
17.32
15.65
58.33
28.07
51.93
42.31
71.33
52.01
50.66
GLAT
17.61
19.66
55.18
27.56
48.89
40.03
69.86
53.91
49.24
GSA
16.69
16,02
58.33
28.33
50.98
42.03
72.25
53.43
50.89
DeltaNet
17.14
18.80
56.82
27.39
49.77
39,94
71.76
51.78
49.58
MoM
16.64
14.83
55.35
27.99
50.95
43.43
71.27
56.83
50.97
C EXPERIMENTS DETAILS
For the 380M models, train 15B tokens with a batch size of 0.5M tokens. The warmup tokens count is set to 0.25M.. We set the hidden ratio of our model to 3 to keep the activated parameter count approximately the same. For the 1.3B models, we train on 100B tokens with a batch size of 2M tokens. The warmup tokens count is IB. We employ Adam optimizer (Loshchilov et al., 2017; Sun al., 2024b) with learning rate of 3e-4 with cosine learning rate schedule (Zhou al., 2020). The weight decay is set to 0.01 and gradient clipping is 1.0. Our experiments were conducted using 32 NVIDIA A800 GPUs. Training the 380M parameter model required approximately 10 hours, while the 1.3B parameter model took around 6 days.
D COMMONSENSE REASONING TASKS
As shown in Table 7, report the language modeling perplexity and zero-shot performance of com- monsense reasoning tasks following (Zhang et al., 2024) which includes WikiText (Merity et al., 2016), LAMBADA (Paperno et al., 2016), ARC-easy, ARC-challenge (Clark al., 2018), Hel- laSwag (Zellers et al., 2019), PiQA (Bisk et al., 2020) and WinoGrande (Sakaguchi et al., 2019). The evaluation results based on the Im-evaluation-haren haannsss (Gao et al., 2024).
Experimental results show that MoM outperforms other linear models and surpassed the Trans- former model well.
E TRAINING LOSS COMPARISON
To further assess the learning efficiency of MoM, compared the training loss curves of MoM with those of other baseline models. As depicted in Figure 8, MoM consistently maintains the lowest loss throughout the entire training phase. Even as training nears convergence, MoM continues exhibit clear advantage over other methods.
F THE HYBRID OF MoM AND TRANSFORMER
We delve deeper into the hybridization of MoM and Transformer layers by integrating I Trrnsforrmer layer after every MoM layers, resulting in only Transformer layers across total of 24 layers. The performance commonsense reasoning and recall-intensive tasks is presented in the table 8. Hob-H demonstrates significantly improved results compared to Transformer models, despite using only layers of global attention.
16

---

Published as a conference paper at ICLR 2026
3.2
3
2.8
15k
25k
Transformer
6LA
HGRN2
65A
Retteet
MoM
Gaced Deltanet
Figure 8: Training Loss. Loss curves for training 380M models on 15B tokens with fixed random seed of 42.
Table 8: Hybrid Model Performance.
The hybrid model integrates Transformer layer after every MoM layers, resulting only 3 Transformer layers across total of 24 layers.
Model
FDA
SWDE
SQUAD
NQ
TriviaQA
Drop
Avg.
Transformer++
46.14
25.87
33,22
18.94
45.97
20.03
31.70
MoM
22.98
29.90
29.69
16.60
48.82
20.99
28.16
MoM Hybrid
58.13
44.05
35.71
20.18
48.10
20.60
37.80
ARC-e acct
ARC-c
Hella.
Lamb.
PIQA
Wino.
Model
Avg.
acca
acc,
acef
accf
accf
Transformer+
44.91
25.94
34.95
26.90
64.31
51.07
41.35
MoM
44.65
24.74
36.54
27.93
66.16
51.78
41.97
MoM Hybrid
46.55
24.49
36.45
28.86
65.51
52.41
42.38
G FAIRNESS
To enhance memory capacity, MoM applies sparse activation to the key and value projections. Al- though these projections constitute a small portion of the overall model parameters, this inevitably increases the parameter count. Due to differing linear model structures, aligning both parameter count and memory capacity exactly is challenging. Thus, to ensure fairesss, we conduct compar- isons from two perspectives: equal activated parameter count and equal memory capacity.
G.1 EQUAL ACTIVATED PARAMETER COUNT
To ensure fair comparison of parameters, reduced the MLP hidden ratio to 2 and retrained the MoM model using the same training configurations as in Section 4.1. Both MoM and Gated Deltanet were set with 400M activated parameters. Although the smaller hidden ratio might impact the model's commonsense knowledge, we tested on commonsense reasoning tasks and recall-ine inttenin benchmarks. MoM consistently outperformed Gated Deltanet in both tests, further validating the effectiveness of the MoM approach. The results are presented in Table
G.2 EQUAL MEMORY CAPACITY
To ensure a fair comparison of memory capacity, we also compared the single extended memory model with the MoM model. Notably, the single extended memory model has more parameters than the activated parameters in the MoM due the extension of the dimension. MoM expands memory more elegantly and significantly outperforms both recall and commonsense tasks. This comparison result is presented in Table 4.
17

---

Published as a conference paper at ICLR 2026
H DETAILED SCALING RESULTS
To further examine the scalability of MoM from the perspective of memory capacity, we evaluate the effect of enlarging the memory pool beyond the main settings. Specifically, activation ratios, where the number of active memories accounts for either 0.5 or 0.25 the total memory states. In all cases, additional shared memory is included. Starting from single memory as the baseline, we expand the number memories to 2, 4, 8, and 16, and report the averaged results on both recall-intensive and commonsense benchmarks. The results, shown in Fig. 9, indicate that under a fixed activation ratio, increasing the memory size consistently improves performance on both categories of tasks.
compare two
30
kmn-25%
kin-25%
kmn=50%
43.5
k0n500%
swerage Recall Accuracy (%)
n=1 (start)
Aueraga Common Accurary (36)
n=1 (start)
28
27
42.5
26
42.0
25
24
0
41.0
23
Number Memories
Number of Memories
Figure 9: Detailed scaling results. Performance shows general improvement on both recall- intensive and commonsense tasks as the number of memories increases.
1 FULL RESULTS
1.1 ABLATION ANALYSIS ON SHARED AND SPLIT MEMORIES
To rigorously verify the necessity of the mixture mechanism, we further disentangle the contributions of the shared memory and the split memories in Ta- ble 9. As discussed in Section 3.2.2, the shared mem- ory is designed to capture global context, serving as complementary component to the split memories which specialize local token subsets. It is impor- tant note that the "Shared Memory Only" config- uration is mathematically equivalent to the standard Gated DeltaNet baseline.
Table 9: Ablation study disentangling the effects of shared and split memories.
Model Configuration
Avg. (Recall)
Shared Memory Only
24.78
Split Memory Only
26.06
Mixed Memory (MoM)
28.16
1.2 ROBUSTNESS ROUTER INITIALIZATION
To verify the stability of the routing mechanism, conducted ablation study router initial- ization. Using the 380M model trained on 5B tokens, we fixed the initialization seed for all other model parameters (seed=42) while varying the random seed specifically for the router network.
As shown in Table 10, the variations in both validation loss and downstream task performance (aver- age accuracy commonsense benchmarks) negligible, with the loss variance remaining within 0.07%. This empirical evidence confirms that the MoM architecture is robust to router initialization strategies.
18

---

Published as a conference paper at ICLR 2026
Table 10: Sensitivity analysis of router initialization. Models were trained for 5B tokens.
Router Seed
Loss
Avg. (Commonsensse)
2.9396
40.37
42
2.9409
40.20
1234
2.9426
40.07
Model
SQA
MQA
Sum
Syn
Code
Zh-Avg
En-Avg
Avg.
RetNet
9.23
7.23
6.3
15.76
2.64
40.52
15.44
13.5
13.61
HGRN2
7.38
15.5
2.61
40.11
14.28
13.12
13.02
GSA
8.21
6.63
7.75
20.29
1.92
42.83
15.06
15.2
14.61
Gated DeltaNet
8.52
6.61
7.14
18
2.1
41.52
14.19
14.63
13.98
MoM
8.14
7.11
6.89
21.26
2.63
47.79
17.33
15.71
15.64
Table 11: Complete Results of LongBench. SQA: Single-doc QA, MQA: Multi-doc QA, Sum:
Summarization, FS: Few-shot learning, Syn: Synthetic
0.5
Mem-11-0
0.49
0.4
Mem-2
0.35
0.50
0.48
0.00
0.33
0.03
0.10
0.50
0.01
0.50
0.50
0.50
0.32
0.01
0.02
0.03
0.00
0.38
0.49
0.00
0.3
Mum-3 9.39
0.05
0.49
0.48
0.50
0.01
0.01
0.04
0.08
0.03
0.34
0.40
0.49
0.33
0.11
0.44
0.04
0.35
0.03
0.02
0.07
0.50
0.1
Mem-4 -0.00
0.45
0.02
0.02
0.00
0.49
0.45
0.50
0.42
0.49
0.00
0.03
0.00
0.11
0.08
0.48
00.36
0.45
0.32
0.47
0.50
0.42
0.45
0.23
L3
L5
L6
L10
L16
L17
L19
L20
L21
1.2
123
124
Layers
Figure 10: Memory Load Balance Analysis. Token Routing Distribution Across Layers and Mem- ories without Aux Loss.
J THE USE OF LLMS
Large language models (LLMMs) were only used to refine the grammar and spelling of some para- graphs in this paper. They were not used for generating research ideas, designing experiments, or writing substantive content.
19


Reviews:
[{"review_id": "X2lbItumeV", "reviewer": "Reviewer_vGVV", "summary": "This paper presents Mixture-of-Memories (MoM), a framework for linear sequence modeling that replaces a single fixed memory with multiple independent ones combined through a routing network. Drawing inspiration from both neuroscience (multi-item memory mechanisms) and the mixture-of-experts (MoE) concept, MoM expands memory capacity and reduces interference, making it especially effective for recall-intensive tasks. Experiments on language benchmarks show that MoM delivers clear gains in recall-focused and long-context tasks, often matching or surpassing Transformer baselines in performance.", "questions": "1. Could the authors clarify how gradients are propagated through the Top-K router? \n\n2. For recall-intensive benchmarks, the source of improvement is unclear. It is not specified whether the gains come from increased memory capacity, larger parameter count, or better routing dynamics. Could a control experiment with a single enlarged memory help isolate these factors?\n\n3. In the hybrid architecture experiments, do the observed gains mainly result from increased depth, or is there measurable synergy between global attention and mixture-based memory modules?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper pinpoints the weakness of compressing an entire sequence into a single memory state in linear models and connects this limitation to memory interference.\n\n2. The empirical study is extensive, covering recall-intensive benchmarks, long-context tasks, memory behavior analysis, scaling and ablation studies.\n\n3. The authors provide qualitative insights into memory specialization and routing distributions, enhancing the interpretability of MoM.\n\n4. Implementation details show careful consideration of computational cost, employing Triton kernels for batched memory routing and maintaining O(n) training and O(1) inference efficiency.", "weaknesses": "1. The router formulation specifies a scheme combining softmax and Top-K but lacks crucial implementation details. It does not clarify how ties in Top-K selection are resolved, whether gradients are propagated through the Top-K operation or treated as non-differentiable, or how the learned matrix affects the sparsity of routing and the memory load balance. It also leaves open whether the router is robust to input distribution shifts or token imbalance.\n\n2. Table 5 provides qualitative insights into memory specialization, but the analysis remains descriptive. Quantitative measures would better substantiate claims of interpretability.\n\n3. The theta-gamma memory analogy provides an appealing motivation but lacks a rigorous computational mapping. The biological discussion lack ablation or formal modeling to validate the link between oscillatory cycles and MoM’s memory dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hcp1dEEnB7", "reviewer": "Reviewer_EVGR", "summary": "This paper addresses a well-known weakness of linear sequence models (such as linear RNNs and state-space models): their reliance on a single, fixed-size memory state, which leads to memory interference and poor performance on recall-intensive tasks. The authors propose Mixture-of-Memories (MoM), a new architecture that utilizes multiple independent memory states. A top-k routing network directs each input token to a specific subset of these memories for updating. This mechanism aims to increase the model's effective memory capacity and reduce interference by allowing different types of information to be stored in separate states. The final output is computed by querying a weighted mixture of the activated memory states. The paper demonstrates through extensive experiments that MoM significantly outperforms strong linear model baselines on a variety of downstream tasks, particularly those requiring information recall. Notably, the model achieves performance comparable to standard Transformer models while retaining the linear-time training and constant-time inference benefits of linear RNNs.", "questions": "See my weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper targets a critical and widely recognized weakness of linear-time sequence models: their poor performance on recall-intensive tasks due to the bottleneck of a single fixed-size memory state.\n- The experimental results are comprehensive and robust. \n\n- The core idea of using a top-k router to manage multiple, independent RNN memory states is a novel and clever application of sparse activation principles.", "weaknesses": "- The ablation study in Table 6 indicates that the \"Shared Memory\" component is critical. Removing it causes a large performance drop (e.g., 2.1 points on Recall tasks). This makes it difficult to disentangle the gains from the \"Mixture\" mechanism (routing to $k$ of $N$ memories) from the gains of simply having a parallel, always-on \"Shared Memory\" state. The paper's narrative focuses heavily on the top-k mixture, but a large portion of the gains might be coming from this simpler shared component.\n- The main results in Table 2 appear to compare models of slightly different sizes. For example, Table 4 shows the \"Gated DeltaNet MoM\" model has 444M parameters, while its baseline \"Gated DeltaNet\" is 380M. This ~17% parameter difference complicates the main comparison. While a fairer study with equal activated parameters is commendably included in Appendix G, the primary results table in the main body of the paper should ideally be based on a more strictly controlled comparison.\n- The paper repeatedly claims MoM is a \"new paradigm\" and \"fundamentally differs\" from MoE (e.g., in Appendix B). This distinction feels overstated. The core mechanism, a top-k router that sparsely activates a subset of modules, which is the defining feature of MoE. Applying this concept to RNN states and key/value projections instead of FFN layers is a novel and useful application, but it does not seem to be a fundamental departure from the MoE paradigm.\n-  The analysis of memory specialization in Section 4.2.6 and Table 5 is weak. The paper provides no quantitative methodology for how token categories (e.g., \"Basic nouns/verbs\" vs. \"Proper nouns/scientific terms\") were defined or how the \"Potential Function\" (e.g., \"Simplify semantic information\") was determined. As presented, these conclusions appear entirely subjective and anecdotal, lacking the empirical rigor demonstrated in the rest of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xCRA0zbMMP", "reviewer": "Reviewer_SvM8", "summary": "This paper introduces Mixture-of-Memories (MoM), an architecture designed to enhance linear sequence models by addressing their fundamental limitation: compressing entire sequences into a single fixed-size memory state. MoM employs multiple independent memory states with a router network that directs input tokens to specific memories using a top-k selection mechanism. The approach is inspired by theta-gamma oscillations in the hippocampus and borrows conceptually from Mixture-of-Experts (MoE) architectures. The authors demonstrate that MoM maintains O(n) training complexity and O(1) inference complexity while significantly improving performance on recall-intensive tasks. Experiments on 380M and 1.3B parameter models show MoM outperforms existing linear sequence models and approaches Transformer performance on recall tasks, though with modest increases in parameter count. The work provides a general framework compatible with diverse memory update mechanisms and includes a hardware-efficient implementation using varlen operations.", "questions": "1. Could shared memory become a new bottleneck? If all tokens access shared memory, will its capacity quickly become saturated? Have you analyzed the actual utilization of shared memory across different layers?\n2. More details about reproduction：\n\ta. What is the computational overhead of the router network?\n\tb. How sensitive is performance to router initialization?\n\tc. What are actual wall-clock training times compared to baselines?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Well-motivated problem formulation: The paper clearly identifies memory interference and limited capacity as core issues in linear sequence models, providing both intuitive explanations and neuroscience-inspired motivation from hippocampal theta-gamma oscillations.\n2. General and flexible framework: MoM's compatibility with various memory update mechanisms (Table 1 lists 11 different methods) demonstrates its generality. This is a significant practical advantage, allowing easy integration with future linear modeling innovations.\n3. Comprehensive experimental evaluation: The paper includes extensive experiments across multiple benchmarks (recall-intensive tasks, long-context tasks, commonsense reasoning), multiple model scales (380M, 1.3B parameters), and thorough ablations (memory count, activation ratios, auxiliary loss, shared memory).\n4. Strong empirical results on target tasks: MoM achieves substantial improvements on recall-intensive tasks (28.16 vs 24.78 average for Gated DeltaNet at 380M; 36.04 vs 32.30 at 1.3B), effectively narrowing the gap with Transformers (31.70 and 37.31 respectively).\n5. Hardware-efficient implementation: Section 3.3 provides a mathematically rigorous description of the varlen-based implementation, demonstrating practical viability. Figure 3 shows clear efficiency advantages over Transformers at long sequences.", "weaknesses": "1. While related work in linear sequence modeling and MoE has been mentioned, it lacks in-depth comparisons and differential analyses with recent methods (such as RWKV-7 and Titans). In particular, Table 1 lists these methods but does not systematically experimentally compare the performance differences of MoM under different memory update mechanisms.\n2. Table 5 claims to have discovered \"specialization\" in different memories, but it is based solely on qualitative observations of an intermediate layer and lacks systematic and statistical significance verification. This \"discovery\" is more like cherry-picking data than rigorous analysis.\n3. The authors claim that the core advantage of MoM is \"eliminates memory interference\" by separating memory states, but also introducing shared memory to capture global information. Does this contradict the core design? In particular, in the ablation of table 6, there seems to be a lack of settings for \"shared memory only\" and \"neither shared memory nor single memory exists\". This lack of ablation experiment is fatal because it directly relates to whether \"mixture\" is truly necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oU58Q4Z5cd", "reviewer": "Reviewer_51kq", "summary": "This paper proposes Mixture-of-Memories (MoM), a new architecture designed to improve the performance of linear sequence modeling methods such as linear attention, state space models, and linear RNNs. The key idea is to maintain multiple independent memory states instead of a single fixed-size state, with a router network assigning input tokens to specific memory slots. This design aims to enhance memory capacity and reduce interference between token representations, addressing a common limitation in existing linear models when handling recall-intensive tasks. MoM is presented as a general and flexible framework that can be integrated with various linear memory update mechanisms. The authors report that MoM achieves better results than prior linear sequence models across multiple language tasks—particularly those requiring long-term recall—and approaches the performance of Transformer-based architectures, while preserving linear training complexity and constant inference cost.", "questions": "1.\tHow is the optimal predefined number of subsequences/memories determined?\n\n2.\tHow is the best top-k parameter selected?\n\n3.\tThe intuition of using different memories to store different tokens is compelling. However, the pre-routing step before the input hidden representation is somewhat unclear. How can the router determine which memory to use without having access to the information stored in each memory, given that routing occurs before the key-value projection and memory update?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-written, with a clearly motivated method and strong experimental evidence to support its claims.", "weaknesses": "1.\tExperiments with larger model sizes could further strengthen the paper.\n\n2.\tSome hyperparameter choices—such as the top-k value in the routing process and the predefined number of memory slots—are not thoroughly discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)