Resona: Improving Context Copying in Linear Recurrence Models with Retrieval
Authors: Xinyu Wang
Linrui Ma
Jerry Huang
Peng Lu
Prasanna Parthasarathi
Xiao-Wen Chang
Boxing Chen
Yufei Cui
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Linear Recurrent Models
In-Context Learning
Retrieval
TL;DR: We improve the in-context performance of linear recurrent models by augmenting them with a parallel cross-attention branch that can mix in information from the context.

Abstract: Recent shifts in the space of large language model (LLM) research have shown an increasing focus on novel architectures to compete with prototypical Transformer-based models that have long dominated this space. Linear recurrent models have proven to be a viable competitor due to their computational efficiency. However, such models still demonstrate a sizeable gap compared to Transformers in terms of in-context learning among other tasks that require recalling information from a context. In this work, we introduce __Resona__, a simple and scalable framework for augmenting linear recurrent models with retrieval. __Resona__ augments models with the ability to integrate retrieved information from the provided input context, enabling tailored behaviour to diverse task requirements. Experiments on a variety of linear recurrent models demonstrate that __Resona__-augmented models observe significant performance gains on a variety of synthetic as well as real-world natural language tasks, highlighting its ability to act as a general purpose method to improve the in-context learning and language modelling abilities of linear recurrent LLMs.

Published as a conference paper at COLM 2025
RESONA: Improving Context Copying in Linear Recurrence Models with Retrieval
Xinyu Wang
Linrui Ma
Jerry Huang
70* Peng Lu Boxing Chen Yufei Cui
Prasanna Parthasarathi
Xiao-Wen Chang
McGill University
Noah's Ark Lab, Montreal
Université de Montréal O Mila
Abstract
Recent shifts in the space of large language model (LLM) research have shown an increasing focus on novel architectures compete with proto- typical Transformer-based models that have long dominated this space. Linear recurrent models have proven to be viable competitor due to their computational efficiency. However, such models still demonstrate sizable gap compared to Transformers in terms of in-context learning among other tasks that require recalling information from context. In this work, introduce RESONA, a simple and scalable framework for augmenting lin- ear recurrent models with retrieval. RESONA augments models with the ability to integrate retrieved information from the provided input context, enabling tailored behavior diverse task requirements. Experiments on a variety of linear recurrent models demonstrate that RESONA-augment models observe significant performance gains on a variety of synthetic as well as real-world natural language tasks, highlighting its ability to act as a general purpose method improve the in-context learning and language modeling abilities of linear recurrent LLMs.
1 Introduction
Improvements in building state-of-the-arr large language models (LLMs) (OpenAI, 2024; Grattafiori et al., 2024; Qwen, 2024; Gemma leam, 2024) through increased scale (Chung et al., 2024; Kaplan et al., 2020) and downstream tuning (Ouyang et al., 2022; Dubois et al., 2023) have enabled them to attain human-level performance on a number of complex tasks. One feature that has enabled this is in-context learning (Brown et al., 2020), where models can use essv-drovided content to provide a specific response tailed to that example. This relies on the Transformer (Vaswani et al., 2017) backbone that underlies many of these models, enabling for models to observe the complete past when generating content.
Recently, linear recurrent models (LRMs) (Gu al., 2022; Peng al., 2023; Orvieto al., 2023; Gu and Dao, 2024; Yang et al., 2024b) have emerged as an alternative, aiming to address computational bottlenecks associated with attention mechanisms (Bahdanau al., 2015). Unlike Transformers, LRMs do not operate over all previous parts of the input. Instead they compress prior context into a unified hidden representation/staa state with recurrent structure, similar to original recurrent neural networks (RNNs) (Rumelhart, 1989; Hochreiter and Schmidhuber, 1997; Cho et al., 2014; Balduzzi and Ghifary, 2016; Lu et al., 2019; Martin and Cundy, 2018), enabling more efficient inference. However, this unified representation introduces an information bottleneck, as it limits the capacity to represent the full range of vocabulary elements within fixed-size state that does not scale with the combinatorial complexity of token sequences. This has raised questions about the ability of LRMs to effectively learn from input contexts (Jelassi et al., 2024; Park et al., 2024; Grazzi et al., 2024; Lu et al., 2023) and perform comparably to Transformer-based LLMs in such settings.
"Equal Contribution
I 'Corresponding author: yufee..cii@uuaaei.ccmm

---

Published as a conference paper at COLM 2025
Onee time, land
Onca upon
a land
away [..] Alice apples. [_] Bob has
away, [-]] Alice has apoles, [_]]
Chunk Contest
pisars, Charlie
apples. Afcee
pears. Chair la apples. [..]
has oranges. [.]] Bob has apoles. [-]
has 5 oranges [..] Bob has apples. I.J]
How many apples do Alice and Cherfe
How many apples do Alice and Charlie
Alice has 2 aeples.
have?
have?
many apples
Alice Charlie have?
Disk Vsabucton
Encade Chanks
Encoder
Integrate Knowledge
LRM
Model
Retriever Search For Relevant Conteets
Ri
Concents
Contect Chunk Embesldiggs
apples and Charlehes apples.
Alice apples Charle has 4 apples.
Ouppu
Figure 1: Simplified overview of RESONA. The input is separated into the example-specific context along with a tasksspecific instruction. The context is chunked and encoded, after which retriever uses the instruction to determine which chunks are relevant to solving the task. The retrieved context is integrated into the model's reasoning for improved response.
Linear methods rely on fixed-size states to address limitations of traditional attention. A trade-off, however, is that these fixed-size states cannot perfectly preserve all historical infor- mation, making exact retrieval challenging. This manifests itself in practice in various tasks such language modeling; with the key-value associative memory system that underlies such methods, adding new key-value associations leads to accumulating retrieval errors that hinder performance. These errors may build up in variety of data-independent (Gu et al., 2022; Orvieto et al., 2023) or dependent (Yang et al., 2024a; Gu and Dao, 2024; Peng et al., 2024; Zhang et al., 2024; Beck et al., 2017) manners. While recent works have proposed strategies to mitigate such errors (Yang et al., 2024b; 2025; Sun et al., 2024), they retain fixed-size hidden state that remains a fundamental constraint.
In an effort to bridge this gap, we propose RESONA, retrieval-based method designed to improve context-based learning LRMs (Figure 1). By introducing a retrieval mecha- nism that facilitates information flow from the context, RESONAmitigates the hidden state bottleneck and enables more effective in-context learning. Specifically, we augment LRM layers within the backbone model with contextual search component. The input context is first chunked into passages, which are then retrieved based on the current LRM state. A knowledge integration module subsequently incorporates the retrieved passages into the model's 's output by directly modifying its representation. This architecture allows previous context to bypass the fixed-size hidden state constraint, improving information flow from context to generation. These processes (chunking, retrieval, and integration) are parallelized with the main LRM, ensuring easy adaptation and improved in-context learning across variety of models.
Empirical results on a range of representative tasks, spanning both synthetic and real- world data, demonstrate that RESONA significantly improves the ability of LRMs to utilize context-specific information with minimal or no latency overhead. We evaluate RESONA on diverse set of tasks, including synthetic retrieval and recall tasks, language modeling and question-answering tasks, across multiple scenarios such as pre-training and direct fine-tuning. Our analysis demonstrates the effectiveness of using RESONA for overall performance improvements as well as test-time model customizations, such as balancing the trade-off between efficiency and performance.
2

---

Published as a conference paper at COLM 2025
2
Related Works
Linear Recurrent Models. Despite vast improvements in building language models that solve real-world natural language tasks since the introduction of the Transformer (Vaswani et al., 2017), significant concerns remain about their efficiency and scalability. While this has spurred interest in rendering them more efficient (Katharopoulos et al., 2020; Dao et al., 2022; Yang et al., 2024a), LRMs (Gu et al., 2022; Orvieto et al., 2023; Qin et al., 2023; Gu and Dao, 2024; Dao and Gu, 2024; Peng et al., 2023; Sun et al., 2023; Lu et al., 2025; Yang et al., 2024b) have grown a popular alternative due to highly efficient inference costs compared to attention-based alternatives while retaining the ability to be train on elements of a sequence in parallel, an issue with traditional recurrent models. Further attempts at leveraging advantages from both paradigms (Lieber et al., 2024; De et al., 2024; Dong et al., 2025) have also garnered interest, leading to further exploration of similar models.
Serrivval-Augmetter Generation (RAG). RAG-based methods augment the input of an LM with passages retrieved from outside source (Guu et al., 2020; Lewis et al., 2020). Such methods have significantly improved performance on knowledge-intensse is difficult to store the information required for strong performance explicitly within the model's 's parametric knowledge (Roberts et al., 2020). Further improvements have come under the form of improved filtering of retrieved passages (Asai et al., 2024; Ma et al., 2025), robustness to irrelevant passages (Yoran et al., 2024; Xu et al., 2024) or tuning of more components (Lin al., 2024). However, RAG is not directly applicable to learning from contexts, as such methods do not search within the input-specific query but rather from an external database, leading to a critical failure point of such methods.
tasks, where it
Linear Recurrent Models and Context Usage. Despite their practical benefits, questions have arisen regarding the ability of LRMs to learn from input contexts (Akyürek et al., 2024). Jelassi et al. (2024) show that they struggle to directly copy information from contexts due to their fixed-sized latent state. Park et al. (2024) further show that they can struggle at retrieval-based (Arora et al., 2024) in-context learning, only solving such tasks through the addition of attention. Such observations have extended to real-world data, where LRMs have been shown to exhibit similar difficulties as Transformer LLMs (Wang et al., 2024; Huang, 2025; Liu et al., 2024; Huang et al., 2025) for long contexts (Ivgi et al., 2023; Hsieh et al., 2024). Accordingly, we introduce RESONA additional information flow paths from the context to the generated input, enabling better utilization of the context for problem-solving.
potential solution that provides
Mehory-enhanced Transformers. Due the quadratic complexity of self-attention with respect to the length of sequence, Transformer models face significant computational challenges when processing long inputs. Numerous approaches have been proposed to enhance Transformers for long sequential data, both to reduce the time complexity of the models as well as improve performance. Dai et al. (2019); Munkhdalai et al. (2024) segment long inputs into shorter sequences and process them recurrently. Mohtashami and Jaggi (2023) append landmark tokens to represent each block of input and uses group attention to select relevant information. Borgeaud et al. (2022) enhance Transformer perfor- mance with external data by leveraging a separated retriever module. Nunez et al. (2024) develop Span-Expanded Attention for the hybridized attention model to retrieve the most relevant block and integrate it with the recent context for attention computation. However, it remains unclear whether and how retrieval-basett modules can enhance the performance of generalized linear recurrent models such as GLA (Mao, 2022; Yang et al., 2024a; Lu et al., 2025), Mamba (Gu and Dao, 2024; Dao and Gu, 2024), RWKV (Peng et al., 2023), and DeltaNet (Schlag et al., 2021; Yang et al., 2024b).
3 RESONA
We introduce RESONA (Figure and Algorithm 1) as a framework to enhance the context- copying ability of LRMs through retrieval, without sacrificing original performance and
3

---

Published as a conference paper at COLM 2025
versatility. Our end-to-end training enables models to utilize the context as retrieval base from which information can be extracted. This helps the model to first overcome the fixed-size latent space bottleneck by integrating information from the context directly into the hidden state. This is contrast to traditional LRMs, where the information from the context must flow through the hidden state.
Algorithm RESONA Algorithm.
Require: Model M
1: Input: Input $ E RD and LRM hidden
states H RLLXxx
Renwort up
LiVe
Output: RTxD
3: Embed sequence into X E RTTD
HM Lape
Krocledge Inspraio
Contunal Searc
Resonnll ayyt
Hole
4: for 1 L do
Netteck
Outcut
5:
H + H; > Hidden state of layer
Nayorn
6: if Layer is an RESONA A layer then
7:
(H, Y"T) = Mj RM (X
8:
& LRM Output
9:
M M S (x, H)
Figure 2: Primary components of RESONA: 1) Contextual Search which searches the context for relevant information and 2) Knowledge In- tegration which re-integrates the retrieved in- formation into the model state. These enable information from the original network input to flow to arbitrary depths in the model, overcom- ing information decay within the model.
10:
Chunk-and-Searell
11:
Y MKI (X, H,M M)
12:
Knowledge Integration
13:
a fXX)
14:
Y a Y =a)Y"
15:
else
16:
H, Y) M₁ (X
17:
X
18:
return
3.1
Problem Formulation and Overview
Let V be vocabulary, i.e. set of discrete elements, of size [V.. A model M applies V* a function f :v + V", taking as input a sequence of tokens from the vocabulary (of
arbitrary length) while outputting sequence of tokens from the vocabulary. We denote the input sequence as x = xr,,, which we refer to as the model's prompt. The corresponding output sequence y = =xxx) is referred to the model's answer or generated response. Furthermore, a equueeceto--okkn mapping is function g V - V used to define f through auto-regressive inference. Specifically, given input sequence x E V*, the output tokens are generated one at a time using the recurrence: Xi+j =
xi+j-1)
and f (x1.)= (x,+1, X,42 where < i aadd] j N.
An LRM defined by a state update rule u x V-V+ - and an output function r S S - V, where S is finite set of states and a state is representation of the system after processing sequence from V* Let 5p E S be some initial state. Given some sequence x of length L, the state of the model at iteration i is denoted by S; x;) and the output token is denoted by (, x;). These are defined recursively as:
1) So (0) = so,
2) h; S,(x1,
...,...... x1) (
3) y1 = R (xI,...,,, = (x
Information from flows y through a state IR where d, is fixed and finite. Thus for increasing sequence length information dense settings, LRMs can struggle from the limited size of
We observe that the LRM is directly limited by the size of its hidden state, which can be insufficient for modeling problems with many possible states, unless the hidden size grows with the size of the possible set of states. RESONA introduces two flexible components to overcome this constraint without sacrificing the primary benefits of LRMs (namely parallel training and inference time efficiency): contextual search operation that operates the
4

---

Published as a conference paper at COLM 2025
input to retrieve context-specific information and 2) a knowledge integration component that mixes the retrieved information with the LRM output.
Algorithm Chunk-and-Search Algorithm.
Encoder
Contect
Require: Context and Query Encoders C, Q
1: Input: Input X RTxD and LRM hidden
state H E RTXH
2: Output: Attention mask M R
3: Chunk X into X' C RRNUDD
Chunk-and Search
Search
4: Use C' to encode each context chunk {X
X' into context embeddings Č R /i=1
5: Use Q encode H into query embeddings
Qie...
Q E RTxE
6: Compute chunk index sets for each Qf
Knowlddas Integnation
Query Encoder
(1) = (Top-k( Qq. C
7: With U/)/-1' compute mask M E R
Liyy
such that M₁;- =
Figure A breakdown of our Chunk-and-Searee implementation. The initial input context is chunked while the hidden state of the LRM layer is used a query. Corresponding indices are retrieved for each query, creating a mask that used for Knowledge Integration.
Contextual Search. order to retrieve relevant context, RESONA implements contextual search as chunk-and-sean mechanism (Algorithm 2). The initial input X is first split into N chunks, each consisting of U tokens, to create X' E NxUxD where D is the model input dimension. First, each of these chunks C, is encoded using a context encoder C into a context embedding C,. Simultaneously, the hidden state E R of the adjacent linear-recurrent layer, is used to encode a number of queries into query embeddings Qu. using query encoder Q. For each query, search for the top-k similar contexts using a cosine-similarity search, which produces chunk indices that we can then use to retrieve the relevant input token positions. These are used to create mask, which is passed to the phowleege-Integratonn module.
Knowledge Integration. To integrate knowledge from the retrieved chunks, RESONA does as follows (Figure 4 and Algorithm 3). The knowledge integration module cross- attention module that can directly integrate information from the initial embedding into the LRM layer representation. To do so, the queries Q are directly computed from the hidden state of the prior LRM layer', while the keys K and values are computed directly from the input embeddings X that directly follow after the initial embedding matrix E. Within the cross-attention module, we use a mask computed from our Chunk-and-Searce implementation of the contextual search. This ensures that the cross attention module can mix in only the most relevant information from the input back into the cross-attention module, producing an output Y" E R TxD,, which is then integrated with the output of the adjacent LRM layer Y"", computed as
Y = a Y" + (1 a). Y".
& can be computed on an input-dependent basis for each element of Y or can alternatively be set as fixed hyper-parameter value. Because only the chunk-and-search design, each query attends to at most ku elements from the initial input in k contiguous blocks, enabling us to compute Y" efficiently using existing sparse attention mechanisms. For simplicity, we maintain the use of fixed constant for the experiments that follow.
Inn the event that the first layer is augmented with RESONA, the queries are generated directly from the initial embeddings.

---

Published as a conference paper at COLM 2025
Algorithm 3 Knowledge Integration.
Require: Attention weights Wo. Wx, Wy and output weights Wout
Layer 1-
1: Input: Input X R and LRM hidden
state h E R R mask M E RRTxT
Output: Y E R
3: From It compute queries Q C R using
K
Wo- In parallel, compute K, V E IR Txd, from X with Wx, Wy.
Adtetion
Compute multi-head attention output O CrossAttn (Q.K, V, M), where O DER Txxe
5: Project O using Wout into Y" & RTxD.
Figure 4: Knowledge Integration portion of RESONA. The hidden state is used as a query while the initial network input used the query and keys. A mask constructed from the contextual search is used to determine which information needs to be mixed back into the LRM representation, which then used to compute an attention output that is integrated into the adjacent LRM output.
3.2 Training and Inference
When training RESONA, an important consideration is the auto-regressive nature of the model that therefore requires causal mask. Specific to our implementation, to maintain this during the parallel nature of training, we ensure that for the query at index i, only chunks that solely contain information prior this position in the input are considered within the Chunk-and-Seane process. This ensures that the mask M allows no information from a given token affect the representation of those ahead of it in the sequence.
During inference, tokens are dynamically chunked based on predefined size and embed- ded into chunked cache in parallel with the main model's embedding, introducing no extra latency. For long prompts, chunking is integrated into the pre-filling stage, aligning with the token embedding pipeline and minimizing computation overhead.
4 Experiments, Results and Analysis
4.1 Tasks and Datasets
To test our method, we evaluate RESONA on both number of synthetic benchmarks as well as real-world language benchmarks. In Section 4, we explain the experimental setting as well as evaluation methods for each. For each setting, we report standard baseline where a backbone model is not augmented with RESONA. These baselines vary based on which backbones are capable of adequately learning the task without RESONA.
4.2 Main Results
4.2.1
Results Synthetic Benchmarks.
We first evaluate on synthetic benchmarks, namely multi-query associative recall (MQAR) (Arora et al., 2024) and the Mechanistic Architecture Design (MAD) suite of tasks (Poli et al., 2024). For each, we report accuracy on held-out test set, where a correct answer requires the entire output is being correctly predicted. We initialize models from scratch and train them on the task of interest, in particular a layer model with a vocabulary size of 8192. Each model uses a hidden size of 128 and context chunk size of 64 for those augmented with RESONA. Models are trained using 20K and evaluated on 1K examples.
Figure 5 and Table demonstrate that RESONA A augmentations improves performance across all baselines, some by wide margins, Baseline models are often able to perfectly solve MQAR for shorter sequence lengths or smaller number of KV-pairs, but fail catastrophically upon

---

Published as a conference paper at COLM 2025
BaseConv
Hyena
1.00
0.25
Accuracy
Mamba
Mamba2
RWKV
1.00
0.50
0.25
100
500
cequence Length
HiddenSize
Type
Results on MQAR tasks on varying sequence lengths. Baseline models remain limited in their ability to model increasingly long sequences even with increasing hidden size, whereas augmentation with RESONA perfect performance on arbitrary lengths.
Figure 5:
Table 1: Performance on synthetic MAD (Poli et al., 2024) tasks. The best result for each metric is highlighted in bold. RESONA consistently boosts retrieval performance and shows gains even on compression and memorization tasks.
Model
Comp.
ICR
Noisy ICR
Fuzzy ICR
SC
Mem.
Average
Transformer
53.4
96.7
99.7
69.4
98.7
89.4
84.1
Mamba
38.3
76.7
74.9
9.3
33.2
88.5
53.5
+RESONA
38,2 (1.3))
99.9 (123.))
100.0 (125.))
63.4 (t 541)
42.7 (19.)
88.8 (1
72.1 it 1866
Mamba2
43.6
96.4
96.7
21.1
93.3
86.9
73.0
+RESONA
46.6 (1)))
100.0 (*5.))
100.0 (1
62.9 (t 41.8
93.6 (*)
88.1 (.))
81.9 (t 8:9)
RWKV5
36.8
96.4
96.6
12.1
52.7
55,0
58.3
+RESONA
40.4 (1.34)
99.7 3:3)
99.8
59.7 47.6)
58.0 (.5.)
70.6 (5
71.5
Hyena
42.2
79.3
77.4
9.96
72.8
88.9
61.7
+RESONA
42.6 (1:44)
99.9
99.9 (1:22.)
66.2
56.2)
74.3 (-1)5)
89.0 (1))
78.7 (1:100)
increasing either of these values. ResONAmeanwhile retains near perfect accuracy even after these values. Similarly, models can struggle at specific tasks within the MAD suite, but RESONA achieves a significant improvement in performance. On tasks in which the base LRM models show strong performance, no degradation is observed through the addition of RESONA, highlighting its flexibility. We observe that this is consistent across multiple models (Poli et al., 2023; Fu et al., 2023), which are incapable of learning some of the simpler settings but upon the addition of RESONA layers are able of maintaining nearly perfect accuracy for arbitrarily long sequences, showcasing its specific benefits in context- recall intensive settings.
4.2.2 Language Modeling
To assess language modeling, we compare a baseline models with one augmented with RESONA layers. Here, we train on the WIKITEXT- dataset (Merity al., 2016). In order to train models augmented with RESONA, we also augment the dataset. Specifically, we first consolidate all samples from the same Wikipedia entry into single sample, eliminating excessively short title lines empty lines. We then use a LLaMA3. 1-70B (Grattafiori et al., 2024) model to augment each sample such that we can make use the RESONA retrieval mechanism. We then conduct the Chunk-and-Searrd process offline to create masks, in order to save computation during training. To account for the additional parameters introduced by RESONA, present results in the baseline settings for two version, one where each
7

---

Published as a conference paper at COLM 2025
Table 3: Results on QA benchmarks, where augmentation with RESONA improves perfor- mance over all evaluation metrics. Hymba indicates RESONA is added as third branch with both the Hymba's ss original linear and attention branches, while Hymba(256) indicates where we reduce the window size of Hymba's sliding window attention from 1024 to 256, in order to create a fair comparison with our chunk size of 256.
Model
TRIVIAQA
CoQA
NARRATIVEQA
BLEU
Rouge-L
Meteor
F1
BLEU
Rouge-L
Meteor
F1
BLEU
Rouge-L Meteor
F1
Mamba
28.3
66.0
44.8
40.2
35.5
73.0
49.3
60.6
12.6
47.0
34.7
34.7
RRESONA
30.7
68.0
45.6
41.8
44.3
75.0
52.2
61.2
13.2
50.0
36.9
36.2
Hymba(256)
15.3
58.0
41.4
34.3
31.8
62.0
42.1
50.0
10.7
39.0
29.1
28.4
w/ w/RETRIEVAL
2,9
57.0
39.0
33,9
4.6
52.0
38.2
41.3
3.9
36.0
30.8
27.6
w/RAG
12.9
66.0
46.0
38.8
23.2
67.0
47.9
54.5
12.5
49.0
38.4
35,2
RESONA
43.4
41.5
69.0
13.1
46.0
Hymba
16.5
64.0
44.8
36.9
40.0
77.0
53.3
63.1
20.5
39.0
44.2
43.1
+RESONA
60.0
LRM layer matches exactly that of the augmented model as well as version where the hidden size of the layer has been increased to match the parameter count of the augmented counterpart. Results in Table 2 demonstrates that integrating RESONA consistently achieves lower perplexity than baseline counterparts and their parameter-aligned variants, highlight- ing its applicability for language modeling. Notably, modifying Hymba's sliding window mask to our retrieval-based mask significantly improves performance. Furthermore, results on short-context tasks (Table 8) demonstrate no performance degradation, suggesting the ability to maintain performance on tasks that are not recall intensive.
4.2.3 Direct Fine-Tuning
To understand how well the addition of ReS- ONA modules can improve performance on context- dependent tasks, we make use of pre-trained models in which we insert RESONA layers. The models are then fine-tuned directly, as described in Section 4.1, and we record performance using task-specific met- rics. Due some computational limitations, we provide results only for models where the baseline model capable of performance above random on all benchmarks. In these settings, we choose 3 layers of the model to augment with RESONA. The mod- els are then tuned using the corresponding training dataset of the task, before being tested on a held-out test set.
Table 2: Comparison of pre-training perplexity (PPL) on WikiText-103
across base LRM architectures, their parameter-aligned variants, and RES- ONA-enhanced versions.
Model
Param
PPL
GLA
131M
14.265
GLA (SP)
142M
14.223
+RESONA
145M
13.892
DeltaNet
131M
13.044
DeltaNet (SP)
142M
12.946
+RESONA
145M
12.541
RetNet
132M
15.471
Under this direct-fine-tuning setting, we evaluate on question answering benchmarks including NARRA- TIVEQA (Kocisky et al., 2018), the Conversational Question Answering (CoQA) challenge (Reddy et al., 2019) and TRIVIAQA (Joshi et al., 2017). For each task, we report results in terms of BLEU, ROUGE-L, Meteor (Banerjee and Lavie, 2005) and F1 scores. Table presents these results, where we can observe improvements on both pre-trained Mamba and Hymba (with a sliding window 256) models through the addition of RESONA layers. This is particularly evident with improvements across all metrics for all datasets, showing the general benefits that RESONA provides towards better context-dependent reasoning skills.
RetNet (SP)
146M
15.431
+RESONA
142M
14.742
Mamba
140M
16.261
Mamba (SP)
157M
16.173
+RESONA
154M
15.943
Hymba(64) + +RESONA(SP)
133M
16.688
135M
15.887
8

---

Published as a conference paper at COLM 2025
5 Analysis and Discussion
RESONA vs. RAG-like methods. A natural comparison to make with RESONA is typical RAG-like methods, which append the retrieved passages directly to the input prior to applying the model. To compare with such methods, we design the following methods with RESONA: Using the input context directly as the data-store from which a pre-trained retriever can retrieve passages directly that are used as context (RETRIEVAL) and 2) we remove the Knowledge Integration component and instead directly append all (decoded) retrieved passages from the Chunk-and-Searry procedure to the initial input (RAG). Rows 5 and in Table shows these results on a Hymba model, again on the tested QA datasets. variant remains significantly more performant than the other alternatives, demonstrating the benefits that acting directly on the representations can have.
The
Relationship with Hybrid Methods. Some previous works have suggested meth- ods that enable linear recurrent models to improve their in-context learning abilities. Park et al. (2024) introduce a MambaFormer architecture which interleaves self-attention and Mamba layers, improving the ability to learn in-context on tasks in which pure Mamba models struggle. Similarly, RES- ONA can be interpreted as a form of hybrid mixture of attention and recurrence, similar to Dong et al. (2025), with the difference lying in the frequency of attention and its sparsity within different layers. To better compare these two methods, we provide an ablation (Table 4) where we replace Hymba layers with RESONA. A. In this setting, 3 consecutive layers are trained in either setup. We observe a meaningful increase in performance on CoQA, indicating that for such types of tasks, the mechanism presented by RESONA could be more robust and suitable a number of real world tasks.
Table 4: A comparison of RESONA and Hymba (Dong et al., 2025). Three consecu- tive layers are un-freezed for fine-tuning. In the TRSONA-augmente branch of the selected layers is modified to the RESONA retrieval mechanism. A slicing window size of 256 is used while the corre- sponding number indicates the start index of the replaced layers.
setup, the attention
COQA
Model
Layer
BLEU
Rouge-L
Meteor
F1
Hymba
31.8
62.0
42.1
50.0
+RESONA
36.5
69.0
48.8
57.2
Hymba
25.1
62.0
42.5
48.8
+RESONA
51.5
Hymba
21.8
60.0
41.5
48.0
15
+RESONA
37.1
70.0
48.9
57.7
Hymba
5.9
41.0
27.8
32.3
+RESONA
29.1
62.0
42.8
50.1
Hymba +RESONA
5.8
40.0
27.6
30.3
29
31,2
57.0
39.2
Ablating on the position of layers. Due to its nature, a natural question that emerges re- lates to the ease and effectiveness of determining the layers at which RESONA augmentations need be made. The same results (Table 4) show that the placement of the 3 RESONA layers do not have a significant impact on the performance improvement relative to the baseline, which can be improved upon in nearly all ways in which the layers are selected. This highlights a level of robustness of the framework and method, hinting towards an ability to be used for a variety additional tasks. This robustness suggests that the method is not overly sensitive to architectural fine-tuning, reducing the need for extensive hyperparameter optimization when integrating RESONA into existing models. Such flexibility is particularly advantageous in practical applications, where manual layer selection may be non-trivial or computationally expensive.
Efficiency Evaluation. Given the architectural modifications that following from the RESONA augmentations, we provide a comparison between the efficiency tradeoffs with backbone models. We conducted ablation experiments using Mamba, DeltaNet and GLA, comparing both backbone models and those augmented with RESONA to demonstrate these tradeoffs in Table 5-7. While the RESONA augmentations do lead to marginal increase in each factor, the plug-in remains lightweight and does not add significant overhead in computation, particularly for longer sequences. Furthermore, direct comparison with a Transformer shows this method to be significantly more lightweight while retaining Transformer-like performance our tasks.

---

Published as a conference paper at COLM 2025
Pre-filling Length
Transformer
Mamba
Mamba Resona
DeltaNet
DeltaNet Resona
GLA
GLA Resona
2k
29
45
52
53
64
37
46
88
59
8k
149
170
72
103
62
104
16k
173
294
349
106
181
109
202
32k
503
571
653
208
338
202
386
64k
1665
1118
1285
412
652
407
757
128k
6094
2257
2412
807
1289
806
1518
Table 5: The time (in milliseconds) used to pre-fill a context of a pre-specified length. Numbers are rounded the nearest millisecond.
Pre-filling Length
Transformer
Mamba
Mamba Resona
DeltaNet
DeltaNet Resona
GLA
GLA Resona
2k
2679
2972
3329
2945
3686
2749
3528
2758
3044
3360
3042
3772
2777
3543
8k
2866
3155
3499
3023
3789
2774
3523
16k
3389
3164
3613
3057
3829
2869
3689
32k
5759
3491
3910
3080
4207
2912
4044
4656
3171
4352
128k
24050
4747
3601
3611
5278
3509
5076
Table 6: The time (in milliseconds) taken to generate 128 tokens following prespecified pre-filling length. Numbers are rounded to the nearest millisecond.
Pre-filling Length
Transformer
Mamba
Mamba Resona
DeltaNet
DeltaNet Resona
GLA
GLA Resona
3.6
4k
3.5
2.8
4.1
3,0
4,9
3.0
3.7
16k
6.4
3.7
5.1
3.7
5.5
3.6
4.4
6.5
5.4
64k
17.7
7.2
9.2
6.4
8.4
5.9
7,3
14.7
11.7
Table 7: The total memory consumption (in GB) given pre-specified pre-filling length.
6 Conclusion
In this work, we propose RESONA, a lightweight retrieval-based knowledge integration mechanism that significantly improves the ability of LRMs use example-specific context. RESONA utilizes novel mechanism to use the input-context as a retrieval data-store and integrate such information with the input during training and inference, enabling models to use it more effectively and overcome information bottleneck concerns. Across number of both synthetic and real-world datasets, LRMs augmented with RESONA demonstrate significant performance gains compared to their base counterparts, demonstrating its ability to function as general method applicable to broader scenarios.
10

---

Published as a conference paper at COLM 2025
References
E. Akyürek, B. Wang, Y. Kim, and Andreas. In-context language learning: Architectures and algorithms. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL Forum?2d=329cRr55rt.
net/
S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré. Zoology: Measuring and improving recall in efficient language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: https://openrevieee wtwp://nwwiee.eeeoorummmmmm....a
A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openrevieww wwttfooutt litef
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL / ////abs/1409 0473.
D. Balduzzi and M. Ghifary. Strongly-typed recurrent neural networks. In M. F. Balcan and K. Q Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1292-1300, New York, New York, USA, 20-22 Jun 2016. PMLR. URL wttpttt:tt/pseeeeeeeeeeeee v48/balduzzi 16. html.
mlr, press/
S. Banerjee and A. Lavie. METEOR: automatic metric for MT evaluation with improved correlation with human judgments. In J. Goldstein, A. Lavie, C. Lin, and C. R. Voss, editors, Proceedings of the Workshop Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarizatinn@AA. 2005, Ann Arbor, Michigan, USA, June 29, 2005, pages 65-72. Association for Computational Linguistics, 2005. URL htttps://acaantteee W05-0909/.
org/
M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brand- stetter, and S. Hochreiter. xlstm: Extended long short-term memory. In Advances in Neural Information Processing Systems 38: Anmual Conference Neural Information Processing Systems 2024, December 10-15, 2024, Vancouver, Canada, pages 5998-6008, 6008, 2017.
S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. lrving, O. Vinyals, S. Osindero, K. Simonyan, J. W. Rae, E. Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. In K. Chaudhuri, S. Jegelka, L. Song. C. Szepesvári, G. Niu, and S. Sabato, editors, International Confer- ence on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240. PMLR, 2022. URL https://prroeee //pp:////ooocc/ngss.. mlr pttppe//vll//bbrgeaudda.. html.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler,
Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL hath/1157cdddFFcc454/41118 8bt.e6/c77668888811118 42eee44A44664aaaaaaaaaaaa html.
www.ddiiinnnnnnnninnnng neurips. cc/ cc/paper/2020/
K. Cho, B. van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical
11

---

Published as a conference paper at COLM 2025
machine translation. In A. Moschitti, B. Pang, and W. Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qutar, A meeting of SIGDAT, Special Interest Group of the ACL, pages 1724-1734. ACI, 2014. doi: 10.3115/V1/D44-117 URL https: org. 3 d14-1179.
H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S, Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Y. Zhao, Y. Huang, A. M. Dai, H. Yu, S. Petrov, H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling instruction-finetuedd language models. Journal Machine Learning Research, 25: 70:1-70:53, 2024. URL https://jml. org/ http//ppps////////2 23- 0870 html.
Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q V. Le, and R. Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In A. Korhonen, D. R. Traum, and L. Mårquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages Association for Computational Linguistics, 2019. doi: 10.18653 /V153///11991228 URL https://doi. hwedoo..o//..//// org/10. 18653/v1/p19--2255.
T. Dao and A. Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openree
met//forum1d===ttBFCRRR1
T. Dao, D. Y. S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http: / /pppprpappppppp/ppp/eee/eeeeessa nips. cc/ cc/paper_ hcc/paperffiless files/ wabsrract-Confeeeeeee
haper/2222/hash/555 http://hhhhh55577733322200000112
html.
S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y. Chen, S. Srinivasan, Desjardins, A. Doucet, D. Budden, Y. W. Teh, R. Pascanu, N. D. Freitas, and C. Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficient language models, 2024. URL https: / http///axxvvvo/oeeee wwtpxiwworr//b////22.. 19427.
X. Dong, Y. Fu, S. Diao, W. Byeon, Z. Chen, A. S. Mahabaleshwarkar, S.-Y. Liu, M. Keirsbilck, M.-H. Chen, Y. Suhara, Y. Lin, J. Kautz, and P. Molchanov. Hymba: A hybrid- head architecture for small language models, 2025. URL https: /ttpss///penrevview net new.f/ruettffmmm id=Alztozypga.
Y. Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember 10 - 16, 2023, 2023. URL http::/papperrr nips. wwtp::scssssccccccpceeepcccpcppe httpcccpppppp.cpeeeeeee wwtrrfflllssspeeeee haap3//s222//hahhh html. htmll.
D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Ré. Hungry hungry hippos: To- wards language modeling with state space models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: /ttpnr/iiee/.reie.... htt./formmiiioo ddwpmmmddodoDyywYG......
Gemma Team. Gemma Improving open language models practical size, 2024. URL https://arxiv.orr wwtp:/xwwwwwwg.///////e///// 0014008801111000 18.
A. Grattafiori et al. The llama herd models, 2024. URL 21783.
12

---

Published as a conference paper at COLM 2025
R. Grazzi, J. Siems, S. Schrodi, T. Brox, and F. Hutter. Is mamba capable of in-context learning? In Third International Conference on Automated Machine Learning, AutoML 2024, Paris, France, September 9-12, 2024. OpenReview.net, 2024. URL https: forum?id=r JhoGePpnr.
A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, COLM 2024, Philadelphia, PA, United States, October 7-9, 2024. OpenReview.net, 2024. URL https: htttp:://oppnreviieee wtt.ppeenreeeeeeeeeeruuuuumaaaea feee VY2.
A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: wwwperrreeie.eeefooumiiilie. uYLFozIvIAC.
K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR, 2020. URL http: http://pproeeednng mmr. www.ppess//vl.... guu20a. html.
S. Hochreiter and Schmidhuber. Long short-term memory. Neural Comput., 179,1731780 1997. doi: (0.116//NECO.1990 8.819.77....... URL /nttps::///ii http::::///doooor//////// 1735.
1rg/ 1t/////11111221 1162/neco. 1997.9.8
C. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. RULER: what's the real context size of your long-context language models? In First Conference on Language Modeling, COLM 2024, Philadelphia, PA, United States, October 7-9, 2024. OpenReview.net, 2024. URL https: /ttp////ppp/eeeeeeeeeeeeeeeeeeee
J. Huang. How well can a long sequence model model long sequences? comparing architech- tural inductive biases on long-context abilities. In International Conference on Computational Linguistics, COLING 2025, January 19 - 24 2025, Abu Dhabi, UAE, 2025.
J. Huang, P. Parthasarathi, M. Rezagholizadeh, B. Chen, and S. Chandar. Do robot snakes dream like electric sheep? investigating the effects of architectural inductive biases on hallucination, 2025. URL https: / https/////xxxvo//o//eeo
M. Ivgi, U. Shaham, and J. Berant. Efficient long-text understanding with short-text models. Trans. Assoc. Comput. Linguistics, 11:284-299, 2023. doi: 10.1162/TACL / .00547. .000547 URL
https://doi. org/10 0 1162/tacl 00547.
S. Jelassi, D. Brandfonbrener, S. M. Kakade, and E. Malach. Repeat after me: Transformers are better than state space models at copying. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:tt/ppenrwwweteee
M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer: Triviaqa: A large scale distantly super- vised challenge dataset for reading comprehension. In R. Barzilay and M. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL. 2017, Vancowver, Canada, July 30 August 4, Volume 1: Long Papers, pages 1601-1611. Association for Computational Linguistics, 2017. doi: https://doi.org: org/0/.......//.. 101...165///1/11-11111
URL
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020. URL https: //arxiv. wwrxxxvorg//bss/20/0 08361.
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th Interna- tional Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5156-5165. PMLR, 2020. URL http: http://prroceedings. mlr. htt.pprssstt11//aaaaappplllsss html.
13

---

Published as a conference paper at COLM 2025
T. Kociský, Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. The narrativeqa reading comprehension challenge. Trans. Assoc. Comput. Linguistics, 6:317- 328, 2018. doi: T01112//TACI, A) 00023, URL https://dol org/10.../.. 1162/tacl 00023.
H. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for NLP tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Decem- ber 6-12, 2020, virtual, 2020. URL https: http::::///rocceeeee http///poceeen..eeee///// wenneeurppp wwerips.cck wwtpcccpppppeee////// Jb493230285F780e110
knowledge-intensse
html.
O. Lieber, B. Lenz, H. Bata, G. Cohen, Osin, I. Dalmedigos, E. Safahi, S. Meirom, Y. Belinkov, S. Shalev-Shwartz, O. Abend, R. Alon, T. Asida, A. Bergman, R. Glozman, M. Gokhman, A. Manevich, N. Ratner, N. Rozen, E. Shwartz, M. Zusman, and Y. Shoham. Jamba: A hybrid transformer-mamba language model, 2024. URL 19887.
X. Lin, X. Chen, M. Chen, W. Shi, M. Lomeli, R. James, P. Rodriguez,) J. Kahn, G. Szilvasy, M. Lewis, L. Zettlemoyer, and W. Yih. RA-DIT: retrieval-augmented dual instruction tuning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.nres forum?id=220Tbutugg.
N. Liu, K. Lin, Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Lingwistics, 12:157- 173, 2024.
doi: (0.1162/TACCL _A|_00638 URL https: org/10. 1112/tacc...005533
Learning task-specific representations in multi-task learning for sequence labeling. In Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2396- 2406, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1 124 URL https: aac wwwclanhhloog..rrgg...o org/N11 9- 1249/.
Lu, T. Bai, and P. Langlais. SC-LSTM:
P. Lu, S. Wang, M. Rezagholizadeh, B. Liu, and 1. Kobyzev. Efficient classification of long documents via state-space models. In H. Bouamor, J. Pino, and K. Bali, edi- tors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2023, Singapore, December 6-10, 2023, pages 4559-6565 Association for URL
Computational Linguistics, 2023. doi: 10.18653 WV112023.EMNLP-MMM https://doi. org/10 /0. 18653/v1 /2023 w0t.....ml.mm.al..mmm emnlp-main.44 404.
P. Lu, I. Kobyzev, M. Rezagholizadeh, B. Chen, and P. Langlais. Regla: Refining gated linear attention. In L. Chiruzzo, A. Ritter, and Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Compulational Linguistics: Human Language Technologies, NAACL 2025 Volume Long Papers, Albuquerque, New Mexico, USA, April 29 May 4, 2025, pages 2884-2898, Association for Computational Linguistics, 2025. doi: 10.18653/ /V1.2025.NAACL-LOO.. naac1-long, 147.
org/10. 18653/v1/2025. URL https://doi. http:::///oiooor///////
Ma, Xu, X. Jiang, Li, H. Qu, C. Yang, J. Mao, and J. Guo. Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guider retrieval augmented generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevieet net net/forum?icnoo /ttf:/rrmmmiooo/////22000/
H. H. Mao. Fine-tuning pre-trained transformers into decaying fast weights. In Y. Gold- berg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 10236-10242. Association for Computational Linguistics, 2022. doi: 10.18653 10665//VI /V1 111//222.EMLLPMMAAII emnlp-main.63 697.
URL https: //doi. http//doooor///////////ooo org/10 18653/v1/2022.
14

---

Published as a conference paper at COLM 2025
E. Martin and C. Cundy. Parallelizing linear recurrent neural nets sequence length. In International Conference on Learning Representations, 2018.
S. Merity, C. Xiong, Bradbury, and R. Socher. Pointer sentinel mixture models, 2016.
A. Mohtashami and M. Jaggi. Landmark attention: Random-access infinite context length for transformers. CoRR, abs 0t////300.1630000 2023. doi: 10.48550 0035550/AAXIV2225516663000 URL https://doi org/10 / 48550/arXiv....
16300.
T. Munkhdalai, M. Faruqui, and S. Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. CoRR, abs/2404.07745. 2024. doi: 10.48550/ ARXIV. 2404.07143. URL https: org.0/..rgr...o 485000/45500//// 2404. 07143.
E. Nunez, L. Zancato, B. Bowman, A. Golatkar, W. Xia, and S. Soatto. Expansion span: Com- bining fading memory and retrieval in hybrid state space models. CoRR, abs 2242.11328, 2024. doi: 10.48550/ ARXIV.2412.13325 133. URL https://don org/10. 13328.
2412.
OpenAT. Gpt-4 technical report, 2024.
08774.
A. Orvieto, S. L. Smith, A. Gu, A. Fernando, Ç. Gülçehre, Pascanu, and S. De. Res- urrecting recurrent neural networks for long sequences. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 26670-26698 PMLR, 2023. URL https://proceedeee
mlr. www.mm................mmwewweeee
html.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: wwwersppppppppppp//pp/ nips. cc/ cc/paper files/ paper/2022 http:///22/aaaahhheeeeeeeeeeaaaa 4F50447779911558880001111111 Aattrr3tt--bbaaat----aooo coneerraact-oooffeennenee html.
J. Park, J. Park, Z. Xiong, N. Lee, J. Cho, S. Oymak, K. Lee, and D. Papailiopoulos. Can mamba learn how to learn? A comparative study on in-context learning tasks. In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: www.:/te//oommmiidofffffmm
B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, L. Derczynski, X. Du, M. Grella, K. K. GV, X. He, H. Hou, P. Kazienko, Kocon, Kong, B. Koptyra, H. Lau, J. Lin, S. I. Mantri, F. Mom, A. Saito, G. Song, X. Tang, J. S. Wind, S. Wozniak, Z. Zhang, Q. Zhou, J. Zhu, and R. Zhu. RWKV: rein- venting rnns for the transformer era. In H. Bouamor, J. Pino, and K. Bali, editors, Find- ings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 14048-14077. Association for Computational Linguistics, 2023. doi: URL httpp://doo hwaeddii..oggg///// org/10. 18653/v1/2023.
mininggs-emnlp.99. 936.
B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou, P. Kazienko, K. K. GV, J. Kocon, B. Koptyra, Krishna, M. Jr., N. Muennighoff, F. Obeid, A. Saito, G. Song, H. Tu, S. Wozniak, R. Zhang, B. Zhao, Q. Zhao, P. Zhou, J. Zhu, and R. Zhu. Eagle and finch: RWKV with matrix-valued states and dynamic recurrence. In First Conference Language Modeling, COLM 2024, Philadelphia, PA, United States, October 7-9, 2024. OpenReview.net, 2024. URL https: //openreview. www. httt//ooummid==so/EEtt
M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and
C. Ré. Hyena hierarchy: Towards larger convolutional language models. In A. Krause,
15

---

Published as a conference paper at COLM 2025
E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and Scarlett, editors, International Con- ference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28043 28078. PMLR, 2023. URL https:/ / http:////ooceeennee/e/ee . mlr. www...peessse2222 wtw.v/2222pplll httpp/ll...aaa... html.
M. Poli, A. W. Thomas, E. Nguyen, P. Ponnusamy, B. Deiseroth, K. Kersting, T. Suzuki, B. Hie, S. Ermon, C. Ré, C. Zhang, and S. Massaroli. Mechanistic design and scaling of hybrid architectures. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL htctp:://opemreiieww.r forum?id=GDp7Gyddnf.
Z. Qin, S. Yang, and Y. Zhong. Hierarchically gated recurrent neural network for se- quence modeling. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember - 16, 2023, 2023, URL http:: nips. ccppaper Filess/paper/2223/haat htmml.
Qwen. Qwen2.5 technical report, 2024. URL B
S. Reddy, D. Chen, and C. D. Manning, Coqa: A conversational question answering challenge. Trans. Assoc. Comput. Linguistics, 7:249-266, 2019. doi: 10.1162/TACL 00266. URL https: //doi. http///i..rrgg///// 1162/tacl 00266.
A. Roberts, C. Raffel, and N. Shazeer. How much knowledge can you pack into the parameters of language model? In B. Webber, T. Cohn, Y. He, and Y. Liu, edi- tors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2020, Online, November 16-20, 2020, pages 5418-5426. Association for Computational Linguistics, 2020. doi: 10.18653 (118653/V1// N1/2200.MM.LPMMAIN.... URL https://doi. org/10 10. 18653881
emnlp-main.49
D. E. Rumelhart. Parallel distributed processing, 9th Edition. MIT Pr., 1989. ISBN 026268053X. URL https: / h/www//wwwwwwwwwo worrd.cat...m www.ll.rg//.ooooooooooooooooo /rrg/occc/00445550
1. Schlag, K. Irie, and J. Schmidhuber. Linear transformers are secretly fast weight pro- grammers. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9355-9366. PMLR, 2021. URL http: http://pprceedinng. mlr. wwtppsseete//cclleeeeeeps html.
Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, Xue, J. Wang, and Wei. Retentive network: A successor to transformer for large language models. Xiv preprint ar wxXx:2307.086211 2023.
Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, and C. Guestrin. Learning to (learn at test time): Rnns with expressive hidden states, 2024. URL https: / http///axvvvv.///.//ee////e org/ 2407 04620.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I. Jones, A. N. Gomez, I.. Kaiser, and T. Polo- sukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annul Conference on Neural Information Processing Systems 2017, , De- cember 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL https: ptttps://proceedingss neurips. cc/ wwtpcccppp/ppeeeeee/ee
1400ee -al.aa44aa-bbbtrctttt html.
fbd053c c4a845aa
S. Wang, I. Kobyzev, Lu, M. Rezagholizadeh, and B. Liu. Resonance rope: Improving context length generalization of large language models. In ACL (Findings), 2024.
F. Xu, W. Shi, and E. Choi. RECOMP: improving retrieval-augmeened Ims with context compression and selective augmentation. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https///oppnrevieeeennt//frrrrrn
16

---

Published as a conference paper at COLM 2025
S. Yang and Y. Zhang. Fla: A triton-based library for hardware-efficient implementa- tions of linear attention mechanism, Jan. 2024. URL https://github..com cttp://..oooooooaaooooaooao- flash-linear-attentio
S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim. Gated linear attention transformers with hardware-efficcent training. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a. URL https://
S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim. Parallelizing linear transform- ers with the delta rule over sequence length. In A. Globersons, Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Ad- vances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December wwwee/faaeeeeeeeeeee//e html.
10 - 15, 2024, 2024b. URL http://papers... nips. d1333eae7236666 d
Yang, Kautz, and A. Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirfennth International Conference on Learning Representations, 2025. URL
https://openrevieer
O. Yoran, Т. Wolfson, O. Ram, and J. Berant. Making retrieval-augmenter language mod- els robust to irrelevant context. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreviers
Y. Zhang, Yang, R.-J. Zhu, Y. Zhang, L. Cui, Y. Wang, B. Wang, Shi, B. Wang, W. Bi, P. Zhou, and G. Fu. Gated slot attention for efficient linear-time sequence modeling. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, December 10-15, 2024, Vancouver, BC, Canada, 2024. URL
17

---

Published as a conference paper at COLM 2025
A
Additional Results
A.1
Detailed Pretraining Results
Table 8: Detailed ineeealattnnnnnnrreee evaluation results from pre-training.
Param
Wiki.
ARC-C
ARC-E
Hella.
OBQA
PIQA
PM
RACE
Wino.
AVG
ppl
a0c
305
3CC
accun
acc
20c_n
30c
300
305
a0c_n
00c
30C
30c
Dcc
GLA
14,265
0.1644
0.3683
0.2671
0.5392
0.5109
0.2364
0.5008
0.3259
14.223
0.2073
0.2701
0.234
0.5321
0.5141
0.358
0.3243
RESONA
105M
13,992
0.192
0.2517
0.3635
0.3253
0.2646
02716
0.174
0.72
0.5157
0.5261
0.352
0.2169
0.5249
0.3297
DellaNet
131M
13.044
00.2201
0.2628
0.2849
0.613
0.622
0.224
0.3174
(SF)
142M
12.946
0.2115
0.256
0.303
0.2845
0.2892
0,3633
0178
0.278
0.5306
0.5125
0.34
0.366
0.4728
0.3199
+RESONA
165M
12,541
0.2073
0.2119
0.2963
0,293
0.2686
0.757
0.155
0.272
0.5199
0,5283
0.304
0.2517
0.5012
0.3210
A.2 Detailed Supervised Fine-Tuning Results
Table 9: Results on Needle-in---Haystcck (NIAH) using a haystack of varying sizes. Models are scored on performance on a continuous scale from 0 (worst) to 5 (best). In all settings, there is single needle placed arbitrarily within the haystack. Different variants mean that the format of the needle or haystack changes, such as being a number, keyword or UUID
sequence. Here a x Ir denotes RESONA is trained with learning rate multiplied by a.
4K
8K
16K
Model
Setting
V1
V2
V3
MV
V2
V3
MV
V1
V2
V3
MV
Baseline
5.00
1.80
0.75
0.898
0.65
0.45
0.15
0.458
0.00
0.35
0.20
0.494
Mamba
20xlr
5.00
3.65
1.80
1.358
5.00
0.85
0.45
0.528
4.90
0.05
0.00
0.669
50xlr
5.00
3.00
1.10
1.267
4.95
0.60
0.20
0.533
4.00
0.05
0.00
0.550
Baseline
2.15
2.95
1.45
1.301
1.90
1.45
0.70
0.321
1.00
0.25
0.00
0.883
DeltaNet
20
5,00
4.10
0.60
0.876
5.00
2.60
0.80
1.453
5.00
0.60
0.10
0,973
50xlr
5.00
4.40
1.45
1.676
5.00
1.25
0.35
1.312
5.00
0.45
0.05
0.885
Basel ine
2,90
3.65
0.05
1.389
0.25
0.55
0.00
0.846
0.00
0.05
0.00
0.291
GLA
20xlr
2.55
3.90
0.10
1.310
0.30
0.50
0.00
0.930
0.00
0.10
0.00
0.578
50xlr
3.00
3.85
0.15
1.353
0.20
0.60
0.00
0.900
0.00
0.05
0.00
0.657
Table 10: Detailed Im-evaluation-harree evaluation results from supervised fine-tuning of different pre-trained LRMs. After undergoing the same SFT as the backbone models, RESONA A A-enhanced models achieve comparable or superior zero-shot Im-harness evalua- tion scores to baselines. Combined with Table 9, these results demonstrate that the RES- ONAmodule enhances the backbone's in-context learning (ICL) capability while maintaining
its foundational language modeling performance.
ARC-C
ARC-E
Hella.
OBQA
PIQA
Wino.
acc
acc_n
acc
acc_n
acc
acc_n
acc
acc_n
acc
acc_n
a0c
aCc
a0c
acc
GLA
0.2381
0.2747
0.5442
0.5046
0.3852
0.4911
0.198
0.314
0,7008
0.7008
0.552
0.3110
0.5288
0.4417
+RESONA
0.2466
0.2918
0.5497
0.5227
0.3738
0.4722
0.188
0.308
0.6944
0.7010
0.550
0,3139
0,5399
0.4424
DeltaNet
0.2363
0.2637
0.5636
0.5341
0.3452
0.4893
0.198
0.7035
0.7002
0.552
0.3388
0.5375
0.4475
RESONA
0,2440
0,2722
0,5812
0.5455
0,3909
0.4982
0.198
0.314
0,7024
0.6997
0.556
0.3292
0.5375
0.4514
Manba
0.3558
0.3805
0.6953
0.6427
0.4958
0.6490
0.270
0.382
0.7533
0.7335
0.684
0.3598
0.6440
0.5435
+RESONA
0.3623
0.3951
0.6907
0.6904
0.4804
0.6292
0.300
0.406
0.7372
0.7383
0.690
0.3445
0.6338
0.3475
A.3 Training Details for Synthetic Benchmarks
Multi-Query Associative Recall (MQAR). We evaluate RESONA on the MQAR task by training six base architectures: BaseConv, H3, Hyena, Mamba, Mamba2, and RWKV. All models are trained with a 4-layer configuration and a hidden dimension model) ranging from 32 to 256. The sequence lengths vary from 64 to 512, with the number of key-value (KV) pairs corresponding to 4-32, respectively. To integrate RESONA, we insert the Resona Layer into either the first or third layer of each model, using the same d moddel settings. The learning rate is swept using the default settings from Arora et al. (2024). For the Resona Layer, we use chunk size of 2 and a top-k value of 1. The exact results for each model, sequence length, and hidden dimension can be found in Table 11.
Mechanistic Architecture Design (MAD) Suite. For the MAD tasks, we adopt a 4-hybrid block configuration, where each block consists of linear recurrent layer followed by
18

---

Published as a conference paper at COLM 2025
Table 11: Best Accuracy for Different Models with Varying Sequence Length and Model
Dimensions
64
128
1.
256
1.
512
Model
KV Pairs
KV Pairs
KV Pairs
KV Pairs
32
64
128
256
32
128
256
128
256
32
64
128
256
BaseConv
0.401
0.439
0.805
0.960
0.051
0.115
0.469
0.948
0.017
0.046
0.076
0.549
0.003
0.007
0.015
0.023
RESONAA
1.000
1.000
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1,000
1,000
1.000
1.000
1.000
1.000
+ RESONAA
1,000
1.000
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
H3
0.861
0.892
0.991
0.991
0.225
0.477
0.974
0.965
0.094
0.409
0.303
0.896
0.003
0.007
0.038
0.078
RESONAA
1,000
1.000
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1,000
1,000
1,000
1.000
1.000
1.000
RESONA2
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
Hyena
0.430
0.874
0.922
0.972
0.051
0.501
0.646
0.840
0.018
0.089
0.313
0.928
0.002
0.007
0.053
0.173
RESONAO
1,000
1,000
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1,000
1,000
1,000
1.000
1.000
RESONA2
1,000
1.000
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1,000
1.000
1.000
1.000
1.000
1.000
RWKV
0.727
0.955
0.992
0.953
0.216
0,451
0.807
0,986
0,066
0.093
0,435
0.820
0.001
0.002
0.005
0.010
+ -RESONA
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
+RESONA2
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
Mamba
0.987
0.992
0.975
0.990
0.906
0.992
0.991
0.920
0.864
0,990
0,990
0.991
0.000
0.419
0.968
0.000
+RESOVAD
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000)
1.000
1.000
1.000
1.000
1.000
1.000
1.000
RESONA2
1,000
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
Mamba2
0.990
0.992
0.991
0.993
0.974
0.969
0.992
0.991
0.755
0.000
0.991
0.592
0.000
0.000
0.001
0.000
RESONAA
1.000
1.000
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1,000
1,000
1.000
1.000
1.000
1.000
RRESONA2
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
SwiGLU layer. We follow the benchmark settings from Poli et al. (2024), using batch size of 128 and a learning rate range of 1e-3 to le-4. For the RESONA layers, we set the chunk size to 6 and the top-k value to 1.
A.4 Training Details for Real-World Tasks
Pretraining. During pre-training, we prioritized maintaining consistent model depth across architectures. For GLA, DeltaNet, and RetNet, we adopted the architecture imple- mentations from FLA-Hub (Yang and Zhang, 2024), configuring them with 24 layers and hidden size of 600. For Hymba, we use NVIDIA's official 150M parameter implementation (24 layers with hidden size of 512). For Mamba, we employed FLA-Hub's implementation with 48 layers and a hidden size of 600. For comparative models with equivalent parameter counts, we adjusted the hidden size from 600 to 640. Following the methodology outlined at the shallowest, middle, and deepest
in the Hymba paper, we integrated layers to reinforce critical information flow. Each model was trained for 8,000 steps, with model selection performed using dedicated validation set. The training configuration employed a cosine annealing scheduler with warmup over 5% of the training steps, the AdamW optimizer (learning rate of 1e-3 and weight decay 0.01), and gradient clipping of 1.0.
Finetuning. We employed a base learning rate of 1e-5 for fine-tuning on the three in- dividual datasets, with other training configurations remaining similar to those used in pre-training, except that RESON Amodules benefited from a higher learning rate. For CoQA, NarrativeQA, and TriviaQA, we trained for 2K, 8K, and 10K steps, respectively. During general supervised fine-tuning, we created a unified training set by shuffling 10K samples from each of the three QA datasets, ensuring significant diversity in sequence length and content. Additionally, we constructed a general test set by selecting 500 validation samples from each dataset. We trained on this combined dataset for 4K steps.
A.5 NIAH Scoring Details
For the NIAH (Needle In A Haystack) task, we employ an automated scoring protocol based on prefix matching. The scoring methodology operates as follows: A full score of 5 points is awarded if the model's 's response contains the complete and accurate needle. If not, we iteratively truncate the needle from the end (removing the last few characters incrementally) and perform prefix matching. Partial credit (a proportional fraction of the
19

---

Published as a conference paper at COLM 2025
5-point maximum) is granted when truncated prefix matches exactly, with the remaining character percentage determining the awarded score. Responses containing no matching prefix of the needle receive 0 points. The final task score is obtained by averaging the scores across 100 test samples.
20


Reviews:
[{"review_id": "nUPb6g8xXL", "reviewer": "Reviewer_ymyb", "summary": "This article introduces a novel technique aimed at improving the contextual information handling capabilities of Linear Recurrent Models (LRMs), a recently proposed efficient alternative to transformer-based models based on the principles of recurrent networks. Despite their advantages in terms of computational and spatial efficiency, LRMs suffer from limitations when copying and integrating previous input fragments due to their fixed-size hidden states, thereby creating an information bottleneck. To address this, the authors propose extending LRMs with a method inspired by Retrieval-Augmented Generation (RAG). Unlike traditional RAG, which retrieves external documents, this approach encodes and retrieves embeddings of previous context passages within the input itself. These embeddings then influence the model's hidden state updates and subsequent layer outputs. The authors present rigorous experiments demonstrating significant performance improvements for LRMs such as Mamba, RWKV, and Hyena in tasks that heavily depend on context copying capabilities.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental validation provided is robust, demonstrating clear performance improvements over baseline LRMs. The proposed approach effectively addresses a critical limitation in LRMs, potentially broadening their applicability as alternatives to transformer-based models, in general, and transformer-based LLMs in particular.", "reasons_to_reject": "To the best of my knowledge of linear recurrent networks (though I consider myself more of an expert in classical recurrent or transformer-based networks), I do not see any major flaws in the manuscript. One possible addition could be a statistical significance analysis, but the improvements reported are substantial enough that such an analysis may not be strictly necessary. The paper could benefit from a clearer explanation of the computational overhead introduced by integrating the proposed technique.", "questions_to_authors": "There are wrong sentences or typos at lines 49, 179, and 207. It would be useful to provide more detailed explanations of the benchmarks used and the kind of tasks they are designed to evaluate.", "extra_scores": {}}, {"review_id": "xlv3gFQ4SH", "reviewer": "Reviewer_9ueu", "summary": "While Linear Recurrent Models (LRMs) demonstrate computational efficiency advantages, their fixed-size hidden states create an information bottleneck that limits effective utilization of long-context information. To address this, we propose **Resona**, a framework that enhances LRMs' in-context learning capabilities through a retrieval mechanism. This approach bypasses the hidden state constraints of traditional LRMs by enabling direct information flow from input contexts to model reasoning. Extensive experiments demonstrate that Resona significantly improves LRMs' performance without requiring major architectural modifications, offering plug-and-play usability. I \n recommend clear accept.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Resona focuses on in-context retrieval rather than external databases, differentiating it from traditional RAG methods. By integrating retrieval mechanisms with LRMs, it fills an architectural gap and achieves outstanding performance on long-sequence tasks.\n- The paper formally analyzes the information bottleneck caused by fixed-size hidden states in LRMs. The *Rank Limitation of Linear Recurrence* analysis in appendices mathematically explains LRMs' limitations, providing theoretical grounding for the methodology.\n-  Comprehensive experiments confirm Resona's significant performance improvements across multiple benchmarks.\n-  The framework requires minimal architectural changes to existing LRMs and supports plug-and-play deployment.", "reasons_to_reject": "- The impacts of critical parameters (chunk size *U*, number of retrievals *k*, mixing coefficient *α*) on performance lack systematic analysis.\n- Insufficient discussion about Resona's adaptations for different LRM architectures (e.g., Mamba/Hyena/RWKV), particularly regarding hidden state interactions and retrieval module parameterization.\n- While claiming comparable inference efficiency to baselines, the paper lacks empirical comparisons of FLOPs, memory consumption, or practical latency (e.g., pre-filling phase chunk encoding overhead).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rYff539hVj", "reviewer": "Reviewer_UqFQ", "summary": "This paper proposes a new neural network module named Resona, that resembles an RAG procedure on a per-layer basis. The context is first chunked and embedded and relevant chunks are retrieved using queries corresponding to the hidden state of each token. The information retrieved chunks are then fused back into the model using cross-attention.\n\nThis paper is well-motivated as it is known that one of the critical bottlenecks of linear attention modules is that it could not retrieve perfectly from the context. While a hybrid model with softmax attention has been proposed to address this issue, this paper's more explicit way of fusing retrieval is a meaningful exploration.\n\nWhile the paper is overall clear, there are two issues of the paper that need to be addressed (I will expand more in the reasons to reject):\n\n1. While the 'formal' theory part has good intuition, the formulation of this theory is unfortunately incorrect.\n\n2. There are some important experiment details that are not properly explained, which may hinder reproduction.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of the paper is well delivered.\n\n2. The results on synthetic data and realistic data show that the designed module indeed improves the language modeling performance over the original linear architecture. The finetuning experiment also suggests that Resona layer is better than sliding window attention.", "reasons_to_reject": "1. The theoretical argument is not rigorous. The authors defined the state to be the hidden vector of dimension H with precision p, which restricts the maximal size of the set of state to be $2^{Hp}$, making Lemma 3.1 vacuous. I don't consider this a strong reason to reject but this is something the authors need to fix. \n\n2. Regarding the language modeling, the authors mentioned that Chunk and Search process is done 'offline'. I found this incompatible with the architecture design authors proposed, because the search process required the hidden state produced by the layers. \n\n3. There is no measurement of wall clock time of this module, making it hard to judge its generality. \n\n4. Also, the authors can be more clear on what exactly is $h$ used in the module. For linear architecture like deltanet, the recurrent RNN state is actually a matrix instead of a vector and it is hard to get every recurrent state using the fastest implementation. This is related to point 3. If the authors are actually using the output of the linear module, then this should be made clear.\n\n5. There should be an ablation comparing Resona with full attention.", "questions_to_authors": "Please refer to Reasons to Reject.\n\nAlso, I think there are some missing citations:\n\n[1] discussed RNN's limitation on memorizing, which has a theory that is a more formal version of Lemma 3.1. It also proposes retrieval as a potential fix.\n\n[2] The way the data is constructed seems to resemble in-context pretraining.\n\n\n[1] https://arxiv.org/abs/2402.18510\n\n[2] https://arxiv.org/abs/2310.10638", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Resona, a framework that augments Linear Recurrent Models with retrieval mechanisms to address their well-known issues in context copying and ICL. The main idea is to add a parallel cross-attention branch that retrieves relevant information from the input context, bypassing the bottleneck created by fixed-size hidden states.

Strengths:

- The motivation is clear and addresses a genuine limitation of LRMs
- The experimental eval is comprehensive across multiple archs(mamba, RWKV, Hyena) on both synthetic and real-world tasks
- The approach is practical and plug-and-play
- Authors provided detailed overhead analysis in their rebuttal

Weaknesses:

- The theory analysis (Lemma 3.1) seems flawed as one reviewer correctly pointed out
- Some implementation details were initially unclear around the "offline" chunk and search process and how hidden states are handled across different architectures
- perhaps lacks systematic ablation on some hyperparameters (though the authors partially addressed this during rebuttal with some results on eg chunk size)
- Missing important related work (fixable)


Overall it's a solid piece of work, that all Revs and I agree.
Decision: Accept