PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding
Authors: Kangcong Li
Peng Ye
Chongjun Tu
Lin Zhang
Chunfeng Song
Jiamin Wu
Tao Yang
Qihao Zheng
Tao Chen
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Long Contexts
Brain-inspired LLMs
Feed-Foward Networks
Working Memory
Persistent Activity
Cortical Expert
Primary area: deep_learning

Abstract: While Large Language Models (LLMs) demonstrate strong performance across domains, their long-context capabilities are limited by transient neural activations causing information decay and unstructured feed-forward network (FFN) weights leading to semantic fragmentation. Inspired by the brain’s working memory and cortical modularity, we propose PaceLLM, featuring two innovations: (1) a Persistent Activity (PA) Mechanism that mimics prefrontal cortex (PFC) neurons’ persistent firing by introducing an activation-level memory bank to dynamically retrieve, reuse, and update critical FFN states, addressing contextual decay; and (2) Cortical Expert (CE) Clustering that emulates task-adaptive neural specialization to reorganize FFN weights into semantic modules, establishing cross-token dependencies and mitigating fragmentation. Extensive evaluations show that PaceLLM achieves 6% improvement on LongBench’s Multi-document QA and 12.5–17.5% performance gains on $\infty$-Bench tasks, while extending measurable context length to 200K tokens in Needle-In-A-Haystack (NIAH) tests. This work pioneers brain-inspired LLM optimization and is complementary to other works. Besides, it can be generalized to any model and enhance their long-context performance and interpretability without structural overhauls.

PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding
Kangeong Li Peng Yea.3, Chongjun Tu', Lin Zhang', Chunfeng Song", Jiamin Wu?, Tao Yang', Qihao Zheng"t Tao Chen
School of Information Science and Technology. Fudan University Shanghai Artificial Intelligence Laboratory Thhe Chinese University of Hong Kong
Abstract
While Large Language Models (LLMs) demonstrate strong performance across domains, their long-contex: capabilities are limited by transient neural activa- tions causing information decay and unstructured Teed-forward network (FFN) weights leading to semantic fragmentation. Inspired by the brain's 's working mem- ory and cortical modularity, we propose PaceLLM, featuring two innovations: (1)
Persistent Activity (PA) Mechanism that mimics prefrontal cortex (PFC) neurons' persistent firing by introducing an activation-level memory bank dynamically retrieve, reuse, and update critical FFN states, addressing contextual decay; and (2) Cortical Expert (CE) Clustering that emulates task-adaptive neural specialization reorganize FFN weights into semantic modules, establishing cross- cross-token depen- dencies and mitigating fragmentation. Extensive evaluations show that PaceLLM achieves 6% improvement performance gains on oc-Bench tasks, while extending measurable context length 200K tokens in Needle-In-A Beedle-In-A-Hayseee In-A-Haystack (NIAH) tests. This work pioneers brain- inspired LLM optimization and is complementary to other works. Besides, it can be generalized to any model and enhance their long-context performance and interpretability without structural overhauls.
LongBench's 's Multi-document QA and 12.5-17.5%
Introduction
Large Language Models (LLMs) have revolutionized natural language processing, achieving state- of-the-art results in tasks ranging from open-ended text generation [3] to complex multi-step reasoning 47 These advances have made LLMs the backbone of many real-world applica- tions 43
seowledge-intensing tasks [19]. As these applications scale, there a growing demand for models to handle longer input sequences, particu- larly in scenarios such multi-document question answering (50) long-form summarization (30), and conversational memory (36). Modeling such extended contexts requires LLMs not only to retain information over longer spans, but also reason distributed and interdependent content. This has brought renewed attention to the internal mechanisms that govern context modeling and memory persistence within LLMs.
42 from dialogue systems [2
Existing approaches to address long-context challenges generally fall into three categories. The first enhances LLMs' reasoning capacity through architectural training improvements [27 40 37
32
The second focuses on input compression, reducing redundancy while preserving key information 23
"Equal contribution. Corresponding authors: zhenggthao@pjlee org. en, eetchen@fudan. edu, cn.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prpae plaseet
Preset
Pryera
Prose
Deowery
Chudbic the
Chadwic restary
Chedwick
reeerch atforts He was the hradd of Bincch
What person provaded mvdenne
the Nobel Physics for dissovery
team that worked on the Manhattan Project during World War IL He was Inighted in
sugpest the existonce the neutron
partisipant of?
CantesCCpestTr
Length
Figure 1: Schematic diagram of the PaceLLM (bottom) and its neuroscience counterpart (top). In this case, which introduces James Chadwick's ' character, the brain processes and retains key information through working memory. When the content working memory appears in the subsequent text, such as "Britain", relevant neurons will persistently to be re-active. When the final question is input, the neuron with the keyword "neutron" will also persist to re-activated, connect with other relevant neurons, and finally find the answer "Manhattan Project". Analogical to the mechanism of brain, PaceLLM expertly clustered FFN weights, and designed an Activation Memory Bank (AMB) to interact with activations.
26 31 45 9]. The third introduces external components, such as memory modules 34 and
retrieval
retrievalauugmenter augmented generation (RAG) 49 33 38) to compensate for limited attention spans. However, these approaches often overlook fundamental internal limitation: the role of feed-forward networks (FFNs). Specifically, transient neural activations cause information to fade over time, and unstructured FFN weights may fragment semantics across tokens, jointly undermining coherence in long context understanding.
To alleviate this problem, we draw inspiration from neuroscience to explore the untapped potential of FFN activations. Notably, the brain's 's working memory [51] and cortical modularity [ demonstrate remarkably effective mechanisms for long-context processing, as illustrated at the top of Figure [ Working memory refers to the brain's ability to temporarily retain and manipulate task-relevant infor- mation through persistent neural activity in the preffronta cortex (PFC) (8] When previously stored information reappears, relevant PFC neurons remain active, helping preserve relevant content and counteract information decay. Concurrently, the cerebral cortex 225 is functionally partitioned into distinct regions [5], enabling specialized "neuron experts" to handle different tasks. This modular organization improves semantic consistency and supports efficient long-context comprehension.
Inspired by above brain's mechanisms, we propose PaceLLM, as illustrated at the bottom of Figure Our approach consists of two key components: (1) Activation Memory Bank (AMB) to emulate PFC persistent activity (PA). This component flattens, retrieves, fuses, and stores intermediate activations. Retrieval computes similarity between current and historical activations, allowing highly similar representations to be reactivated and reused. (2) Cortical Expert (CE) via clustering and reordering. We first cluster the gated projection matrix with equal experts per cluster. Then, the gated and upper projection matrices are reordered by rows, and the lower projection matrix is reordered by columns, yielding structured FFN with expert-specialized layout.
We evaluate the proposed PaceLLM on LongBench [2] and oo-Bench [46 using Qwen-2-7B --BB-2--BB Instruct (39 and Mlama-2-7B-chal 28 as base models. Under the training-free free setting, our method consistently outperforms baselines. When aligned with fine-tuning baselines, we achieve 6% improvement on the Multi-document QA task in LongBench. On xc-Bench, the performance of Dn.liale and En.Multi-Choice tasks is improved by 12.5% and 17.5%, respectively. In the Needle-Ie In-A A-Haystack (NIAH)[18 test, our method handles contexts up to 200K tokens, substan-

---

tially surpassing Activation Beacon 45 128K limit. Our contributions can be summarized as follows:
(1) pioneering brain-inspired approach to enhance LLMs' long-context understanding. Seaennciesssscciclalllll. fleeting
While prior efforts achieve great success, they overlook internal activations that weaken retention and disordered FFN weights that disrupt semantic continuity. We propose the first brain-inspired solution targeting these core limitations.
(2) Training-free persistent activity (PA) and cortical expert (CE) clustering mechanisms. We introduce memory bank that mimics working memory by operating at the activation level, enabling finer-grained retention than token-level storage. Our cortical modularity method structures FFNs to better capture inter-token semantics. Our method is model-agnose and Blug-and-pin
(3) Strong performance across long-context benchmarks and NIAH. Our approach achieves over 10% gains on several tasks and extends the usable context length to 200K tokens, demonstrating both improved reasoning capabilities and robust scalability.
2 Related Work
2.1 Modeling and Understanding Long Contexts with LLMs
Enhancing LLMs' ability to process long contexts remains an active research challenge with three mainstream directions. Input preprocessing techniques like prompt engineering [27] 48) position encoding [26 6 and KV cache compression [22
31 45
reduce input complexity and guide L.MSs
to focus on key information; I.LM structural optimizations, such as continual learning 37 and
model editing [2). adapt model parameters to better handle extended contexts. External augmenta- tion methods, including memory banks [34] 7 and
Generation 49
33 38)
supplement the model's internal capabilities by storing historical information or retrieving relevant content. Despite demonstrated improvements, these approaches have limitations: preprocessing methods often operate at coarse granularity (token or embedding level), structural optimizations incur significant computational costs, and external augmentations introduce system complexity and operational overhead.
It has been increasingly recognized that feed-forward networks (FFNs) in Transformers operate as key-value memories, where each neuron responds to specific input patterns and produces associated outputs [1O]. Our proposed PaceL_MM differs from existing studies by focusing the feed-forwart networks (FFNs) within transformer layers, an aspect largely overlooked in previous long-context solutions. PaceLLM addresses two core issues: transient neural activations causing information decay and unstructured FFN weights leading to semantic fragmentation. Our approach operates at activation-level granularity and reorganizes FFN weights into semantic modules, providing complementary solution that can be integrated with existing methods to further enhance long-context understanding.
2.2 Brain-Inspired Interpretability in LLMs
Brain- frain-inspired approaches have emerged as a promising direction for improving LLM interpretabil- ity and performance. HippoRAG [17 implements a retrieval system modeled after neocortex-
hippocampus interactions. HMT [1 introduces three-level memory hierarchy mimicking human memory processes. Larimar (5] augments LLMs with external episodic memory module for knowledge editing and long-context processing. NeuroMFA (35) quantifies emergent abilities in LLMs by analyzing structural dynamics of neuron interaction networks. These approaches demon- strate how mechanisms in the brain can enhance model architecture, processing mechanisms, and interpretability, establishing valuable cross tross-disciplinary connections.
PaceLLM extends brain-inspired research by focusing on neural persistent activity (PA) and cortical expert (CE), which are two underexplored yet fundamental neurobiological principles. In contrast to prior work emphasizing external modules or attention layers, our method targets the FFNs, which account for most model parameters but lack neuroscience-guue guided design. By embedding activation- level memory and expert clustering into the computation flow, Pacel.LM enhances long-context performance with minimal architectural changes.
3

---

Clustered FFN(Ney)
Clustered FFNivalue)
Feed FeadiFoor word Network
Activations
Memory
Oustered FFhlicyy)
PPN Output
ree
Feed- forword Netwwork
Feed For-orrd Network
mNJvvlus)
Update Activations....
Lookup Memory
Auga
Clustered FiNjkey))
EE
barbarrt
Fuger Rease Memary
Our c Sugart Orck Mis Activations
Reglase Mensor
Supur 6
Rauds For woreorr Natwork
Oustered mijaalus
Acthaation Memory Bank
Weigheed
Activations
Sent
AM
Keep
FFN(lbyy)
2
So Ao--Attention
Bostomn K
Werghte
Update Memory Bank
Figure 2: The illustration of PaceLLM. The left of the figure is an overall pipeline. Note that Activation Memory Bank (AMB) doesn't 't interact with all FFN layers. The top right of the figure is a detailed illustration of the modified FFN layer. The bottom right is detailed processing flow of AMB. @Lookup Memory shows the process of similarity retrieval, taking the topk, and adding noise. @ shows the selection of reusing strategies by comparing similarity with threshold. @ shows three strategies for updating the AMB.
3 Method
3.1 Preliminary
Modern LLMs are primarily built upon the Transformer 229 architecture, which contains two core components: the multi-head self-attention mechanism and the position-wise feed-forward network (FFN). While attention modules enable dynamic global interactions, FFNs process token-level information in parallel and contribute substantially the model's capacity and computational cost.
Multi-Head Self-Attention. dynamically models global contextual dependencies between tokens by computing attention scores across all positions in the sequence:
QK
Attention( Q, V) softmax
Vd
(1)
MultiHead Q, K, V) = Concat Concat(head. ..... head, W
where head, = AttentionnQW kW. VW
Position-wiid wise Feed-Forward Network. It applies non-lineat transformations to refine individual token representations, operating independently on each position. For an input token representation x E I dundd the FFN layer performs:
FFN(x) = W. - (Wix b,) + bg
(2)
where W1 E ddeed and W2 E Badaas dd are learnable weights. du typically set as 4dmmxel defines the expanded intermediate dimension. Activation function a (e.g.. ReLU,, GeLU) enables non-linear feature interactions.

---

3.2 PaceLLM
Inspired by working memory and cortical processing the brain, we propose PaceLLM (Persistent Activity and Cortical Experts LLM) to enhance long-context understanding. As shown in Figure 2 PaceLLM integrates two biologically motivated components: (1) Activation Memory Bank (AMB). which mimics persistent neural activity in working memory by caching and retrieving FFN activations; and (2) Cortical Experts (CE) Clustering, which introduces a similarity-based expert selection mechanism, inspired by specialized processing in the cerebral cortex. We describe each component below.
3.2.1 Activation Memory Bank (AMB)
To simulate persistent neural activity, we augment the FFN with an Activation Memory Bank (AMB) that stores and reuses intermediate activations. Specific FFN layers are equipped with memory bank M KK, V, u} where K, V E RM denote the memory keys and values, and IR M tracks usage frequency. The workflow consists of memory lookup, activation update, and memory update.
Memory Lookup. Given intermediate activations X Cxdn, we compute their cosine similarity with stored keys K:
X
kj
sim,j
Vi E [C], E [M].
xx k
We then retrieve the top-k most similar historical entries and bottom-k' least similar ones introduce diversity:
r" S = (si, k),
1 S = (opK(-siin) R').
(5)
Activation Update. The final output o, is computed by integrating current and retrieved activations based on similarity confidence:
pos + Apa neg
if max(S lop > Onigh
Avg(x;, (00) + Ape
neg
1 < (S I Onigh otherwise
Xj,
where pi pes and ue
neg mean vectors of top and bottom activations, and À is noise scaling factor.
Memory Update. After computing outputs, AMB updated using a similarity- aware strategy:
High similarity ToOp
(righ): : No update; only increment usage counter
Medium similarity (flow
Stop
I Bmigh): Update stored memory by merging current
activation:
K₁- f- Avg(K,; pc), V, + Avg Avg(V (v
Low similarity (Suop < 0 Replace least-used slot using LRU policy (4]
While CAMELOT 12 uses similarity trigger memory updates based on novelty, it replaces the least recently used slot, ignoring semantic importance. In contrast, PaceI.LM selectively retains and updates memory based both similarity and contextual relevance, mimicking persistent neural activity in working memory. This mechanism allows PaceLLM to persist and reuse relevant activation traces dynamically across long contexts.
3.2.2 Cortical Expert (CE) Neuron Clustering
Inspired by the functional modularity of the brain cortex 11, where localized neuron groups are activated by similar input signals, we reinterpret the FFN layer as an overparameterized neuron pool that can be decomposed into semantically coherent cortical experts. This decomposition enables both specialization and modularity in later decoding. We propose a two-stage transformation pretrained FFN weights: (1) expert discovery via balanced clustering, and (2) parameter reorganization to form modular expert blocks. This design mirrors cortical specialization, where neurons with similar activation properties co-locate and collaborate. This process does not require retraining.
5

---

Table 1: Performance comparison between PaceLLMM and baseline models on LongBench tasks in training-free setting. CE denotes cortical expert neuron clustering and PA denotes persistent activity memory mechanism.
Model
Method
SQA
MQA
Sum.
FSL
Cod.
Vanilla
37.76
49.03
28.93
70.36
50.05
Vanilla CE
37.68
48.80
28.85
70.61
50.36
Qwen-2-7B-Insteee
Vanilla PA
38.09
49.36
28.86
70.92
49,60
Vanilla CE PA
38.49
50.28
29.02
70.96
49.95
Vanilla
23.92
23.42
24.43
63.02
55.48
Vanilla
24.49
23.73
24.38
62.86
55.17
Alama-2-7B-chan
Vanilla
24.65
23.15
24.18
63.23
54.98
Vanilla CE PA
25.35
23.75
24.61
63.58
55.28
Expert Discovery via Constrained Clustering. Given FFN weight matrices W Rduxxduak and W2 we treat the rows of W as candidate neurons and apply KMMeanSonnttraiee 21 clustering:
Wi
Wa
Vi
{L....,da)
(8)
[wal
K
dx
min (Cp)
Σ
W. 1tyl?
s.t. 1C
j=1
where K is the predefined number of experts and Cj denotes the cluster for expert j.
Parameter Reorganization. Let be the index permutation obtained by concatenating all cluster memberships. We reorganize FFN weights as follows:
Wnew iK (i + 1) W
W1 Wi[w(KK (i K],
(10)
new
F, ik (i + 1) = W2li.Tiik (i 1]
(11)
This expert-wise rearrangement preserves the integrity of each neuron cluster while maintaining compatibility with the original FFN structure.
Implementation Details. Caching: Expert indices are cached per layer avoid redundant clustering
during repeated runs. In-place Processing: Reordering is performed in-place to reduce memory overhead. Inference Compatibility: Output shapes and computational graphs remain unchanged, ensuring zero-cost integration.
Experiments
4.1 Settings
Datasets. We evaluate PaceLLM on three established long-context benchmarks: LongBench (2]. oo-Bench [46] and Needle-In- In---aasstacck (NIAH) [18). To evaluate the generalization ability of our method beyond long-context tasks, we also evaluate on MMLU 13), which features shorter context lengths.
Implementation. We apply PaceLLM to -lama-2-7B-chan 28 and Qwen-2-7BB 1rren22-BB-nsttrue Instructt [39 in training-free and low-cost fine-tuning settings. For low-cost fine-tuning, we follow the setting of Activation Beacon 45
All experiments are conducted with 4x A 1100-40G GPUs.
Baselines. We compare PaceLLM with the original base models and several context compression methods, including LongLLMLingus [16), SnapKV (20). and Activation Beacon (AB) [45). As PaceLLM is orthogonal to these methods, we also integrate PaceLLM with Activation Beacon to demonstrate complementary benefits.
4.2 Experimental Results
Results on LongBench. Table presents training-free performance results. For both Qwen-2 and Llama-2, the components of our method (cortical expert neuron clustering CE and persistent activity

---

Table 2: Performance comparison between PaceLLM and baseline models on LongBench tasks in low-cost fine-tuning setting. CE denotes cortical expert neuron clustering, and PA denotes persistent activity memory mechanism.
Model
Method
SQA
MQA
Sum.
FSI.
Cod.
Vanilla-FT
41.00
40.60
26.80
68.50
66. 10
Lo 16
24.70
20.30
26.30
55.90
50.10
SnapKV
A0
38.70
37.60
26.20
67.10
60.30
Qwen-2-7B-Insse
Activation Beacon
45
40,50
40.30
26.80
68.40
66.40
Activation Beacon + PA
41.10
42.80
27.90
69.31
67.51
Activation Beacon CE
40,90
44.58
27.36
68.98
67.26
Activation Beacon CE PA
42.62
46.55
28.74
70.56
67.52
Table 3: Results on oc- oo-Bench.
Table 4: Results on MMLU.
Fiu
RumSum
Fn.QA
7h.QA
Fr.MC:
Code. Run
STEM
Social Sciences
Humanities
Others
Avg.
AB
65
3.00
3.37
9.57
22.31
46.72
0.50
AB 15
61.891 61.974
79,780 80,047
72.721 72,915
70.530
70.250
Ours
15,5
4.11
14.14
24.84
64.19
2,50
Ours
71,075
70,510
memory PA) individually improves the performance. When combined, they work synergistically and achieve the best overall performance, with improvement up to 1.4% on certain subtasks without
any training. To ensure fairness compared to the fine-tuning method, Table 2 shows PaceLLMM's
low-cost fine-tuning performance. Applying our method to Activation Beacon 45 leads to significant performance improvements across all task categories, especially the Multi-document QA task having improved % performance. The consistent performance gains demonstrate that our brain-inspired approach effectively enhances the model's ability to process long-range contextual information. The best performance is achieved by combining the two mechanisms.
Results oc-Bench. Table shows the experimental results oc-Bench, another long-conters context benchmark. Without any additional training, our method outperforms Activation Beacon significantly across all tasks. For example, 12.5% on Dialogue task and 17.5% on Nn..uuli--Chiie task.
Results on Boeell-In-A-Haysttee the official settings [18] and illustrate the results in Figure 3 The context length is expanded to 200K for further evaluation. As can be concluded, our proposed PaceLLM consistently retrieves the needle more precisely than Activation Beacon's 's 128K context length.
We further evaluate on Needle- In-A- (NIAH) following
Results on MMLU. As can be seen from Table 4 while our method is specially designed for long- context scenarios, it maintains performance improvements on the short-context MMLU benchmark. This indicates that PaceLLM has not compromised in its general language understanding capabilities.
4.3 Discussion
The experimental results of each model on different datasets can prove the effectiveness of PaceLLM. To further improve the interpretability of our method, we also design a visualization experiment. The selected model is Qwen2-7B and the task is GovReport in LongBench. As shown in the Figure 4 during model evaluation, we record activations from both current input and AMB at different moments and convert them back to tokens with semantics. According to the semantic information, they are drawn in a two-dimensional semantic figure, where points with similar distances indicate similar semantics, the color of the points indicates the usage frequency according to the legend on the right, and the red point indicates the activation corresponding to the current input.
The visualization shows that the current input activation form clusters with semantically similar historical activations, while the historical activations in each cluster are fully reused. Therefore, can be inferred that PaceLLM can retrieve the semantically similar historical activations stored in AMB for different current activations, which can re-activated and reused sufficiently many times by analogy with working memory. This demonstrates that PaceI.LM indeed has mechanism highly similar to the brain's working memory, which effectively enhances the understanding of long contexts.
7

---

Needle In A Haystack
0
10
11
22
8
Depth Percent
33
Accuracy
55
5
66
3
88
100
1000
23111
45222
67333
89444
111559
133666
1
555777
177988
200000
11
9
22
8
Depth Percent
33
7
44
6
Accuracy
55
5
4
66
3
88
2
1
100
1000
23111
45222
67333
89444
11 111555
133666
155777
177888 200000
Context Length
Figure 3: Evaluation on Boaele-InA--Haystackk PaceLLM (bottom) can retrieve the needle up to
200K than Activation Beacon 128K (top).
4.4 Ablation Studies
To facilitate a fair and systematic comparison, we establish base configuration using Qwen2-7B on LongBench. In the base setting, the bank capacity M is set to 100, the fusion threshold Onigh is 0.7 and 9low is 0.3, with AMB applied the 13th and 27th layers. Based this setting, conduct ablation studies about deployment location, fusion thresholds, and the design noise adding in memory lookup as follows.
Ablation of deployment location. Since our approach flexible and can be integrated into any layer of the model, examine the effect of applying method different network depth and report the results in Table (a). For single-document question answering (SQA) and code generation (Cod.), sparse deployment (e.g.. layers 13 and 27) performs better due to lower requirements for long-range coherence and higher variability input texts. For summarization (Sum.) and multi- document question answering (MQA), which demand stronger global context modeling, denser layer configurations (e.g., every other or fourth layer) yield better results. Deploying at all layers consistently underperforms and increases computational cost. Therefore, adopt different sparse deployment locations for different tasks.
Ablation of fusion thresholds. Table 5 (b) shows the impact of different fusion thresholds Ohigh and flow across tasks. For complex tasks such as MQA, Sum., and few-shot learning (FSL), better results achieved with lower fow (c.g.. 0.1). indicating that direct reuse of high-similarity activations from the AMB improves consistency and coherence long-range context modeling. Among these, MQA particularly benefits from combining current and historical representations, suggesting its need for both contextual understanding and knowledge retrieval. In contrast, for simpler tasks like SQA code generation, where input contexts shorter and exhibit less inter-dependency, moderate thresholds (e.g., 0.5) yield optimal performance. This suggests that excessive memory reuse may introduce noise rather than useful information for such tasks.
Ablation of noise adding design. Results in Table 5 (c) confirm that adding negative entries (Equation 5 into activations consistently improves performance. This design draws inspiration

---

Activations Semantics Clustering
Ourrent Activrion vctionaion
70
Usage Frequency Count
Semanntcs Distance
40
10
Semantics Distance
Figure 4: Visualization of current and historical activations. The orange circles encireled the clusters of current and past activations, which means they have similar information and useful past activations are sufficiently reused. It illustrates PaceLLM leverages the AMB to retrieve semantically similar past activations, enabling repeated reuse in a manner analogous to working memory.
Table 5: Performance comparison: (a) across different network layers, (b) under various fusion threshold settings, and (c) with/without noise addition.
(a) Applied at different network layers
(b) Under various fusion threshold settings
Layer No.
SQA
MQA
Sum.
FSL
Cod.
Bhigh. 6ko
SQA
MQA
Sum.
FSL
Cod.
Baseline
40.90
44.58
27.36
68.98
67.26
Baseline
40.90
44.58
27.36
68.98
67.26
1,2,3, 288
40.48
44.94
28.39
68.59
64.05
0.9, 0.9
40.97
44.98
69.09
66.98
2.4.6.-
41.20
45,49
28.78
68.02
65.48
0.1. 0.1.
41.43
45,49
28.64
70.1
66.94
20,-10.---22 -26
41.52
45.34
28.47
69.90
66.49
0.5. 0.5
41.58
45.13
28.56
68.83
67.13
18,26
41.28
44.84
28.02
69.96
66.81
0.9,
40.52
45.83
28.57
69.71
2,27
41.46
45.41
28.36
69.79
67.36
0.7, 0.3 (hase)
41.62
44.68
28.19
69.41
66.71
14,26
41.23
45.03
28.34
69,52
66.64
13,27 (base)
41.62
44.68
28.19
69.41
66.71
(c) Ablation results with and without noise addition
41.53
45.12
28.38
69.35
66.97
Serting
SQA
MQA
Sum.
FSI.
Cod.
41.67
44.97
28.17
69. 16
66.98
41.62
41.21
44.77
28.60
69.33
67.09
who moise
10.70
43.90
27.90
68.97
66.49
from human memory systems, where both relevant and contrasting information contribute to robust decision-man maaiin.. For each query, if all top-k samples extremely similar, introducing small number of least-similar samples can serve multiple purposes, such as providing additional context or counter-examples, preventing excessive repetition, and enhancing adaptability diverse scenarios.
5
Conclusions & Limitations
Inspired by the prefrontal cortex's 's working memory and cerebral modularity, we propose PaceLLM, brain-inspired framework to enhance long-context understanding in LLMs. Our method introduces two key innovations: Persistent Activity Memory Mechanism (PA) dynamically retrieves and reuses FFN activations through an external Activation Memory Bank (AMB), simulating the persistent firing. By selectively storing high-value activations and employing similarity-based fusion strategies, this mechanism mitigates context degradation in long sequences. Cortical Expert Neuron Clustering (CE) reorganizes disordered FFN weights into task-specialized modules, establishing semantic links

---

between isolated token representations. This mimics the brain's 's cortical modularity. Experimental results demonstrate significant improvements across multiple benchmarks.
Our method has great highlights in performance, biological plausibility and interpretability of LLMs. It is the first brain-inspired improvement in the FFN layer for solving long-context problems, which is complementary to most existing methods and is plug-and-play However, AMB is an additional module based on the original model, which will introduce certain extra calculation and storage costs. In addition, given that our method is orthogonal to most works, we believe that our method will not be limited to the field of plain text understanding, and we can extend our method embodied intelligence and other fields in the future to fully realize the potential of brain -aaii-esspired AI technology progress.
mola,,
Acknowledgments and Disclosure of Funding
This work is supported by National Key Research and Development Program of China (No. 2022ZD0160101), Shanghai Natural Science Foundation (No. 232R 2329040,,00), Shanghai Science and Technology Commission Explorer Program Project (24TS1401300 (000), Shanghai Municipal Science and Technology Major Project 20112222000))000033 and Shanghai Artificial Intelligence Laboratory. The computations in this research were performed using the CFFF platform of Fudan University.
References
[1] G. Auda and M. Kamel. Modular neural networks: a survey. International journal of neural systems, 9(22):199-151, 1999.
[2] Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 31 111131337 2024.
[3] Y. Bai, J. Zhang. X. Lv, L. Zheng, S. Zhu. L. Hou, Y. Dong. J. Tang, and J. Li. Longwriter: Unleashing 10,000+ word generation from long context Ilms, arXiv preprint arXiv:240S.070S. 2024.
[4] L. A. Belady. A study of replacement algorithms for a virtual-storage computer. IBM Systems journal, 5(2):78-101, 1966.
[5] P. Das, S. Chaudhury, E. Nelson, Melnyk, S. Swaminathan, S. Dai, A. Lozano, G. Kollias, V. Chenthama- rakshan, Jiff, Navrátil, S. Dan, and P.-Y. Chen. Larimar: Large language models with episodic memory control, 2024.
[6] Y. Ding, L. L. Zhang, C. Zhang, Y. Xu, N. Shang. J. Xu, F. Yang, and M. Yang. Longrope: Extending Ilm context window beyond 2 million tokens. arXiv preprint arXiv:2402. 13753, 2024.
[7] Z. Fountas, M. Benfeghoul, A. Oomerjee, F. Christopoulou, G. Lampouras, H. B. Ammar, and J. Wang. Human-inspired episndic memory for infinite context LLMss. In The Thirteenth International Conference on Learning Representations, 2025.
[8] J. M. Fuster and G. E. Alexander. Neuron activity related to short-term memory. Science, 173(3997):652- 654, 1971.
[9] S. Ge, Y. Zhang, L. Liu, M. Zhang, J. Han, and J. Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.0180. 2023.
[10] Geva, R. Schuster. J. Berant, and Levy. Transformer fred-forward layers are key-value memories. 2021.
[11] Z. He, Y. Cao, Z. Qin, N. Prakriya, Y. Sun, and J. Cong. Hmt: Hierarchical memory transformer for efficient long context language processing. arXiv preprint arXiv:2405.0600 2024.
[12] Z. He, L. Karlinsky, D. Kim, J. McAuley, D. Krotov, and R. Feris. Camelot: Towards large language models with training free consolidated associative memory, 2024.
[13] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and Steinhardd Mcasuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
[14] C. Huang. P. Ye, T. Chen, T. He, X. Yue, and W. Ouyang. Emr Emr-merging: Tuning- free high- performance model merging. 2024.
10

---

[15] R. A. Jacobs, M. L. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neurul computation, 3(1)799-77, 1991.
[16] H. Jiang, Q. Wu, X. Luo, D. Li, C.-Y. Lin, Y. Yang, and L., Qiu. Longllmlingu:: Accelerating and enhancing Ilms in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1658-1677. 2024.
[17] B. Jimenez Gutierrez, Y. Shu. Y. Gu, M. Yasunaga, and Y. Su. Hipporag: Neurobiologiaally inspired long-term memory for large language models. Advances in Neural Information Processing Systems, 37:59532-55955 -99399 2024.
[18] G. Kamradt. Needle in haystack- pressure testing Ilms. atttps://gite com/ wtt..oo./gaaaa//iiiiiia.. Ie 2023.
[19] J. Ko, G. Park, Lee, and K. Lee. FeRG-LLM: Feature engineering by reason generation large language models. In I.. Chiruzzo, A. Ritter, and I.. Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025. pages 4211-4225 Albuquerque, New Mexico, Apt. 2025. Association for Computational Linguisties.
[20] Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen. Snapky: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-22970, 2024.
[21] M. I. Malinen and P. Fränti. Balanced k-means for clustering. In Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshop, S+ SSPR 2014, Joensuu, Finland, August 20-22, 2014, Proceedings, pages 32-41. Springer, 2014.
[22] P. Nawrot, A. Lañccuci. M. Chochowski, D. Tarjan, and E. M. Ponti. Dynamic memory compression: Retrofitting Iims for accelerated inference. arXiv preprint arXiv:2403.090 09666, 2024.
[23] J. Park, K. Atarashi, K. Takeuchi, and H. Kashima. Emulating retrieval augmented generation via prompt engineering for enhanced long context comprehension in llms, 2025.
[24] K. Qian, M. Chen, S. Li, A. Sharma, and Z. Yu. Bottom-up synthesis of knowledge-grounded task-oriented dialogues with iteratively self-refined prompts. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 827-844, Albuquerque, New Mexico, Apr. 2025, Association for Computational Linguistics.
[25] E. T. Rolls. Brain computations: what and how. Oxford University Press, 2021.
[26] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Newrocomputing, 568:127063, 2024.
[27] K. Team, A. Du, B. Yin, B. Xing, B. Qu, B. Wang, C. Chen, C. ZZhnng C. Du, C. Wei, C. Wang, D. Zhang, D. Du, D. Wang, E. Yuan, E. Lu, F. Li, F. Sung, G. Wei, G. Lai, H. Zhu, H. Ding, H. Hu, H. Yang. H. Zhang, H. Wu, H. Yao. H. Lu, H. Wang, H. Gao,, H. Zheng, J. Li, J. Su, J. Wang. J. Deng, J. Qiu, J. Xie, J. Wang, Liu, J. Yan, K. Ouyang. L. Chen, L. Sui, L. Yu, M. Dong, M. Dong, N. Xu, P. Cheng, Q. Gu, R. Zhou, S. Liu, S. Cao, T. Yu, T. Song, T. Bai, W. Song, W. He, W. Huang, W. Xu, X. Yuan, X. Yao, X. Wu, X. Zu,
X. Zhou, X. Wang, Y. Charles, Y. Zhong, Y. Li, Y. Hu, Y. Chen, Y. Wang, Y. Liu, Y. Miao, Y. Qin, Y. Chen,
Y. Bao, Y. Wang. Y. Kang, Y. Liu, Y. Du, Y. Wu, Y. Wang. Y. Yan, Z. Zhou, Z. Li, Z. Jiang. Z. Zhang,
Z. Yang, Z. Huang, Z. Huang, Zhao, and Z. Chen. Kimi-VL technical report, 2025.
[28] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava,
S. Bhosale. D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu,
B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, Korenev,
Koura, M.-A. Lachaux. Lavril, Lee, Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, 1. Molybog. Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. F. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, 7. Yan, I. 7arov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Ilaam 2: Open foundation and fine-tuned chat models, 2023.
Khabsa, I. Kloumann,
[29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, ł. Kaiser, and I. Polosukkin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[30] D. Wan, J. Chen, E. Stengel-Eskiin and M. Bansal. MAMM-refine; A recipe for improving faithfulness in generation with multi-agent collaboration. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers). pages 9882 29019, Albuquerque. New Mexico, Apt. 2025, Association for Computational Linguistics.
11

---

[31] Z. Wan, X. Wu, Y. Zhang, Y. Xin, C. Tao, Z. Zhu. X. Wang. S. Luo, J. Xiong, L. Wang, and M. Zhang- D2o: Dynamie discriminative operations for efficient long-context inference of large language models. In The Thirteenth International Conference on Learning Representations. 2025.
[32] P. Wang. Z. Li. N. Zhang, Z. Xu, Y. Yao, Y. Jiang, P. Xie, F. Huang. and H. Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. Advances in Neural Information Processing Systems, 37:53764-53777 2024.
[33] Z. Wang. Z. Wang, Le, S. Zheng. S. Mishra, Perot, Zhang, A. Mattapalli, A. Taly, J. Shang. C. Y. Lee, and T. Pfister. Speculative rag: Enhancing retrieval augmented generation through drafting. 2025.
[34] C. Xiao, P. Zhang, X. Han, G. Xiao, Y. Lin, Z. Zhang, Z. Liu, S. Han, and M. Sun. Inflim: Unveiling the intrinsic capacity of Ilms for understanding extremely long sequences with training-free frree memory, arXiv, 2024.
[35] X. Xiao, H. Ping. C. Zhou, D. Cao, Li, Zhou. S. Li, N. Kanakaris, and P. Bogdan. Neuron based multifractal analysis of neuron interaction dynamics in large models. In International Conference on Learning Representations, 2025.
[36] H. Xiong, Z. Yang, Yu. Y. Zhuge, L. Zhang. J. Zhu, and H. Lu. Streaming video understanding and multi round interaction with memory enhaneedd knowledge, 2025.
[37] C. Xu, W. Ping, P. Xu, Z. Liu, B. Wang. M. Shoeybi, B. Li, and B. Catanzaro. From 128k 4m: Efficient training of ultra-long context large language models, 2025.
[38] P. Xu, W. Ping. X. Wa, C. Xu, Z. Liu, M. Shoeybi, and B. Catanzaro. Chatqa 2: Bridging the gap proprietary Ilms in long context and rag capabilities, 2025.
[39] A. Yang. B. Yang. B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint 241v:2412.15115, 2024.
[40] L. Yang. Z. Yu, T. Zhang. S. Cao, M. Xu, W. Zhang, J. E. Gonzalez, and B. Cui. Buffer of thoughts: Thought- auggent--ugmentte reasoning with large language models. Advances in Neural Information Processing Systems, 1111131999115444 2024.
[41] P. Ye, He. S. Tang, B. Li, T. Chen, Bai, and W. Ouyang. Stimulative training++: Go beyond the performance limits of residual networks, 2023.
[42] P. Ye, C. Huang. M. Shen, T. Chen, Y. Huang, and W. Ouyang. Dynamic model merging with mixture of weights. IEEE Transactions on Circuits and Systems for Video Technology, pages 1-1, 2025.
[43] P. Ye, Li, Y. T. Chen, J. Fan, and W. Ouyang. 3-darts: Beta-decay regularization for differentiable architecture search, 2022.
[44] P. Ye, S. Tang, B. Li, T. Chen, and W. Ouyang. Stimulative training of residual networks: social psychology perspective of loafing, 2022.
[45] P. Zhang, Liu, S. Xiao, N. Shao, Q. Ye, and Z. Dou. Long context compression with activation beacon. In The Thirteenth International Conference on Learning Representations, 2025.
[46] X. Zhang, Y. Chen, S. Hu, Z. Xu, Chen, M. Hao, X. Han, Z. Thai, Wang. Z. Liu, et al. oc-bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages -5577,, 2024.
[47] Z. Zhang, X. Bo, C. Ma, R. Li, X. Chen, Q Dai, J. Zhu, Z. Dong, and J.-R. Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404. 13501, 2024.
[48] Z. Zhang, J. Li, Y. Lan, X. Wang, and H. Wang. An empirical study on prompt compression for large language models, 2025.
[49] D. Zhu, L. Wang, N. Yang, Y. Song, W. Wu, F. Wei, and S. Li. Longemhed: Extending embedding models for long context retrieval, 2024.
[50] Z. Zhu, C. Luo, Z. Shao, F. Gao, H. Xing, Q. Zheng, and Zhang. A Simple yet Effective Layout Token in Large Language Models for Document Understanding. arXiv e-prints, page arXiv:2503. 18434, Mar. 2025.
[51] J. Zylberherg and B. W. Strowbridge. Mechanisms of persistent activity in cortical circuits: possible neural substrates for working memory. Annual review of neuroscience, 40(1):603-627 2017.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All major claims stated in the abstract and introduction are thoroughly substan- tiated by the results and analyses presented Experiments
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Our work's limitations are discussed in Condlusions&
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
No theoretical results are included in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All of the necessary details to make our work reproducible are included in this paper. Our methodology is described in detail in Method 3.2.2 and our experimental setup and implementation details are included in Experiments 4.1 Additionally, all of our code and data will be included in the submission and released upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All of the code and data used in this study as well as the necessary documenta- tion to run it will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https ctspps.cc/pulli///ge hotpe///ggideeeeeeouuuubiiibiai Cotm//umeeeeeeeeeeeeeee ssi onsliicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All of the necessary details for testing in terms of experimental setup and implementation details, including training splits and hyperparameter tuning can be found in
Experiments 4.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Our method inherently training-free, meaning it does not involve stochastic training processes (c.g., random initialization data shuffling) that typically require multiple runs to assess variability. Since the approach operates deterministically on fixed pretrained models, identical inputs and configurations, as demonstrated in Experiments [4. will always produce the same outputs. Results are reproducible across identical hardware and software environments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss the local computing resources utilize in Experiments 4.] and detailed time and costs will be put in Supplementary Materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We reviewed the NeurlPS Code of Ethics and made sure that our paper conforms to it in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
16

---

Our work promotes interdisciplinary research between Al and neuroscience. This may positively impact future research on interpretable and cognitively aligned Al. As methodological contribution, it poses no foreseeable negative societal risks.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We release no models and the data we release is either already publicly available purely the output of an LLM doing OpenlE such data. We believe that this paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credit the owners of all code, models and data used in this work. Much of this information can be found in Experiments 4.1
Guidelines:
17

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All of the code and data assets released alongside our paper are appropriately documented for reproducibility.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper involves crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
18

---

Justification:
Our paper involves no crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The LLMs used, such ChatGPT, are limited to slight polishing and gram- matical corrections of the paper language. They had no role the design of core research methods, data analysis, experimental procedures, or interpretation of results. Therefore, the use LLMs did not have an impact on the scientific rigor, originality or substance of the study, it was not necessary to make formal declaration in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy http::::/eerriss.cc//oofeeeeeee for what should or should not be described.
19

---

A Inference Efficiency Analysis
To quantitatively assess the computational overhead introduced by our proposed method PaceLLM, conduct series of rigorous inference time measurements the Qwen2-7B model using the Qasper task from LongBench- evaluation focuses both absolute inference time and relative time increase compared to baseline methods. The results are summarized Table
representative long-context question answering benchmark. Our
Table 6: Inference Time Comparison on Qwen2-7B (Qasper Task)
Method
Sdpa Attention
Flash Attention
Vanilla
7s31s
7m03s
Activation Beacon
4m42s
4m19s 19s
Ours
6m32s
6m09s
Controlled Time Overhead. Compared to the most efficient bascline (Activation Beacon), our method introduces moderate and controlled increase in inference latency. Specifically, the relative time overhead is approximately 1.37 with SDPA and 1.32 with FlashAttention. However, com- pared to the Vanilla baseline without any memory mechanism, speedup-abes about 13.2% faster under SDPA and 13.4% faster under FlashAttention. This highlights that approach strikes a favorable balance between computational complexity and nemory-enhancee modeling capability.
method achieves significant
Compatibility with Attention Optimizations. All methods benefit from attention-level optimiza- tion. Transitioning from SDPA to FlashAttention yields consistent 6-7% speedup across all setups. Importantly, our method is fully compatible with FlashAttention, demonstrating its practical applicabilitt real-world, performance-critical environments.
Breakdown of Overhead Sources. The primary computational overhead in our method stems from the activation memory mechanism, including dynamic activation storage, similarity- based lookup, and selective activation reconstruction. These components central to the model's ability to capture and reuse long-range dependencies. Nonetheless, they are designed be lightweight, ensuring that the overall throughput remains practical.
Peffomancy Trade-off. The additional inference time is well justified by the perfor- mance gains observed in multiple long-context tasks. Compared to the Vanilla baseline, our method reduces latency while improving comprehension. Compared to the Activation Beacon, we achieve stronger results with acceptable overhead. For latency-sensitive applications, the design of our system offers tunable trade-off between inference efficiency and accuracy.
Summary. PaceLLM maintains operational feasibility with predictable computational cost. It integrates well with widely adopted acceleration techniques such as FlashAttention and provides favorable performance efficiency trade-off, making it suitable for both research and real-world deployment scenarios.
performance-efficienccy
B Detailed Performance on LongBench
Table reports the performance of PaceLLM on a variety of long-context understanding tasks from LongBench in training-free setting. We evaluate two major foundation models models--Qeen-2 and Llama-2 and progressively apply our brain-inspired mechanisms: cortical expert neuron clustering (CE) and persistent activity memory (PA).
Component-wis Improvements. Individually, both and PA contribute positively across most tasks. For Qwen-2, CE enhances performance particularly in Single-Document QA (e.g., NrtvQA improves from 25.38 to 25.87) and Code tasks (e.g., RB-P rises from 46.47 to 46.71). PA, on the other hand, is especially effective in Few-shot Learning (c.g.. TREC from 76.00 to 78,00) and
20

---

Table 7: Performance comparison between PaceLLM and baseline models on LongBench tasks in training-free manner. CE denotes cortical expert neuron clustering and PA means persistent activity memory mechanism.
Sugle Qicunerr
Mozt Decuner QA
Semmontation
Few Fow-shor Leoming
Code
Now
Conkappas
TREE
Vanilla
25.38
62.75
45.16
17.76
$5.30
$4.11
16
49.05
36.88
13.52
76
36.91
76:00
98.16
86.1
70.56
51.65
66:41
50.85
SaiilaaCE
42.30
37.68
54.15
55.41
45.60
36,30
23.55
26.3
28.55
76.50
39.91
45.43
70.61
54,00
06.71
50.36
VanillanltA
43:35
26
3941
70.92
49.00
VailliePN--CE
43.88
36.49
96.53
23.54
2M/12
39.41
70.66
Vanilla
16.55
19.77
35.34
23,92
34.27
26,90
9.28
23.42
26.54
20,85
25.99
24. 43
64.50
88.34
41.21
63,12
58.39
52.36
55.48
VanillaCR
16:00
20.3 30
34.49
38.11
20.51
8.58
23.75
26.3 54
21.11
25
34. 98
64.00
81.34
41.24
63.86
58.00
50.27
55.17
MaillaaPA
17.76
20.83
25.36
24.05
33.35
20:31
9.41
25.15
69
20.96
25.00
28.18
65.00
85.42
41
68.23
58.25
51.73
54.9
VanilaaPN
25.35
34.37
9.33
21.08
63.88
55.25
QA (e.g., HotpotQA from 55.29 to 55.62), aligning with its role in preserving longer contextual dependencies.
For Llama-2, the gains are also evident. CE improves complex QA tasks such as MF-en (froom 35.34 36.26) and long-context comprehension tasks like 2WikiMQA. PA further boosts performance in NrtvQA and TREC. These results demonstrate that each mechanism targets complementary cognitive functions and boosts model reasoning in different ways.
Synergistic Combination. When CE and PA are combined, they consistently lead to the best overall performance across all categories and both models. Notably:
For Qwen-2, Mulii-Doummen QA tasks show the most significant gains: HotpotQA improves from 55.29 to 56.53, and Musique from 36.88 to 37.99. These tasks demand multi-hop reasoning and long-span memory, where our dual mechanisms work jointly capture hierarchical and persistent context.
In summarization tasks such as QMSum and MultiNews, CE+PA achieves or closely approaches the best results (e.g., MultiNews from 26.60 to 26.92).
Few-shot Leurning tasks also benefit, where CE+PA maintains the highest scores in TREC and SAMSum.
Cross-Model Robustness. Our approach generalizes well across architectures. Although Llama-2 starts from a lower baseline than Qwen-2, it benefits significantly from our enhancements:
The CE+PA combination raises performance in NrtvQA by +1.69, Qasper by +1.49, and MF-en by +1.10 over vanilla Llama-2.
Multi Documenn QA and Summarization also show consistent gains (e.g., 2WikiMQA from 26.90 to 27.57, MultiNews from 25.90 to 26.25).
Few-shot tasks exhibit either improved performance, indicating the method's stability.
Summary. Overall, the experimental results underscore the effectiveness of brain-inspired design. The CE mechanism enhances specialized, local processing by routing to expert neuron clusters, while PA extends the temporal memory span. Their integration leads robust performance improvements across 15+ diverse tasks without any parameter update, setting a new standard for training-free long- context understanding, Notably, these results are achieved with minimal computational overhead (as discussed in Section ensuring practical deployment feasibility.
C Detailed Methodology of PaceLLM
C.1 Persistent Activity (PA)-Activationn Working Memory Bank Operations
Algorithm Algurithmmddescribees the working memory mechanism of PaceLLM, which dynamically enhances current FFN activations using memory bank. It consists of three key phases: retrieval, enhancement, and memory update.
Input: Activation tensor X dg FFN dimension), and a memory bank (K, V, u) storing previous activation keys, values, and usage counters.
E Bx (where B is batch size, L is sequence length, and
21

---

Algorithm Persistent Activity (PA)-Activation Working Memory Bank Operations
Require: Current activation BBxLxdd memory bank {K, V,uj u}
Ensure: Enhanced activation O, Updated memory bank
1: X < (XX) (X E (BxxL)xdg
2: Initialize fat 0
3: for chunk X, E Partition( Xna, C) do
4: Retrieval
5: Compute similarity matrix: sim +
X.K" [
(sim E (RCM
6: Stop I (sim, k) {k nearest)
7: Steg aaa +- TopK(-sim, k') (k' negative)
8: Enhancement
9:
for f- to C do POI $
V [.
11:
" 80B + + E V[ [i
12:
max (t [i, :):)) > mnigh then
O: < 1000 + Xp
14:
else if Uwow < max (Sopp i :): <
Onigh then
15:
O, A X [i]) A4 nog
16:
else
O, [
18:
end if
O
[i] f O;
20:
end for
Update Phase
22:
Compute chunk mean: Me + 5
E i= X
23:
Stopk Topk +- TopK(sim, 1t)
24:
if
f==1 Stopk >> Onigh then
Update usage: up[ u [opk
26:
else if Bwo
<
1
Σラ= Stopk S Ohigh then
1
27:
K
[opk]
Aug(Ke 1Pp51 pem)
28:
lopk
Aug( V 1ops
29:
else
30:
Find LRU slots: j - (r)
31:
Replace: K[ x V(i*]
Onu
32: end if
33: end for
34: - Reshape O B, L, dx)
35: return O, (K, u,
Output: Enhanced activations O and updated memory bank.
This algorithm enables low-overhead, comtext-sensitive memory usage for LLMs, simulating short- term working memory consolidation and reuse mechanisms.
C.2 Cortical Expert Clustering (CE)
Algorithm 2 shows how PaceLLM leverages cortical-like modularity by clustering FFN neurons across layers into interpretable experts using a constrained KMeans method.
Input: Pretrained model M and target number of experts K.
Output: Updated model M' with clustered and reordered FFN weights.
Explanation of key steps:
1. For each layer, extract FFN weights ( (input projection) and w!! (output projection).
22

---

Algorithm 2 Cortical Expert Clustering (CE)
Require: Pretrained model M, Number of experts K
1: Initialize empty state dictionary 8
2: for layer E {I...., } do
3: Extract FFN weights ( w
4: if cluster indices 71)) not cached then
5: Compute π(1) + KMeansConstrained ( K)
6: Cache T()) to disk
7:
end if
8: WTew + Rearrangel (1)
(()))
9: W + Rearrange W (1) (1)
10: Update S with Wiew W
11: end for
12:
return Model with updated weights M'
2. If the clustering result (()) is not cached, apply constrained KMeans to group neurons into expert clusters. This ensures load balance and specialization.
3. Rearrange the weight matrices according to cluster assignments (4), so that expert-based routing be implemented efficiently during inference.
4. Update the model's weight state dictionary with the new clustered weights.
This modularization allows PaceLLM to activate specific "experts" during computation and aligns with the cognitive hypothesis of cortical column specialization.
D Detailed Explanation of KMeann-Constraine Clustering and LRU Update Strategy
D.1 KMMann and Constrained KMeans Clustering for Expert Partitioning
D.1.1
Standard KMeans Clustering
Given N data points {x:],r C IRd, KMeans aims to find K clusters {C&) ) minimizing the intra-cluster variance:
and centroids {
JK
K
min
1x - 18-11 12 where t4
Σ Xi.
(12)
CC
EC
X.ECi
Iterative procedure:
Assignment:
Cs
xx k arg min ||x; 11yl2
(13)
Update: Hy --
x;
(14)
[Cp)
x.EC,
Repeat until convergence.
D.1.2 Constrained KMeans Clustering
To prevent cluster imbalance, we impose cardinality constraints:
Lmin < |cpl S Lmax: Vk (L...., K}
(15)
Special cases:
23

---

Equal-size constraint: [Cx] NV K
Upper-bound constraint: |Csl I U
Heuristic optimization: Let d3k ||x:
Ha We define the cluster assignment function
π ( arg min das, REA,
A; {k: |Cal <
Lmax]
(16)
That is, each x; is assigned the nearest cluster among those with remaining capacity.
D.1.3 Application in PaCeLLM
In FFN layers, each neuron corresponds to a row w, E Ra of the weight matrix W E R To enable sparse expert routing, we perform constrained clustering:
dg W=
Constrained KMeans
{ where |& di
(17)
Each expert Eg serves as functional block activated conditionally during inference.
Why clustering in PaCeLLM?
Reduces redundant neuron computation via routing.
Ensures fair expert load balancing, avoiding expert collapse.
Enables structure-awars specialization, neurons with similar semantic roles grouped.
D.2
Least Recently Used (LRU) Update Strategy for Memory Management
D.2.1
Mathematical Formulation
Let memory bank M = {(k,, Va, u store key-value pairs and their usage counters. At each time step t:
00 (1(1) = (t-(t11++ +
if slot is accessed otherwise
(18)
When writing new memory (Krow, Voew), we check similarity:
max sim k;) < 8low = need replacement We replace the least recently used slot:
(19)
arg max Wi, (ki-, Vi+) (Knew, Vnew), Ug-
0
(20)
D.2.2 Application in PaceLLM
To model human-like memory with decay, PaceLLM uses bounded-size memory M and LRU strategy for updates:
Prevents unbounded memory growth.
Automatically decays outdated context.
Encourages dynamic adaptation to new content.
Why LRU in PaceLLM?
Emulates neural memory fading (forgetting).
Reduces retrieval noise by replacing stale keys.

---

Aligns with human working memory dynamics, where recent tokens dominate attention.
Together, constrained KMeans and LRU form the foundation of PaceLLM's M architecture:
Expert Routing + Working Memory Adaptation + Efficient and Continual Inference
E
Extra Experiments on More Models
Table 8: Performance comparison between PaceLLLM and Mistral-7B-Instruutt-v on LongBench tasks in training- free manner. CE denotes cortical expert neuron clustering and PA means persistent activity memory mechanism.
Singl-DDeemenn QA
Moli-DDecmmmn Q4
Sammeictin
Fon-dhit leaning
Cooe
GerRopunt
TIRWOO
Vanilla
20-52
41:12
51.78
41
40.87
90.51
28.34
9.34
35.88
35 55
27.88
26.26
76.0
36
89
47.33
70.34
99
5067
99
VanilarCE
26.15
43.44
45.45
38.35
$5.91
56.31
35.42
49.21
35.53
23.44
26.78
28.55
76.8
89. 41
45
71.11
53.76
16.13
48.05
SanilisaRA
20:30
41.16
53.76
4141
5061
30.64
28.06
59.00
36.40
25 64
21.27
79.77
76.0
59 56
47.27
7094
59.14
8083
60.30
NamilaePA+CE
43.8
42.1
56.97
36.3
Table 9: Performance comparison of more baseline models and our method (CE + PA) on LongBench, aggregated into major task categories. Results show consistent improvements across architectures in a training-free manner.
Model
SQA
MQA
Sum.
FSI.
Cod.
17.18
12.15
23.35
71,46
32.30
18.48
12.97
23.49
72.32
33.41
Llama-3. -lama-3.1-8B--me I-sttuutttt
24.22
15.04
28.21
69.49
58.44
Llama-3. 1etaaa...--B-BBB Inssrruttuttt
24.31
15.80
28.47
69.85
59.59
Results on LongBench with Mistral. Table 8 reports the training-free evaluation results of the Mistral model across different LongBench tasks. We observe that both the cortical expert neuron clustering (CE) and persistent activity memory (PA) modules individually enhance the base Mistral model in different task categories.
Specifically, CE brings notable improvements in multi-document QA, with performance in 2WikiMQA and Musique boosted by up to 16.8% and 7.1% respectively compared to the vanilla model. This confirms CE's effectiveness capturing complex cross-document reasoning patterns. On the other hand, PA contributes consistently across all categories, particularly maintaining even slightly improving the base performance in summarization and few-shot tasks, while preserving high accuracy in code reasoning.
When both mechanisms are combined (CE+PA), the model achieves the best overall results, out- performing the vanilla baseline in 13 out of 16 subtasks. Notably, the average accuracy in Single- Document QA improves from 41.56% to 42.61%, and in Multi-Document QA from 39.249 to 49.64%, representing a 10.4% absolute gain. Summarization and code tasks also benefit from the combination, indicating that the two brain-inspired components are complementary.
These results demonstrate that our proposed architecture not only generalizes well across task types but also significantly strengthens the model's long-range reasoning capability in a fully training-free setting.
Results LongBench with Qwen2.5 and Llama3.1. Table 9 presents the performance of our method when applied to two state-of-the-art LLMs Qwen2.5- 1-B22.5--BBB-- 1-B--Insrruc and Llama-3. 3.18-BB Instruct under the same training- free setup. Despite their different architectures and training corpora, both models exhibit consistent improvements across all task categories after integrating our brain-inspired mechanisms.
For Qwen2.5- 1rren2.5-1BB-nsstrucct the integration of CE and PA leads to gains in every domain, with particularly notable improvements in multi-document QA (+0.82) and code reasoning (+1.11). The
25

---

model also achieves higher accuracy in few-shot learning, suggesting that our memory mechanism enhances its ability leverage contextual demonstrations without retraining.
Similarly, on 1- categories,
our method consistently boosts performance across all five though the base model already performs strongly in code and single-document QA. The most significant gains occur in multi-document QA (+0.76) and summarization (+0.26), indicating that CE and PA help compensate for limitations long-context integration, especially in models with smaller context windows or less optimized retrieval capabilities.
These results demonstrate that PaceLLMM's design is not only effective but also highly generalizable, delivering consistent benefits across diverse model families and scales. The fact that both heavily optimized commercial-graen grade model (Qwen) and compact open-weight model (Llama) benefit from our approach underscores its potential as a universal, plug-and-play enhancement for long-context understanding.
26


Reviews:
[{"review_id": "S0lFWvRTQr", "reviewer": "Reviewer_8wok", "summary": "PaceLLM introduces a brain-inspired approach to enhance long-context understanding in Large Language Models through two key mechanisms: \n(1) an Activation Memory Bank (AMB) that implements Persistent Activity (PA) by caching and reusing FFN activations at inference time, mimicking working memory in the prefrontal cortex.\n(2) Cortical Expert (CE) clustering that reorganizes FFN weights into semantic modules through one-time preprocessing.\n\nThe authors identify two core limitations in current LLMs: transient neural activations causing information decay and unstructured FFN weights leading to semantic fragmentation. Their solution operates at the activation level within feed-forward networks, offering a training-free, plug-and-play enhancement compatible with existing models. The method achieves improvements on various benchmarks.", "questions": "Here are some questions that I had:\n1. Can you provide theoretical justification for why M=100 memory slots are sufficient to handle 200K token contexts?\n2. How does the method compare against Recurrent Memory Transformer (RMT) and Hierarchical Memory Transformer (HMT), which seem to address similar problems?\n3. What is the actual wall-clock time overhead in production settings with batched inference? The paper lacks concrete timing measurements.\n4. Why does CE alone show negative results in Table 1, yet becomes beneficial when combined with AMB? This suggests CE may not be necessary at all.\n5. How does performance scale with model size? Would a 70B model require proportionally more memory slots?\n6. Can you demonstrate the method's effectiveness on tasks beyond QA, such as book summarization, multi-document reasoning, or code understanding?", "limitations": "The authors briefly mention computational costs but fail to address several critical limitations:\n1. Memory bank capacity: No analysis of how performance degrades when memory is full or how to scale capacity for longer contexts or larger models.\n2. Task generalization: Unclear which types of documents or tasks benefit most from this approach. The method may be overfitted to QA-style tasks.\n3. Theoretical understanding: The lack of formal analysis means we don't understand when or why the method works, making it difficult to predict failure modes or optimize further.\n4. Performance: Unclear what performance impact there might be.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I like the following aspects:\n1. Novel approach: Focusing on FFN layers rather than attention mechanisms represents a fresh perspective on the long-context problem. The activation-level storage mechanism is creative, simple and well-motivated.\n2. Practical design: The training-free, plug-and-play nature makes this easy to use for open source models. The method can be applied to existing pretrained models without any retraining, requiring only a one-time clustering preprocessing step.\n3.Strong empirical results: Consistent improvements across multiple benchmarks (LongBench, ∞-Bench).\n4. Clear presentation: Figure 1's brain-computation parallel effectively motivates the approach, and the visualization in Figure 4 provides compelling evidence of the mechanism's effectiveness.\n\nI think the following could be improved:\n1. Lack of theoretical foundation: The paper provides no formal analysis for critical design choices - why cosine similarity, why these specific thresholds (0.7, 0.3), or why 100 memory slots suffice for 200K tokens. The biological inspiration doesn't substitute for mathematical rigor.\n2. Limited experimental scope\n  - Only tested on 7B models - scalability to larger models unclear\n  - Evaluation focused primarily on QA tasks - missing diverse long-context applications\n  - No comparison with recent memory-augmented transformers (RMT, HMT, Cached Transformers)\n  - Computational overhead analysis is not particularly deep\n3. Technical clarity issues: The interaction between CE and AMB is complex and poorly explained. The choice of layers 13 and 27 appears arbitrary without comprehensive ablation. The \"noise injection\" mechanism via bottom-k' selections lacks proper motivation.", "was_revised": true, "final_justification": "The authors responded to some of my criticisms and clarified some of my questions. This ameliorated some of my doubts and thus I have increased the score.", "rebuttal": "Dear Reviewer 8wok,\n\nMany thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.\n***\n>[W1,Q1]Theoretical Justification for Design Choices (Cosine Similarity, Thresholds, Memory Slot Size)\n\n[A1]Our work is primarily biologically inspired rather than derived from fine-grained mathematical modeling. Nonetheless, our module’s hyperparameter design adheres to commonly accepted practices in the AI community.\n1. Choice of Cosine Similarity: Cosine similarity is a widely used and computationally efficient method for measuring semantic similarity, like in representation learning and contrastive learning frameworks [1,2,3,4]. Its simplicity and effectiveness make it a standard choice in many existing models.\n2. Threshold Selection: The thresholds are chosen within the normalized range of cosine similarity scores (0,1). The values (0.7, 0.3) serve as a representative even partition in this range and have been validated as effective settings in our ablation study. We also provide results for alternative threshold settings to demonstrate robustness.\n3. Memory Slot Size\n- The decision to use 100 memory slots is a trade-off between performance and efficiency. Our preliminary experiments show that increasing the number of slots to 500 or 1000 yields negligible performance gains while significantly increasing memory usage and slowing down inference. Reducing the number of slots below 100, on the other hand, leads to noticeable performance drops.\n- Moreover, our method is one of the first to explore memory mechanisms within FFN layers for long-context modeling. Achieving 200K token context processing with only 100 memory slots at the FFN level is a strong result. This efficiency can be explained by prior work[5], which shows that **FFN activations are highly sparse—only a small subset of units contribute significantly to semantic activation for a given input**. Hence, a compact memory of 100 slots is often sufficient to capture the critical activations needed for long-range semantic retention.\n- This aligns with the notion of working memory[6] in neuroscience, which is characterized by high efficiency and compression: a small set of high-dimensional, semantically rich representations can be more effective than a large, redundant store.\n\n[1]Tomas Mikolov et al. \"Efficient Estimation of Word Representations in Vector Space.\" arXiv preprint arXiv:1301.3781, 2013.\n\n[2]Ting Chen et al. \"A Simple Framework for Contrastive Learning of Visual Representations.\" ICML 2020.\n\n[3]Tianyu Gao et al. \"SimCSE: Simple Contrastive Learning of Sentence Embeddings.\" EMNLP 2021.\n\n[4]Alec Radford et al. \"Learning Transferable Visual Models From Natural Language Supervision.\" ICML 2021.\n\n[5]Mor Geva et al. \"Transformer Feed-Forward Layers Are Key-Value Memories.\" EMNLP 2021.\n\n[6]J. Zylberberg et al. \"Mechanisms of Persistent Activity in Cortical Circuits: Possible Neural Substrates for Working Memory.\" Annual review of neuroscience 2017.\n***\n>[W2]Limited Experimental Scope\n\nWe appreciate your detailed comments and will address them one by one.\n- [Q2]No comparison with recent memory-augmented transformers (RMT, HMT, Cached Transformers).\n\n[A2]Thanks for highlighting relevant memory-augmented Transformer baselines such as Recurrent Memory Transformer (RMT)[7] and Hierarchical Memory Transformer (HMT)[8]. Both methods tackle long-context processing by introducing recurrence and structured memory hierarchies at the sequence level.\n1. RMT augments Transformer-XL with learnable memory tokens passed between segments in a recurrent fashion. It enables segment-level recurrence without modifying the Transformer architecture, and demonstrates improved performance on algorithmic tasks and language modeling.\n2. HMT mimics the brain's memory hierarchy (sensory, short-term, and long-term memory) and incorporates a hierarchical memory retrieval mechanism to enhance long-context modeling. HMT is designed as a model-agnostic plug-in framework that works with a variety of backbone architectures.\n3. In contrast, our method introduces a plug-and-play memory mechanism within the FFN layers, leveraging activation sparsity to store and retrieve key representations. It is **orthogonal** to the recurrence-based designs of RMT and HMT and **does not modify attention or sequence processing**. Our approach is **compatible** with these methods and can be integrated to jointly enhance long-context modeling. Rather than replacing existing strategies, it offers a complementary, biologically inspired perspective.\n- [Q3]Computational overhead analysis is not particularly deep.\n\n[A3]We apologize that due to current resource limitations, we are unable to conduct large-scale, production-level batched inference. However, our small-batch experiments indicate that our method does not introduce more overhead or performance degradation during batched inference.\nWe test on Qwen2-7B and Qasper in LongBench.\n|Batch_size|Time|Performance|\n|---------------|------|-----------------|\n|1| 25m22s| 43.88|\n|2| 12m45s |43.88  |\n|3| 8m37s| 43.88|\n|4 | 6m22s |43.88 |\n- [Q5,L1]Scalability to Larger Models\n\n[A4]**Our method remains effective as model size increases**. Due to the inherent sparsity of activations[5] in the FFN layers, larger models do not necessarily require a proportional increase in the number of memory slots. The experimental results on larger models are as follows:\n|Model|SQA|MQA|Sum.|FSL|Cod.|\n|---------------|------------|----------|----------|----------|----------|\n|Qwen2.5-14B-Instruct | 17.18 | 12.15| 23.35| 71.46| 32.30|\n|Qwen2.5-14B-Instruct+Ours | **18.48** |**12.97**  | **23.49** | **72.32** | **33.41** |\n|Llama-3.1-8B-Instruct | 24.22 | 15.04 | 28.21 | 69.49 | 58.44 |\n|Llama-3.1-8B-Instruct+Ours | **24.31** |**15.80**  | **28.47** | **69.85** | **59.59** |\n- [Q6,L2]Evaluation on Diverse Long-context Applications\n\n[A5]Although LongBench and ∞-Bench are framed as QA tasks, the included benchmarks already cover a range of long-context tasks such as summarization, multi-document reasoning, and code understanding. Our model has demonstrated its capability to effectively address such questions, exhibiting robust competence across diverse long-context scenarios.\n\n[7]Aydar Bulatov et al. \"Recurrent Memory Transformer.\" NeurIPS 2022.\n\n[8]Zifan He et al. \"HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing.\" NAACL 2025.\n***\n>[W3]Technical Clarity Issues\n1. [Q4]The interaction between CE and AMB\n\n[A6]\n- CE categorizes FFN units into expert groups through clustering. While CE alone does not yield significant performance gains, it enables semantic consistency within expert regions, which becomes beneficial when paired with AMB. **AMB operates at the cluster level**, performing similarity-based memory retrieval and weighting, allowing each expert to specialize and contribute effectively.\n- The effectiveness of CE is best demonstrated through the comparison among baseline+PA, baseline+CE, and baseline+PA+CE. Neither PA nor CE alone leads to significant improvements, but their combination achieves the best performance. This synergy indicates that CE is necessary, as it provides the structural foundation for AMB to function optimally.\n2. The choice of layers 13 and 27\n\n[A7]13,27 is just a representative setting of the selection, and results of other Settings are presented in ablation experiments.\n\n3. The \"noise injection\" mechanism via bottom-k' selections\n\n[A8]\n- The use of bottom-k′ activations as a form of noise injection is motivated by principles from information theory, particularly the **Information Bottleneck (IB) Theory**[9]. According to the IB framework, a desirable representation should retain task-relevant information while discarding redundant or irrelevant input features.\n- In our case, introducing bottom-k′ activations—alongside the top-k informative ones—can be interpreted as deliberately injecting a small amount of non-informative or noisy signals. This encourages the representation to decentralize and avoids the model from overfitting to a narrow subset of high-activation (top-k) semantic features or becoming overly reliant on specific pathways, which could degrade generalization.\n- Moreover, this noise injection increases the entropy of the activation distribution, enhancing the diversity and expressive capacity of the FFN activation space.\n\n[9]Naftali Tishby et al. \"The information bottleneck method.\" In Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing.\n***\n>[L3,L4]Theoretical understanding: The lack of formal analysis means we don't understand when or why the method works, making it difficult to predict failure modes or optimize further. Performance: Unclear what performance impact there might be.\n\n[A9]Our method is a **plug-and-play** module that has been validated across various models and datasets. It consistently shows performance improvements over the baseline in most settings, including additional experiments beyond the main evaluation.\nThese results demonstrate the broad effectiveness and practical robustness of our approach.\n***\nThanks for your comments and suggestions, we will supplement our revision. Feel free to let us know if you have any further questions or concerns :-)."}, {"review_id": "YU9JYe7ZzI", "reviewer": "Reviewer_rZpP", "summary": "This paper proposes PaceLLM, a novel brain-inspired approach to enhance the long-context capabilities of LLMs. The approach consists of two key components: (1) the Persistent Activity (PA) Mechanism, which mimics the prefrontal cortex’s working memory by introducing an Activation Memory Bank (AMB) that dynamically stores, retrieves activations, and (2) the Cortical Expert (CE), which clusters and reorganizes FFNs. Experiments demonstrate the performance improvement of the proposed method.", "questions": "1. To help readers better understand, can you provide a brief introduction to the baselines (LongLLMLingua, SnapKV, and Activation Beacon)? If space constraints prevent this from being included in the main body, it can be included in the appendix.\n2. Can the authors provide an intuitive explanation for the activation update process defined in Equation (6)? And similarly, can you provide an intuitive explanation for Memory Update under three similarity conditions?\n3. Why is the experimental evaluation incomplete? In the training-free setting, results are reported for both Qwen-2-7B-Instruct and Llama-2-7B-chat, but for the low-cost fine-tuning setting, only Qwen-2-7B-Instruct is evaluated. Moreover, in Table 2, Activation Beacon, Activation Beacon + CE, and Activation Beacon + CE + PA are reported, but not Activation Beacon + PA.", "limitations": "As mentioned in Weaknesses.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper draws inspiration from the field of neuroscience, introducing the brain's working memory and the cortical modularity into LLM long text modeling.\n\n2. The proposed module can be directly integrated into existing LLMs without retraining, making deployment simple.\n\n**Weaknesses:**\n\n1. The motivations behind the two proposed modules (i.e., \"transient neural activations cause information to fade over time\" and \"unstructured FFN weights may fragment semantics across tokens\") appear to be speculative assumptions rather than empirically established facts. The paper lacks any prior experiments to demonstrate that these issues genuinely exist in baseline models.\n2. The use of mathematical notations in the paper is confusing. For instance, are μ_i^{pos} and \\overline{S^{top}} the same? Do they both refer to the mean vector of top activations? Furthermore, some notations lack necessary definitions and explanations when first introduced. For example, θhigh and θlow are not explained in line 141, and μ_j in formula (9) is not defined.\n3. The study's experimental scope was limited by the relatively small number of tested models, potentially affecting the generalizability of the findings.\n4. As shown in Section 4.4 (Ablation Studies), different tasks exhibit high sensitivity to the choice of layer deployment positions and Fusion Threshold, with model performance varing significantly under changes in these hyperparameters. This indicates that the actual effectiveness of PaceLLM depends on task-specific configurations and manual parameter tuning, which may impact its robustness and usability in real-world applications.\n5. While the experiments demonstrate improvements on benchmark tasks, they do not provide direct evidence that the performance gains are causally attributable to the hypothesized mechanisms—i.e., that PA addresses transient activation decay or that CE resolves semantic fragmentation due to unstructured FFN weights. Therefore, the experimental findings do not sufficiently validate the core assumptions of the proposed design.\n6. The authors claim that *“PaceLLM has not compromised its general language understanding capabilities”*. However, this conclusion is drawn based solely on the model’s performance on the MMLU benchmark. To convincingly support the claim of not compromising general capabilities, further evaluations are needed on more benchmarks, including commonsense reasoning, mathematical reasoning, and general QA or instruction-following datasets. Without such evidence, the conclusion remains somewhat limited in scope.", "was_revised": true, "final_justification": "After reading the rebuttal and considering the discussion, I appreciate the authors' efforts to clarify several aspects of their work, including:\n\n1. Additional experiments on more models and general capability benchmarks.\n\n2. Improved explanation of the Activation and Memory Update mechanisms.\n\nHowever, the paper still has the following important issues:\n\n1. The paper proposes two modules based on assumptions that are not empirically validated. Specifically, no prior experiments are presented to show that (1) Transient neural activations cause information to fade over time; (2) Unstructured FFN weights fragment semantics across tokens.\n\n2. This experimental result does not convincingly demonstrate that CE resolves semantic fragmentation caused by unstructured FFN weights.\n\nOverall, while the paper introduces potentially interesting components, the central motivations appear speculative, and the experiments do not directly support that the proposed modules resolve the hypothesized issues. This weakens the conceptual clarity and impact of the work.", "rebuttal": "Dear Reviewer rZpP,\n\nMany thanks to your professional, detailed, and valuable reviews. We're going to response to your concerns one by one.\n***\n>[W1]The motivations behind the two proposed modules appear to be speculative assumptions rather than empirically established facts. \n\n[A1]\n1. Our work is brain-inspired, and the motivations are supported by extensive theoretical and empirical findings in neuroscience [1,2]. In the AI domain, there exists a substantial body of brain-inspired research[3,4].\n2. We acknowledge that the current version of the paper lacks direct empirical demonstrations that issues exist in baseline models. However, several LLM interpretability studies [5,6] have discussed these phenomena. We appreciate the reviewer’s suggestion and will include relevant background and evidence in the revised manuscript.\n3. The strong and consistent empirical results across diverse settings and larger models support the validity of our assumptions.\n\n[1]J. Zylberberg et al. \"Mechanisms of Persistent Activity in Cortical Circuits: Possible Neural Substrates for Working Memory.\" Annual review of neuroscience 2017.\n\n[2]G. Auda et al. \"Modular neural networks: a survey.\" International journal of neural systems 1999.\n\n[3]Zifan He et al. \"HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing.\" NAACL 2025.\n\n[4]P. Das et al. \"Larimar: Large language models with episodic memory control.\" ICML 2024.\n\n[5]Mor Geva et al. \"Transformer Feed-Forward Layers Are Key-Value Memories.\" EMNLP 2021.\n\n[6]Damai Dai et al. \"Knowledge Neurons in Pretrained Transformers.\" ACL 2022.\n>[W2]The use of mathematical notations in the paper is confusing.\n\n[A2]We appreciate your careful reading and helpful feedback on the clarity of mathematical notations.\n- $μ_i^{pos}$ refers to the mean vector of the top-k activations in the current step, whereas $\\overline{S^{top}}$ denotes the similarity scores between the current activation and the top-k activations stored in AMB.\n- $θ_\\text{high}$ and $θ_\\text{low}$ are similarity thresholds used in our experimental setup, both ranging between 0 and 1.\n- $μ_j$ refers to the cluster centroid of the j-th expert group in the KMeans partitioning. More detailed definitions are provided in the supplementary material (Section D.1, Section C).\n- Thanks again for pointing out these issues. In the revised version, we will add clear definitions and explanations when each symbol is first introduced.\n- Lastly, the final version of the paper will be proofread and polished by a senior researcher with experience publishing more than 10 papers in top-tier conferences and journals, to ensure clarity and precision in all mathematical expressions.\n\n>[W3]The study's experimental scope was limited by the relatively small number of tested models.\n\n[A3]Our method remains effective as model size increases. Experimental results are as follows:\n|Model|SQA|MQA|Sum.|FSL|Cod.|\n|-|-|-|-|-|-|\n|Qwen2.5-14B-Instruct|17.18|12.15|23.35|71.46|32.30|\n|Qwen2.5-14B-Instruct+Ours|**18.48**|**12.97**|**23.49**|**72.32**|**33.41**|\n|Llama-3.1-8B-Instruct|24.22|15.04\t|28.21|69.49|58.44|\n|Llama-3.1-8B-Instruct+Ours|**24.31**|**15.80**|**28.47**|**69.85**|**59.59**|\n>[W4]Different tasks exhibit high sensitivity to the hyperparameters.\n\n[A4]We appreciate your insightful comment. We would like to emphasize that our method consistently improves performance across most parameter settings, and each dataset has its own optimal configuration.\n1. Task-specific variation is a common challenge in current long-context modeling. In our work, we explicitly categorize mainstream tasks and analyze their differing demands.\n- Summarization tasks require strong global coherence across long documents, and thus benefit from slow and stable memory updates that preserve long-term semantics.\n- Code tasks, which often involve abrupt semantic shifts between tokens, benefit more from faster memory updates and shorter memory retention.\n2. Our ablation studies provide heuristic guidance for selecting hyperparameters based on task type, which we believe can assist future work.\n3. As a future direction, we aim to design more elegant mechanisms for task-aware adaptation and automatic hyperparameter tuning, to further improve robustness and generalizability across diverse applications.\n\n>[W5]The experimental findings do not sufficiently validate the core assumptions of the proposed design.\n\n[A5]\n- For the PA module, its ability to address transient activation decay is directly illustrated in Figure 1. The visualization shows that later activations can successfully retrieve and align with semantically similar historical activations.\n- For the CE module, we provide different input and **semantically clustered weight**:\n|FFN CE|James Chadwick|Manhattan Project|\n|-|-|-|\n|Profession|0.4|0.0|\n|Address|0.1|0.5|\n|Education|0.4|0.0|\n|History|0.1|0.5|\n\nThis clustering enables the model to generate input-dependent expert weights, leading to more structured and disentangled activations. Consequently, the subsequent AMB module can retrieve and process information **at the cluster level** more effectively, thereby mitigating semantic fragmentation caused by unstructured FFN representations.\n>[W6] Further evaluations are needed on more benchmarks.\n\n[A6]Evaluations on more benchmarks are as follows:\n| |ARC-C[7]|BoolQ[8]|GSMK[9]|\n|-|-|-|-|\n|Qwen2-7B-Instruct|62.7|87.1|76.0|\n|Activation Beacon|62.7|87.2|76.2|\n|Ours|**62.9**|**87.9**|**77.1**|\n\n[7]Bhakthavatsalam et al. \"Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge.\" arXiv preprint arXiv:2102.03315, 2021.\n\n[8]Clark et al. \"BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions.\" NAACL 2019.\n\n[9]Cobbe et al. \"Training Verifiers to Solve Math Word Problems.\" arXiv preprint arXiv:2110.14168, 2021.\n***\n>[Q1]A brief introduction to the baselines.\n\n[A7]\n- LongLLMLingua proposes a prompt compression framework that significantly improves the density of critical information in long-context inputs.\n- SnapKV introduces a tuning-free KV cache compression technique based on the observation of attention distribution consistency.\n- Activation Beacon proposes a lightweight plug-in module for Transformers that compresses long contexts into compact activations.\n\n More details will be included in the appendix.\n>[Q2]Intuitive explanations for the Activation and Memory Update.\n\n[A8]The principles can be intuitively understood with the aid of Figure 2.\n\n🔹Activation Update:\nWhen a new intermediate activation arrives, we extract it and compute its similarity with the stored historical activations in the AMB. Based on the top-k most similar activations and their similarity scores, we update the current activation according to three different conditions:\n- High Similarity (above upper threshold):\nIf the top-k historical activations are highly similar to the current one, this indicates that the semantic information carried by the current activation has already occurred in the past and is strongly repeated. In this case, it is intuitive to **reuse** the historical semantic representation directly by injecting the top-k activations into the current one without modification.\n- Moderate Similarity (between upper and lower thresholds):\nIf the similarity is moderate, the current semantic information is partially related to the historical one, but not identical.\nIn this case, we enrich the current activation by **blending** it with the related historical activations, performing a weighted integration of both to enhance the model’s long-range contextual understanding and memory.\n- Low Similarity (below lower threshold):\nIf the similarity is very low, the historical semantic content is considered irrelevant to the current activation. The model simply **skips** historical reuse and proceeds with the original activation.\n- Additionally, to prevent the model from overfitting to memorized content, we inject a small amount of **noise** during the first two update modes.\n\n🔹 Memory Update:\nThe memory update process in AMB mirrors the logic above:\n- High Similarity (reuse):\nWhen the historical activations are highly similar, **no update** to the AMB is necessary, as the existing memory already captures the relevant semantic content.\n- Moderate Similarity (merge):\nWhen moderate similarity is detected, the fused activation is used to **replace** the corresponding top-k historical entries in the memory slots.\n- Low Similarity (new memory):\nIf the current activation encodes entirely new information, it needs to be **added** to memory. We employ an LRU (Least Recently Used) strategy to replace the least recently accessed memory slot with the new activation. Details of this memory management mechanism can be found in Appendix D.2.\n\n>[Q3]Supplementary experimental evaluation.\n\n[A9]\n1. Our experiments follows the original baseline Activation Beacon, and we aligned our settings accordingly using the checkpoints provided by the authors. However, the authors of Activation Beacon did not release the fine-tuned checkpoint for LLaMA-2-7B-chat, and unfortunately, **our direct requests for access received no response**. This is why our low-cost fine-tuning results are only reported on Qwen-2-7B-Instruct.\n2. Regarding Table 2, we appreciate your observation. The missing Activation Beacon + PA results were indeed evaluated and are as follows:\n| |SQA|MQA|Sum.|FSL|Cod.|\n|-|-|-|-|-|-|\n|Activation Beacon+PA|41.10|42.80|27.90|69.31|67.51|\n***\nThank you again for helping us improve the paper and hope our response can resolve your concerns! Please let us know if you have any further questions. We will be actively available until the end of rebuttal period. If you feel your concerns are addressed, please consider reevaluating our work. Looking forward to hearing from you :-) !"}, {"review_id": "tJvSUBK5nD", "reviewer": "Reviewer_RyQo", "summary": "This paper introduces a modification to pretrained LLM, called PaceLLM, which adds a small module between the two layers of the feed-forward net (FFN) of some layers. Specifically, they make two modifications: \n1) Add an Activation Memory Bank (AMB): This keeps M=100 memories of output activations $h =\\sigma(W_1 x)$  of the first layer of the FFN module, based on which ones had the highest overlap with activations for new datapoints. \n2) Add a Cortical Expert (CE) module, which uses k-mean clustering FFN neurons with similar activation and rearranging the FFN weights to put cluster neurons together. they argue this creates localized experts in the weights. \n\nThey show that these two modifications combined lead to significant improvement in long-context tasks such as LongBench and needle-in-a-haystack.", "questions": "1) The role of CE is not completely clear to me. If it were just rearranging weights it would not have any impact, because no weights are discarded, right? But from Fig 2 I assume that the memory similarities are computed at the level of each cluster, is that so? If so, it would make sense, as it would balance the weight we are giving to each cluster of activations. \n2) Is there any smart routing involved for the similarity to each expert? Fig 2 shows only a weighted sum. What are the weights? \n3) What is the compute overhead for dense PaceLLM? Appendix A mentions times for the sparse 2 layer modification, but not the dense one. \n4) you have impressive results for 3 tasks, mentioned at the beginning, but for the rest, you get only 1-2%. What do you think distinguishes those tasks from the rest and why does your model work better for those?", "limitations": "Even though I like the approach and the steps seem fairly optimized, the overhead of 30% for a mere 2 layer modification with just 100 memories raises eyebrows. It's good as a first attempt, but it needs to be made much more efficient to make it practical. Ideally, one would like to keep thousands of memories.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\nI find this a very interesting and solid paper, well written and seems well engineered. Most importantly, it doesn't involve retraining, fine-tuning or anything. The AMB update and comparison processes also seem quite lightweight. \nTheir baseline method only modifies two layer (middle and end, 13 and 27) They do ablation studies with more layers (Table 5) and find that for many settings such a sparse modification yields good results, but for summarization (Sum) and multi-document QA (MQA) applying it to every 2 or 4 layers works best. \n\n\n## Weakness\n1) some aspects of the CE, like the weighted sum or how the similarity is computed (I think separately for each cluster) are not properly explained. Sec. 3.2.1 shows how AMB works in isolation, but how it is used together with CE needs clarification, including indices for clusters, context, etc. This can go into the appendix. \n2) In spite of being lightweight, it seems the process  results in about 30% compute time overhead, right? And this is for only two layers out of 28. \n3) It seems Geva et al 2020 \"Transformer Feed-Forward Layers Are Key-Value Memories\" and related work discussed how FFN acts as a key-value memory. They should be cited. \n4) While the beginning of the paper mentions impressive improvements (6% MQA LongBench, 12.5% and 17.5% on En.Dialogue and En.Multi-Choice of $\\infty$-Bench), most other results are about 1-2% over baselines. So there is a signal, but not nearly as strong.", "was_revised": true, "final_justification": "I have reviewed the rebuttal and think the clarifications and runtime data further improve the paper. I like the way they implement a latent space memory fr the FFN and support acceptance of this work.", "rebuttal": "Dear Reviewer RyQo,\n\nMany thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.\n***\n>[W1,Q2]Some aspects of the CE are not properly explained, like the weighted sum. Is there any smart routing involved for the similarity to each expert? Fig 2 shows only a weighted sum. What are the weights?\n\n[A1]Thank you for your question. While Figure 2 depicts a weighted sum operation, the **weights are not uniform or trivial**—they are derived from a gating mechanism based on the similarity between input activations and each expert cluster.\n- In our implementation, the expert routing is based on **Avg-k gating**, a variant of soft mixture-of-experts routing. Specifically, as shown in the following code:\n```python\nexpert_scores = gate_output.reshape(batch, seq_len, n_expert, -1)\nexpert_scores = torch.mean(expert_scores, dim=-1)  # average across hidden dimension\nexpert_topk_indices = torch.topk(expert_scores, k=topk, dim=-1).indices\n```\nThis computes **per-token activation scores** for each expert via the gate_proj network.\n- We then **select the top-k experts** per token and mask out all others:\n```python\nexpert_topk_mask = torch.zeros_like(expert_scores)\nexpert_topk_mask.scatter_(dim=-1, index=expert_topk_indices, value=1)\n...\nintermediate_states = intermediate_states * expert_topk_mask\n```\n- Hence, the weighted sum shown in Figure 2 is computed over only the **top-k activated experts**, where weights reflect the averaged activation scores of each expert. This design introduces dynamic and content-aware routing, allowing different tokens to be processed by different semantic clusters of FFN units.\n- We hope this clarifies that the CE module does involve smart, token-specific routing, beyond a naive weighted average. We will provide detailed algorithms in the appendix later.\n\n>[W1,Q1]Some aspects of the CE, like how the similarity is computed are not properly explained. Sec. 3.2.1 shows how AMB works in isolation, but how it is used together with CE needs clarification, including indices for clusters, context, etc.\n\n[A2]Thank you for your insightful comment. Your understanding is totally correct: **memory similarity computations in our framework are performed at the level of each expert cluster.**\n- The Cortical Expert (CE) module partitions the FFN units into semantically coherent expert groups via clustering. Although CE alone does not bring significant performance gains, it provides semantic consistency within expert regions, which becomes particularly beneficial when paired with the Activation Memory Bank (AMB).\n- The AMB operates at the cluster level, performing similarity-based retrieval and aggregation within each expert group. This enables targeted memory usage and specialization across clusters, improving both semantic coherence and memory efficiency.\n- The importance of CE is further supported by the ablation results comparing baseline+PA, baseline+CE, and baseline+PA+CE. While neither CE nor PA alone delivers strong improvements, their combination achieves the best performance. This synergy confirms that CE is essential, as it provides the necessary structure for AMB to function optimally.\n- We appreciate your helpful request for clarification. We will revise the manuscript to explicitly describe the cluster-level interaction between CE and AMB, and to clarify how similarity indices are handled within this framework.\n\n>[W2,Q3,Limitations]In spite of being lightweight, it seems the process results in about 30% compute time overhead, right? And this is for only two layers out of 28. What is the compute overhead for dense PaceLLM?\n\n[A3]We fully agree with your concern — adding additional modules inevitably introduces some computational overhead.\n- The compute overhead for dense PaceLLM is as follows:\n| |Time|\n|-|-|\n|Ours Dense（32）|1h20min|\n|Ours Dense（16）|45min12s|\n|Ours Dense（8）|25min37s|\n|Ours Sparse（4）|16min|\n|**Ours Sparse（2）**|**11min07s**|\n|Ours Sparse（1）|6min02s|\n|Activation Beacon/Sparse（0）|9min55s|\n|Qwen2-7B/Sparse（0）|18min24s|\n\n- Despite this cost, we believe that focusing on FFN activations is a worthwhile direction, as it provides new insights into the role of feedforward representations in long-context modeling. We also believe our method can inspire future research in this underexplored area.\n- Improving efficiency will be one of our key priorities in future work, and we will explore optimizations to further reduce computational cost.\n\n>[W3]It seems Geva et al 2020 \"Transformer Feed-Forward Layers Are Key-Value Memories\" and related work discussed how FFN acts as a key-value memory. They should be cited.\n\n[A4]Thank you for the valuable suggestion. Indeed, we are aware of the work by Geva et al. (2020), \"Transformer Feed-Forward Layers Are Key-Value Memories\", and related studies discussing the interpretability of FFN layers in LLMs.\n- In fact, we originally included a dedicated section in the Related Work[1,2,3,4,5]  discussing this line of research. However, due to strict space constraints during the revision process, that part was unfortunately removed.\n- We will reintroduce and properly cite these important works in the updated version of our paper, and clarify how our approach builds on the insight that FFN layers can serve as associative memory structures. Thank you again for pointing this out.\n\n[1]Mor Geva et al. \"Transformer Feed-Forward Layers Are Key-Value Memories.\" EMNLP 2021.\n\n[2]Damai Dai et al. \"Knowledge Neurons in Pretrained Transformers.\" ACL 2022.\n\n[3]Song Wang et al. \"Knowledge Editing for Large Language Models: A Survey.\" ACM Computing Surveys 2024.\n\n[4]Xiaopeng Li et al. \"PMET: Precise Model Editing in a Transformer.\" AAAI 2024.\n\n[5]Zihan Qiu et al. \"Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers.\" Tiny Paper @ ICLR 2024.\n>[W4,Q4]While the beginning of the paper mentions impressive improvements, most other results are about 1-2% over baselines. So there is a signal, but not nearly as strong. you have impressive results for 3 tasks, mentioned at the beginning, but for the rest, you get only 1-2%. What do you think distinguishes those tasks from the rest and why does your model work better for those?\n\n[A5]We fully agree with your observation.\n- PaceLLM is primarily designed to enhance tasks that require preserving semantic consistency and coherence across long input contexts, such as multi-document question answering and summarization. This design motivation aligns with the more substantial improvements observed on tasks like MQA in LongBench and En.Dialogue / En.Multi-Choice in ∞-Bench.\n- That said, through threshold tuning, we also observed that PaceLLM provides modest yet consistent gains on tasks involving more abrupt semantic transitions, indicating a degree of generalization potential beyond its original scope.\n***\nWe hope the above response can help solve your questions. Thanks again for your thorough review and looking forward to your reply!"}, {"review_id": "u8VYwDmTkM", "reviewer": "Reviewer_o88K", "summary": "This paper introduces a brain-inspired method to avoid contextual knowledge decay in long-context settings and improve measurable context length in LLMs. In doing so, it aims to address fundamental limitation of feedforward networks, in which activations are transient and lead to information decay over time. The proposed methodology is complementary to other approaches and does not require training. Evaluation on multi-document and long-context tasks in inference and fine-tuning settings shows up to 17.5% improvement while preserving general capabilities of LLMs.", "questions": "Questions\n* How are k (line 137) and K (line 160) selected?\n* What is the impact of PaceLLM on training time at scale?\n* Do gains achieved with PaceLLM translate to larger models & newer models (e.g., Qwen2.5, Llama3.1)?\n* How do the results compare to black-box proprietary LLMs? Commentary on the observed performance of PaceLLM for moderately sized open-source models vs. proprietary LLMs could further contextualize the results.\n\nMinor Comments\n* Decimal places are inconsistent across benchmarks — MMLU (Table 4) could be condensed to just 2 decimal points.\n* It would be helpful to list dataset name abbreviations used in Tables somewhere, even though some are provided in the detailed tables in the Appendix.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n* The paper adopts a novel perspective on what contributes to long-context failures in LLMs, going beyond approaches presented in existing work to address a fundamental limitation of FFNs in a targeted manner.\n* The proposed methodology is complementary to other potential LLM interventions and therefore lightweight in application.\n* Experimental analyses are diverse and provide effective insights to explain the observed performance.\n\nWeaknesses\n* Use of additional modules to implement PaceLLM adds complexity in practice.\n* Experiments look only at moderately sized and relatively older LLMs; unclear scalability to larger models.", "was_revised": true, "final_justification": "As mentioned in discussion with the authors, I believe the key limitations of the paper are addressed and therefore maintain the positive rating. While the method still has some potential places for critique as mentioned by other reviewers, these are mostly minor and the paper's contributions are well-supported.", "rebuttal": "Dear Reviewer o88K,\n\nMany thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.\n***\n>[W1]Use of additional modules to implement PaceLLM adds complexity in practice.\n\n[A1]We fully agree with your concern — adding additional modules inevitably introduces some complexity and computational overhead.\n- However, our method is designed to be **plug-and-play**, and has demonstrated consistent effectiveness across various models and datasets, including unseen architectures and tasks. This shows that the added complexity does not compromise general applicability or ease of integration.\n- Despite the current cost, we believe this is a worthwhile trade-off, as our work is **the first in the field to explicitly explore FFN activations as a memory mechanism for long-context modeling**. This opens up new perspectives on the underexplored role of feedforward layers and suggests valuable directions for future research.\n- Improving **efficiency** will be one of our key priorities moving forward. We plan to explore optimizations to further reduce the computational burden.\n\n>[W2,Q3]Experiments look only at moderately sized and relatively older LLMs; unclear scalability to larger models. Do gains achieved with PaceLLM translate to larger models & newer models (e.g., Qwen2.5, Llama3.1)?\n\n[A2]Our method remains effective as model size increases. Experimental results are as follows:\n|Model|SQA|MQA|Sum.|FSL|Cod.|\n|---|----|----|---|---|-----|\n|Qwen2.5-14B-Instruct|17.18|12.15|23.35|71.46|32.30|\n|Qwen2.5-14B-Instruct+Ours|**18.48**|**12.97**|**23.49**|**72.32**|**33.41**|\n|Llama-3.1-8B-Instruct|24.22|15.04\t|28.21|69.49|58.44|\n|Llama-3.1-8B-Instruct+Ours|**24.31**|**15.80**|**28.47**|**69.85**|**59.59**|\n***\n>[Q1]How are k (line 137) and K (line 160) selected?\n\n[A3]Thank you for the question. In our implementation:\n- We set k = 5 for top-k similarity retrieval, which aligns with common practice in memory-based or retrieval-augmented models, where top-3 to top-5 candidates are typically used.\n- We set K = 64 as the number of expert clusters, following the standard configuration used in prior expert-based or MoE architectures.\n- These values were selected to ensure a balance between performance and computational efficiency, and we found them to be stable across different datasets.\n- Thanks again for your question! We will update in our revised manuscript.\n\n>[Q2]What is the impact of PaceLLM on training time at scale?\n\n[A4]PaceLLM is essentially a **training-free and plug-and-play** method, which does not require any modification during large-scale training.\n\nTo further assess runtime behavior, we conducted small-batch inference experiments, which show that batched inference introduces no significant computational overhead or performance degradation. We test on Qwen2-7B and Qasper in LongBench:\n\n|Batch_size\t|Time|\tPerformance|\n|-|-|-|\n|1\t|25m22s\t|43.88|\n|2\t|12m45s\t|43.88|\n|3\t|8m37s\t|43.88|\n|4\t|6m22s\t|43.88|\n\nThis suggests that our method remains efficient and scalable even when deployed in practical settings.\n>[Q4]How do the results compare to black-box proprietary LLMs? Commentary on the observed performance of PaceLLM for moderately sized open-source models vs. proprietary LLMs could further contextualize the results.\n\n[A5]Thank you for the insightful suggestion. PaceLLM is a **lightweight, plug-and-play** module with minimal memory overhead, and is **orthogonal** to most existing long-context modeling approaches. It has shown consistent effectiveness across a wide range of open-source baselines.\n- Through preliminary analysis and simple experiments, we observed that several enhanced open-source models—some even without PaceLLM—already outperform certain proprietary black-box LLMs on specific long-context tasks. With PaceLLM integrated, these models can further improve, suggesting its potential to narrow the performance gap with proprietary systems.\n- We appreciate this constructive feedback and will incorporate a comparative discussion on open-source vs. proprietary LLMs in the revised version of the paper to better contextualize our findings.\n***\n>[Minor Comments]Decimal places are inconsistent across benchmarks — MMLU (Table 4) could be condensed to just 2 decimal points.\nIt would be helpful to list dataset name abbreviations used in Tables somewhere, even though some are provided in the detailed tables in the Appendix.\n\n[A6]Thank you for the correction. We will revise this in the updated version of the paper.\n***\nLastly, thank you so much for helping us improve the paper! Please let us know if you have any further questions. We are actively available until the end of this rebuttal period. Looking forward to hearing back from you!"}]

Author rebuttal:

Meta-review: Reviewers are generally supportive of this paper. They found activation-level storage mechanism to be creative, simple, and well-motivated. At the same time, concerns remain about the general motivation of this work pertaining to two problems that the authors tackle: “transient neural activations cause information to fade over time” and “unstructured FFN weights fragment semantics across tokens”. The reviewers were not convinced that baseline models suffer from these issues. The biological inspiration remains speculative too. 

Despite this, reasons to accept this work outweight reasons to reject. In the camera-ready version please discuss similarities/differences of your method with:
1.  the similarity-aware strategies for memory updates used in He et al., CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory, 2024. 
2. Approach of Geva et al 2020 "Transformer Feed-Forward Layers Are Key-Value Memories”.
Decision: Accept (poster)