Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
Authors: Songlin Zhai
Yuan Meng
Yongrui Chen
Yiwei Wang
Guilin Qi
Venue: ICML 2025 poster (2025)
Keywords: Memory Augmentation for LLMs
Knowledge-based Model Editing
Large Language Models
Primary area: deep_learning->large_language_models
TL;DR: This paper introduces the peripheral memory for large language models.

Abstract: Large Language Models (LLMs) have revolutionized various natural language processing tasks with their remarkable capabilities. 
However, challenges persist in effectively integrating new knowledge into LLMs without compromising their performance, particularly in the Large Language Models (LLMs) have revolutionized various natural language processing tasks with their remarkable capabilities. 
However, a challenge persists in effectively processing new information, particularly in the area of long-term knowledge updates without compromising model performance. 
To address this challenge, this paper introduces a novel memory augmentation framework that conceptualizes memory as a peripheral component (akin to physical RAM), with the LLM serving as the information processor (analogous to a CPU). 
Drawing inspiration from RAM architecture, we design memory as a sequence of memory banks, each modeled using Kolmogorov-Arnold Network (KAN) to ensure smooth state transitions. 
Memory read and write operations are dynamically controlled by query signals derived from the LLMs' internal states, closely mimicking the interaction between a CPU and RAM. 
Furthermore, a dedicated memory bank is used to generate a mask value that indicates the relevance of the retrieved data, inspired by the sign bit in binary coding schemes. 
The retrieved memory feature is then integrated as a prefix to enhance the model prediction. 
Extensive experiments on knowledge-based model editing validate the effectiveness and efficiency of our peripheral memory.

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
Songlin Zhail
Yuan Meng'
Yongrui Chen
Yiwei Wang² 2 Guilin Qi!
Abstract
Table I. Comparison between different types of memory. W.M.:
Large Language Models (LLMs) have revolution- ized various natural language processing tasks with their remarkable capabilities. However, a challenge persists in effectively processing new information, particularly in the area of long-term knowledge updates without compromising model performance. To address this challenge, this paper introduces a novel memory augmentation frame- work that conceptualizes memory as a periph- eral component (akin to physical RAM), with the LLM serving as the information processor (analo- gous to a CPU). Drawing inspiration from RAM architecture, design memory as a sequence of memory banks, each modeled using Kolmogorov- Arnold Network (KAN) to ensure smooth state transitions. Memory read and write operations are dynamically controlled by query signals derived from the LLMS' internal states, closely mimick- ing the interaction between CPU and RAM. Fur- thermore, dedicated memory bank is used generate a mask value that indicates the relevance of the retrieved data, inspired by the sign bit in binary coding schemes. The retrieved memory feature is then integrated as a prefix to enhance the model prediction. Extensive experiments on knowledge-baad based model editing validate the effec- tiveness and efficiency of our peripheral memory.
"Working Memory", I.M.: "Implicit Memory", E.M.: "Explicit
Memory" P.M.: "Peripheral Memory (Ours)". _ ) "Well Sup-
ported". D "Partially Supported". D: "Poorly Supported".
Memory Types
W.M.
L.M.
E.M. P.M.
Scalability
Reusability
Configurabilitty
lenges in integrating new information (Wang et al., 2024b; Modarressi et al., 2025), particular in incorporating knowl- edge updates without harming their original performance (Zhang et al., 2024; Fang et al., 2025). These challenges are further exacerbated in dynamic real-world applications (Kaddour al., 2023; Wang al., 2024b). Memory aug- mentation has emerged as promising solution to address these challenges (Zheng et al., 2023; Hartvigsen et al., 2023; Wang et al., 2024a;b; Modarressi et al., 2025), with existing efforts focusing on working memory (contextual key-values pairs from specific layers), implicit memory (extra model parameters), and explicil memory (sparsely activated knowl- edge circuits) (Yang al., 2024). However, summarized in Table 1, these methods are hindered by limitations in scalability (i.e., maintaining query effectiveness and effi- ciency as memory size increases). reusability (i.e., sharing memory across LLMs with varying architectures to avoid redundant storage of identical knowledge), and configura- bility (i.e., dynamically adjusting memory independently of the LLM, such as memory size and structure). These issues primarily stem from the tight coupling between memory and LLM architectures, which restricts dynamic memory allocation and cross-model adaptability.
1. Introduction
Large Language Models (LLMs) have transformed the field of machine learning, achieving state-of-the thetthtrrtttttt performance across wide range of tasks (Singh, 2023; Naveed et al., 2024; Chen, 2024; Luo al., 2024; Azaria et al., 2024). De- spite these remarkable advancements, L.MSs still face chal-
Inspired by modern computer architectures that separate computation and storage (Blum, 1986; Rojas & Hashagen, 2000), we propose a novel memory framework called pe- ripheral memory. Unlike traditional memory systems, decouple memory from LLMs, treating it as an independent module-hence the term "peripheral". In this framework, the LLM functions as the information processor, similar to a CPU, while the peripheral memory operates like RAM, han- dling requests from LLMs retrieve or store information
'School of Computer Science and Engineering, Southeast Uni- versity, Nanjing, China UUniversity of California, Merced, USA. Correspondence to: Guilin Qi <gqi@ seu.edu.en>>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
as needed. This decoupling offers two key advantages: the flexibility design memory independently of the LLM's architecture, and the ability to treat memory contents as dynamic variables, free from constraints imposed by the LLM (e.g., hidden feature dimension, depth of the model).
We conduct extensive experiments on Knowledge- based Model Editing, demonstrating the effectiveness of the proposed memory in improving task perfor- mance and efficiency, along with enhanced knowledge storage management.
For the memory architecture, we draw inspiration from phys- ical RAM (Random Access Memory) (Jacob et al., 2007), structuring the peripheral memory as a set of memory banks, each functioning one-bit data unit. The number of mem- ory banks thus determines the overall memory bandwidth, enabling flexible scaling of memory capacity. For each memory bank, we conceptualize its query process as a map- ping from query signals to memory data, and employ a Kolmogorov-Arnot Network (KAN) (Liu et al., 2024) to model this process, due to recognizing KAN's ability to fit nonlinear interactions with fewer parameters. This could ensure efficient and precise memory retrieval while minimiz- ing computational overhead. Moreover, instead of working independently, each memory bank refines its query itera- tively, passing its output to the next one. This design, termed sequential, allows memory banks to process information sequentially, promoting smoother integration of information and enhancing the handling of interdependent queries.
2. Preliminaries
2.1. Notations and Task Definition
A large language model requires a textual sentence 8 as input, which is tokenized into sequence of tokens + (lo, ................ responding embedding vector which is processed by each hidden layer of the model. We use the bold notation t,, to represent the hidden representation of t, at layer I, where S S L indexes the layers of the LLM. Additionally, we denote the memory as M {M1 M2, ..., MM,, .... Mk); with Mg being an individual memory bank (a specific stor- age unit). To query the memory. query signal q is used. After querying, the resulting memory data is denoted as m = (m1, M2, ..,, My: ..., mx), where my is the query re- sult from k-th memory bank (i.e. M2). For augmenting LLM with memory, the task can be formulated as:
Each token is converted into a cor-
For memory querying, if the required data is not stored in memory, the query results are likely to be useless or even detrimental to the LLM. In this case, directly integrating the query results into the LLM could disrupt its forward ac- cumulation of information. To ensure query reliability, we introduce a confidence memory bank that assesses the rele- vance of retrieved data by generating trust value derived from the sequential outputs of previous banks. This mecha- nism filters out irrelevant or noisy information, ensuring that only relevant memory content is integrated into the I.M,, thereby preserving its original knowledge and preventing interference. Memory writing is implemented through a fine-tuning process, eliminating the need for complex man- ual operations and improving usability. Furthermore, when memory reaches capacity, older data is archived, and new data is stored in fresh memory. enabling efficient long-term memory management, Extensive experiments on knowledge editing validate the effectiveness of the proposed peripheral memory. In summary, the contributions of this paper are:
-fur y y: K(x) & M
K(x)
fe(x|M) =
(I)
where 2 denotes the LLM parameters, and f is a complex function learned by the LLM. K((x) represents the relevant knowledge related to input x. If the relevant information is stored in the memory, i.e., K(xx) E M, the LLM is expected to output the desired prediction y". Otherwise, the model should maintain its original prediction y.
2.2. Architecture of Physical RAM
Physical RAM is critical component in modern computer systems, providing high-speed temporary storage for data. The architecture of RAM typically consists of one two memory ranks. Each memory rank is composed of several memory chips, and each chip contains multiple memory banks. Specifically, a memory bank refers to a memory array for storing data, and is indexed by specific row and column identifiers. Each memory cell within a bank, identified by a unique (rowLid, columnidd) stores specific bits of data and is predefined by the manufacturer.
We propose a novel peripheral memory for the L.LM that decouples the memory module from the model itself. This design alleviates the limitations of existing memory paradigms, enhancing the scalability, reusabil- ity, and configurability of the memory system.
For the memory query process, we can conceptualize it as mapping: X + y, where X represents the query signal from the CPU, and y is the corresponding sequence of O and returned from memory. Specifically, each bit of the query can thus be modeled as:
We introduce a sequential querying mechanism, where the output of each memory bank serves as the input to the next. Additionally, a confidence memory bank is allocated to generate trust values, minimizing interfer- ence with the L.L original knowledge.
dx (X = [(X[M)
(2)
where &e indicates the query function of k-th bit in RAM
2

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
iokan desthrre
-prooess the information from InpidEs
procoss information from memoryy
LCM
lokens
Embedding Layer
imput sentence
-00
o -OO0
Layeer N2
Layee Hi
Lay Jayer NL-1
Layer HL
Output Laye
Layer
OOO GOO
lngits
quary feature
Interact with Ponpiterna Memony
memory fecture
OOO...
Pertpherol Memory
000 ...00
Converter
Converter
9
query
return
conffidence
query signal
memoryy signal
Figure 1. Framework of peripheral memory. Reading& Writing operations are achieved by the query signal from LLM hidden states.
M. Il is an indicator function that outputs either or 1. In this context, [([]M) represents the presence or absence of the desired bit of information from the physical RAM M based on the query signal X.
where my is the memory data at k-th element in the memory feature vector m, retrieved by gk- Specifically, qk is also the k-th element in the query signal q.
Unlike the discrete bits in RAM, which can be understood as indicator functions, the query function in peripheral mem- ory is designed to be continuous and smooth to avoid in- formation loss. To model this smooth function for each memory bank (also denoted as M₁ for simplicity), we em- ploy Kolmogorov-Amng Amoold Networks, which are capable of modeling complex, non- non-linear relationships in continuous spaces with fewer parameters (Kolmogorov, 1957; Liu et al., 2024). Formally, the memory bank Mj. is defined as:
3. Methodology
Different from previous memory architectures that directly store information within the LLM, our peripheral memory decouples the memory module from the LLM, treating it as an independent storage component. In this design, the content stored in memory is treated as a dynamic variable, allowing the L.LM to dynamically retrieve or write the nec- essary information as needed. Figure illustrates the overall framework. The memory is connected to the LLM through specialized converter, which acts like signal cables to facilitate feature conversion between the two components. Memory operations are driven by the hidden state of the last token from the LLM (serves as the query signal), and the query result is then integrated into the LLM as prefix, enhancing the generation process.
Mx( =E De Di g/ (gx))
(4)
where D3 is the memory depth of Mr. which can be config- ured freely. o¹t and Ny are simple one-variable continuous functions parameterized by B-spline function (A (Aziznejad & Unser, 2019; Bohra et al., 2020; Liu et al., 2024).
3.1.2. SEQUENTIAL QUERYING MECHANISM
3.1. Flement-Wist Wise Peripheral Memory
As shown in Eq. 3, each memory is independently retrieved by its corresponding query signal gk- To enable richer in- teractions between memory banks and allow the memory system to capture complex patterns in query data, we intro- duce a sequential querying mechanism. That is, the output of one memory bank serves as the input to the next. As such, the querying process Eq. can be extended as:
3. STRUCTURE OF MEMORY
Similar to physical RAM, our peripheral memory is com- posed of set of distinct memory chips. However, unlike physical RAM, we simplify the memory architecture by as- signing each memory chip to contain single memory bank, with each bank representing a one-bit data unit'. Figure 2 depicts the detailed architecture of the proposed peripheral memory. In line with Eq. 2, the query of each memory bank (e.g., M2) conceptualized as mapping from the query signal to the query result, formulated as:
m = Me(m+-1,94) 2
(5)
where m denotes the query result of the last memory bank M From Eq. 5, we can draw that the peripheral memory contains a total of K memory banks to store data, with the first memory bank accepting only the query signal (formulated by Eq. 3), and subsequent memory banks ac- cepting both the query signal and the query result from the
m My(4s)
(3)
'The terms memory chip and memory bank equivalent in peripheral memory, as each chip contains only one memory hank.
3

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
47
9K-1
9K
where t'y is the hidden state of the last token at layer 1. Wo converts the token feature into the query signal, acting as the outlet cable. Similarly, Wi maps the memory data m into the memory feature adapted to LLM hidden feature space, which can be understood as the leading in cable. The confidence value a is normalized by the sigmoid function a, and then is used to mask the memory feature element by element. t is the masked memory feature, which is merged as prefix vector into the LLM. Eq. 8 allows the memory and the LLM to be used effectively as a unified system. As such, the memory writing could be simply achieved by performing finetuning process with setting requiring gradients of the memory. After the peripheral memory is "full", we can unplug and archive it and store data in a fresh one'. The memory bandwidth K plays a critical role in determining the read/write performance of the peripheral memory. To ensure K operates within optimal range, we leverage the Information Abundance in Guo et al. (2024) to determine that its lower bound is approximately 64.
Query Signol
q
Pergetoral Momory
Argan park Sen Sine
MgA1
Memory
Signal
m
Converter
Memory Feature
Confidence Score
Gating =
Gated Memory Feature
Figure 2. Architecture of the peripheral memory.
previous bank. For the memory bank where 2 2 k S K, Eq. 4 can be redefined as:
4. Experiments
M& (m = Σ De i=1 oe (hi (1t) + (m)-)))
(6)
In this section, we evaluate the proposed peripheral mem- ory within representative scenario involving knowledge updates, specifically Knowledge-basee Model Editing. We primarily focus on analyzing its strengths in terms of scal- ability, reusability, and configurability, highlighting its ad- vantages compared to existing memory architectures.
where (=2(m==1) represents the processing of information from the previous memory bank (Ma-1) also parameterized by the B-spline function.
3.1.3. CONFIDENCE MEMORY BANK
4.1. Knowledge-based Model Editing (KME)
To ensure the quality of the retrieved information and pre- vent disturbances to the LLM's existing knowledge, we
1.1. EXPERIMENTAL SETUP
introduce a confidence memory bank, represented Mx+1
The KME task evaluates the ability of the proposed periph- eral memory to integrate specific knowledge updates into an LLM while ensuring that its original information remains unaffected. Following prior studies (Mitchell et al., 2022a; Meng et al., 2022; 2023; Li et al., 2024), we also utilize the EDIT sets of ZsRE (Levy et al., 2017) and COUN- FACT (Meng et al., 2022) to comprehensively evaluate all competing methods. To assess editing performance, we adopt three fundamental metrics, i.e., Efficacy, Generality, and Locality. Notably, we consider the consecutive editing, where series of sequential updates is performed without parameter roll-back. This setting effectively tests memory's s capability to manage long-term knowledge updates.
in Figure 2. This confidence memory bank receives the
memory states output by the previous memory bank (Mx) and calculates a trust value to assess the relevance of the data retrieved from memory. This value reflects the con- fidence level in the retrieved memory and acts as a mask to filter and refine the retrieved information before it influ- ences the LLM's processing pipeline. The confidence value predicted by the Mx+1 memory bank is formulated as:
a = (K+1(mK)
(7)
Analogously, the MK+1 memory bank can be modeled as EDx+1 o."h k ( (mx)). The masking operation is par-
ticularly useful in Knowledge Updates: where new, rele- vant knowledge should be prioritized over outdated data.
All experiments are conducted on an NVIDIA A100-SXM4- 40GB machine. The baseline methods are implemented using the widely adopted EasyEdit toolkit², ensuring repro- ducibility, reliability, and adherence to established research practices. Hyperparameter settings for all baseline models are based on recommendations provided by EasyEdit.
3.2. Memory Reading & Writing
Before reading from a peripheral memory, we first plug it into an LLM. Then. query feature from the LLM is used to retrieve the memory, defined as:
Section 4.4 discusses memory storage capacity. httpps:// /tte:://gttttt github. com/ z juunnp//asyEdee
t (ax) t. = =(())- (M((t>) (M(()))TTo)) T)) Wi)
(8)
4

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
Table Editing performance compared methods under 3K consecutive editing. our method, query features are derived from the hidden states (the last token representation) of the model's 24-th layer.
ZsRE
COUNTER
Editor
Effiiaacy Generality Locality
Score
Efficacy
Cenerality Locality
Score
GPT-J (6B) (Wang& & Komatsuzaki, 2021)
0.2165
0.2110
0.2138
0.0030
0.0023
/
0.0027
FT-L (Zhu et al., 2020)
0.1104
0.0841
0.0159
0.0701
0.2133
0.0797
0.0127
0.1019
LoRA (Hu et al., 2022)
0.0111
0.0115
0.0005
0.0077
0.0097
0.0067
0.0013
0.0059
ROME (Meng et al., 2022)
0.3187
0.2810
0.1829
0.2609
0.0013
0.0020
0.0003
0.0012
R-ROME (Gupta al., 2024a)
0.5478
0.5176
0.1333
0.3996
0.6927
0.3740
0.4187
0.4951
MEMIT (Meng et al., 2023)
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
AlphaEdit (Fang et al., 2025)
0.0000
0.0000
0.0005
0.0002
0.0000
0.0000
0.0000
0.0000
PMET (Li et al., 2024)
0.0002
0.0002
0.0003
0.0002
0.0000
0.0000
0.0000
0.0000
KN (Dai al., 2022)
0.0000
0.0000
0.0002
0.0001
0.0001
0.0000
0.0000
0.0000
EMMET (Gupta et al., 2024b)
0.5521
0.5167
0.3747
0.4812
0.7020
0.4117
0.3303
0.4813
GRACE (Hartvigsen et al., 2023)
0.3368
0.0126
1.0000
0.4498
0.0002
0.0000
0.9939
0.3314
IKE (Zheng et al., 2023)
0.9973
0.9898
0.4764
0.8212
0.9900
0.4223
0.6393
0.6839
WISE (Wang et al., 2024a)
0.4598
0.4185
0.9895
0.6226
0.3420
0.0907
0.0790
0.1706
Ours (without 1K archive)
0.9934
0.6619
1.0000
0.8851
0.9240
0.1647
1.0000
0.6962
Ours (with 1K archive)
0.9985
0.6941
1.0000
0.8975
0.9907
0.2340
1.0000
0.7416
Llama 3 (8B) (Llama Team, 2024)
0.2627
0.2598
0.2613
0.0087
0.0075
0.0081
FT-L (Zhu et al., 2020)
0.0769
0.0666
0.0069
0.0501
0.0575
0.0047
0.0013
0.0212
LoRA (Hu et al., 2022)
0.1145
0.1116
0.0535
0.0932
0.0077
0.0117
0.0017
0.0070
ROME (Meng et al., 2022)
0.0339
0.0280
0.0015
0.0211
0.2507
0.1323
0.0097
0.1309
R-ROME (Gupta et al., 2024a)
0.0271
0.0243
0.0035
0.0183
0.4892
0.3662
0.0147
0.2900
MEMIT (Meng et al., 2023)
0.0000
0.0000
0.0396
0.0132
0.0000
0.0000
0.0722
0.0241
AlphaEdit (Fang et al., 2025)
0.0001
0.0000
0.0003
0.0001
0.0033
0.0017
0.0007
0.0019
PMET (Li al., 2024)
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
EMMET (Gupta et al., 2024b)
0.0517
0.0486
0.0043
0.0349
0.5450
0.3882
0.0128
0.3153
GRACE (Hartvigsen et al., 2023)
0.0624
0.0095
1.0000
0.3573
0.0003
0.0000
0.9938
0.3314
IKE (Zheng et al., 2023)
0.5233
0.5231
0.5289
0.5251
0.0055
0.0043
0.6509
0.2202
WISE (Wang et al., 2024a)
0.3348
0.3283
0.9997
0.5543
0.1473
0.0763
0.9907
0,4048
Ours (without K archive)
0.9597
0.5619
1.0000
0.8405
0.9038
0.2168
1.0000
0.7069
Ours (with 1K archive)
0.9805
0.6123
1.0000
0.8643
0.9915
0.3108
1.0000
0.7674
4.1.2. OVERALL COMPARISON
desired, the peripheral memory can simply be "umplugged" without causing any modifications to the original model. (2) While EMMET and IKE achieves slightly better perfor- mance in the Generality on COUNTERFACT. its Locality score is considerably lower than that of ours. This suggests that EMMET introduces substantial disruptions to the origi- nal model. (3) Localization-base methods, such as ROME, AlphaEdit and PMET, suffer severe performance degrada- tion after consecutive updates, with their scores eventually dropping zero. This indicates that these methods not only fail to retain newly edited knowledge but also introduce significant interference, severely compromising the origi- nal model's integrity. As a result, they are unsuitable for large-scale or long-term knowledge editing.
Table 2 summarizes the performance of all compared meth- ods under 3K consecutive updates. The results highlight several key observations (1) Our method significantly out- performs all baselines across most metrics, achieving state- of-the-art results. Notably, even when compared to the latest memory-based KME methods, such as GRACE, IKE and WISE, our method demonstrates a significant performance advantage. This underscores the scalability and robustness of our framework, particularly in handling large-scale mem- ory updates. Additionally, our method consistently achieves perfect performance Locality, benefiting directly from our novel memory architecture, the "peripheral" memory. Specifically, when original predictions from the model are
5

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
1.00
1.00
IKE WISE GRACE Ours
Table 3. Reusability of peripheral memory. Memory writing is per-
Emcacy
0.75
WISE GRACE
formed on Llama3 (8B), after which it is shared with other LLMs reading only. The values labeled with represent performance with shared memory. while those without indicate performance with fresh memory written directly on LLM itself.
0.50
Ours
0.25
0.25
0.00
1K
2K
3K
4K
5K
10K
1K
2K
3K
4K
SK
10K
Models
Efficacy Generality Locality
Score
0.6
IKE
0.3
Generalty
0.4
GRACE
WISE
02
WISE
GRACE
ZsRE
Llama3 (8B)
0.9907
0.6090
1.0000
0.8666
0.2
0.9918
0.6989
1.0000
0.8969
0.0
0.0
Gemma2-it (2B)
0.97131
0.6635
1.0000
0.8783
1.0
100
1.0000
0.6547
1.0000
0.8849
Loca ity
WISE
0.
SE
Phi3 (3.8B)
0.0000!
0,0000
1.0000
0.3333
0.8
Durs
0.8
Ours
COUNTERFACT
07
0.6
Llama3 (8B)
0.9990
0.3150
1.0000
0.7713
1K
4K
10K
2K
3K
10K
0.9890
0.1948
1.0000
0.7279
Gemma2-it (2B)
0.9790
0.2045
1.0000
0.7278
Figure 3. Performance of long-term knowledge updates on 7sRE (left) and COUNTERFACT (right) based Llama3 (8B). Our method is evaluated under 1K archiving.
0.9997
0.2875
1.0000
0.7624
Phi3 (3.8B)
0.9997t
0.3075
1.0000
0.7691
4.1.3. SCALABILITY: LONG-TERM RETENTION
sistently underperform compared our method. This lower performance is likely due to noise accumulation in working memory, which affects retrieval accuracy over time.
To further investigate the scalability of our method, we conduct experiments by increasing the number of reads and writes in memory from 1K to 10K, and analyzing the effects of long-term editing. Figure 3 presents the re- sults of our methods compared to several strong memory- based methods across varying numbers of edits. The results demonstrate that our peripheral memory effectively adapts to larger number of memory updates without experienc- ing performance degradation. In contrast, WISE (implicit memory) exhibits sharp decline in efficacy and generality as the number of edits increases. Mcanwhile, GRACE and IKE (working memory) exhibit more stable trend but con-
4.1.4. SCALABILITY: STORAGE EFFICIENCY
In scenarios involving a large number of edits, both per- formance and storage efficiency are critical considerations. Figure illustrates the storage efficiency of our method com- pared others under extensive editing conditions. From the 1K update results, we observe that while model's write latency falls between GRACE (working memory) and WISE (implicit memory), it benefits from high configurabil- ity (see Section 4.4), enabling efficient parallel writes. The results ranging from 1 K to 10K further illustrate that, ow- ing to its parallel storage capability, our method maintains consistently stable and low efficiency values across scales. Among the baselines, WISE exhibits the longest write time since it internalizes knowledge directly into model param- eters, whereas GRACE achieves the shortest latency due to its lightweight working memory mechanism. Addition- ally, ROME, despite being an efficient localization based approach, takes longer than memory- methods.
Time/Writin (Hour)
75
ROME
GRACE
50
WISE
Ours
25
1K
2K
3K
4K
5K
10K
Time/Writng (Hour)
60
ROME
GRACE
40
WISE
4.1.5. REUSABILITY: MEMORY SHARING
Ours
20
As discussed in Section 3.1, the proposed peripheral mem- ory is entirely independent of the underlying LLM and oper- ates with a high degree of autonomy. This independence en- dows our memory with a significant advantage, i.e., reusabil- ity. Specifically, knowledge stored one LLM's peripheral memory can seamlessly shared with other LLMss, elim-
1K
2K
3K
4K
5K
10K
Figure 4. Writing efficiency of extensive knowledge updates on ZsRE (top) and COUNTERFACT (bottom) based on Llama3 (8B). Our method is evaluated using parallel storage of archiving.
6

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
Accuracy
Generality
1.0
Parra
0.4
Param
1.00 0.99
Performance
Performace
Generality
Accuracy
0.2
0.70
0.6
0.4
0.0
0.65
32
64
128 Accuracy
256
512
1024
Generality
5
10
15
20
25
30
1.00
Performance
1.0
Param
# Param
0.99
Accuracy Generality
0.2
Performace
0.1
0.5
0.0
0.65
5
7
11
13
15
0
5
10
15
20
25
30
Figure 5. Effects of memory bandwidth (top) and depth (bottom) based Gemma2-it (2B) and ZsRE with 1K consecutive editing.
Figure 7. Performance queries from different layers within Llama3 (8B) on 7sRE (top) and COUNTERFACT (hottom).
Accuracy
Generality
Time (M)
Performance
1.0
50
Time (M)
0.8
25
linguistic information, thereby lacking sufficient semantic integration in the final token representations. As move the query to higher layers, performance on both metrics gradually improves and ultimately stabilizes. However, for very late layers (i.e., layers 30-32), the model's performance on ZsRE experiences a marked decline. Therefore, to main- tain consistently stable and high performance, we select the 24-th layer as our query layer throughout this paper.
0.5K
1K
2K
3K
4K
5K
10K
Figure 6. Storage capacity with bandwidth 512 and depth 11 based on Gemma2-it (2B) and ZsRE under consccutive editing.
inating the need for redundant knowledge storage. This follows the principle: store once, use everywhere. Table 3 presents the results of 1 K memory sharing, where knowl- edge is first stored using Llama3 (8B) and then transferred to two different LLMs*: Gemma2-it (2B) and Phi3 (3.8B). The results demonstrate that even when memory shared across LLMs with different architectures, knowledge reten- tion remains consistently high, achieving nearly identical effectiveness as knowledge stored from scratch. In contrast, existing memory architectures do not support cross-model reuse due to constraints such as incompatible feature di- mensions, parameter spaces, and internal representations. Notably, the performance of Phi3 under shared memory of ZsRE is O. The possible reason is that ZsRE, being zero- shot relation extraction dataset, relies more the LLM's internal knowledge rather than externally stored memory.
4.3. Memory Configurability
4.3.1. MEMORY BANDWIDTH
evaluate the effects of memory bandwidth, we vary the number of memory banks from 32 1024 (i.e., the num- ber of parallel storage units) while keeping the memory depth fixed 11. As shown in the top section of Figure 5, increasing the memory bandwidth initially improves task accuracy but results in diminishing returns at larger scales. With 512 memory banks (approximately 0.2M parameters), the memory achieves near-perfect aceuracy (0.99), demon- strating sufficient capacity for basic operations. However, when scaling to 1024 memory banks (about 0.4M parame- ters), accuracy drops to 0.98, likely due to increased noise from sparsely activated memory units. Generality, which reflects the memory's 's ability to generalize to semantically equivalent queries, shows a similar trend to accuracy, fur- ther supporting this observation. Based on these results, we adopt 512 memory banks as the default memory band- width in this paper. For most applications, dynamically allocating bandwidth based task-specific requirements, optimizing both performance and efficiency.
4.2. Query Features from Different Hidden Layers
In this paper, we directly utilize hidden-layer representations as query features. However, selecting features from differ- ent layers can significantly impact memory performance. Figure illustrates memory performance of Accuracy and Generality, under 3K consecutive edits (with a 1K archive). As depicted, employing features from earlier layers (i.e., lay- 1-7) as queries results in poor accuracy and generality, mainly because these layers predominantly handle low-level
recommend
4.3.2. MEMORY DEPTH
Memory depth controls the size of each memory bank, with larger depth allowing for richer information refinement but also increasing computational costs. As illustrated in the
We utilize the hidden states from the 24-th layer of Gemma2 the 28-th layer Phi3 as the query features.
7

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
Table 4. ESD of different memories on Llama3 (8B).
Table 5. ESD of different memories across L.MSs (3K data).
Memory
GRACE (W.M.)
WISE (I.M.)
Ours (P.M.)
Memory
GRACE (W.M.) WISE (I.M.) Ours (P.M.)
#Data
1K
10K
1K
10K
1K
10K
L.Ms
GPTJ Llama3 GPTJ Ilama3 GPTJ Llama3
Accuracyt0.077 0.0794
0,0651 0,5512 0.2720
0.9908
0.9878
Accuracyt 0.3368 0.0624 0.4598 0.3348 0.9985 0.9805
#Param4
18.4M
184.3M 244.92244.9)
0.2M
0.2M
#Param,
61.4M 55.3M 268.4M 234.94 0.2M
0.2M
ESD 0.0043 0.0035 0.0023 0.0116
4.95
49.39
ESD1 0.0165
0.0034 0.0051 0.0043 14.98
14.71
bottom subfigure of Figure 5, both the storage Accuracy and Generality improve as the memory depth increases. Model performance stabilizes when the depth reaches 11, indicat- ing that larger depth enhances the memory's ' storage capac- ity. However, this improvement comes at the cost of higher parameter counts in the memory, leading to increased com- putational overhead during memory writing and retrieval operations. To strike a balance between performance and efficiency, we choose a memory depth of 11 as an optimal configuration for our framework.
ferent memory types. For working memory", which caches contextual key-value pairs, the amount of stored information increases dramatically as the data volume grows. This leads to a significant rise in the number of parameters required for memory (e.g. increasing from 18.4M at 1 K to 184.3M at 10K). However, as the cached information increases, the probability of retrieving irrelevant data also rises during in- ference, reducing storage performance (e.g., reducing from 0.0794 at 1K to 0.0651 at 10K). These two factors cause a substantial decrease in effective storage density (e.g. from 0.0043 at K to 0.0035 at 10K). In implicit memory the number of parameters remains relatively stable as the data increases on the same LLM, but its performance signifi- cantly drops (e.g., from 0.5512 at 1 K to 0.2720 at 10K) due to knowledge forgetting, causing poor ESD. Additionally, the number of parameters in working memory and implicit memory inevitably changes the model changes (as shown in Table 5), which will bring inconvenience for memory management. At the same time, both of them rely heavily on the LLMs' structure, limiting their configurability.
4.4. Discussion
1. MEMORY STORAGE CAPACITY
As discussed in Section 4.3, we configured the memory with a bandwidth of 512 and depth of 11. This setup raises the question: What is the storage capacity of the peripheral memory, similar to the storage size of physical RAM. To investigate this, we continuously stored data in memory while evaluating its performance and storage efficiency. As shown in Figure 6, as the amount of stored data increases, the memory storage accuracy remains high (around 98% at 10K). However, the generalization ability for semantically equivalent queries significantly drops, from 0.72 at 0.5K to 0.65 at 10K, especially as the data volume reaches 1000. This highlights the trade-off between memory utilization and semantic discriminability, which arises from the direct querying mechanism using the last token hidden state.
In contrast, the proposed peripheral memory shows stable performance across different LLMs and varying data sizes. Moreover, by leveraging KANs, peripheral memory signif- icantly reduces the number of parameters required (only 0.2M).. This enables it to achieve considerably higher effec- tive storage density, e.g., 1000 times greater than working memory and 500+ times greater than implicit memory for 1K data under Llama3 (8B). Surprisingly, this advantage grows even more pronounced as the data size increases or as the LLM architecture changes (See Table 4 and Table 5).
4.4.2. EFFECTIVE STORAGE DENSITY (ESD)
Memory storage density is crucial factor for evaluating the memory efficiency. Similar to knowledge capacity (Allen- Zhu & Li, 2024), we introduce the concept of Effective Storage Density (ESD) to quantify how many pieces of knowledge are successfully stored per 1K parameters:
5. Related Work
Existing memory mechanisms for LLMS can be broadly classified into three categories (Yang et al., 2024): working memory, implicit memory, and explicit memory. Work- ing Memory retains transient context by caching contextual key-value pairs from hidden layers. Notable methods in- clude Transformer-Xi XL (Dai et al., 2019), which introduces segment-level recurrence for cross-context retention, and Memformer (Wu et al., 2022a), which dynamically updates
#D
ESD(M = #0(M)
* ACC)racy(DIM)
(9)
where #D refers to the number of data in D. #O(M)
calculates the parameters number of M in KB. Accuracy(-) estimates the storage accuracy of D based on M.
*#Parameters are calculated based on Hartvigsen et al. (2023). *#Parameters are calculated based on Wang et al. (2024a).
Table compares the effective storage density across dif-
8

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
memory slots during decoding. Subsequent advancements enhance working memory through methods like approxi- mate KNN lookup (Wu et al., 2022b), subject word embed- ding alteration (Li et al., 2025), retrieval-basee methods (Hartvigsen et al., 2023), and retrieval-augmenned counter- factual models (Mitchell et al., 2022b). However, these approaches focus on short-term context and lack persis- tent, scalable storage. Implicit memory encodes knowl- edge into model parameters for long-term retention. Ap- proaches in this category include fine-tuning (Hu et al., 2022; Modarressi et al., 2025; Yu et al., 2024), direct pa- rameter edits (Mitchell et al., 2022a; Meng et al., 2023; Raunak & Menezes, 2022)), and paraieter-efficientt patch- ing (Dai et al., 2023; Wang et al., 2024b; Wang & Li, 2024; Wang et al., 2024a). While effective for targeted updates, these methods often suffer from catastrophic forgetting and architectural rigidity, they are tightly coupled with the LLM parameters (Kaddour et al., 2023). Explicit mem- ory encompasses retrievable model parameters, externalized knowledge, or coansely-activated neural circuits (Yang et al., 2024). However, these frameworks often rely on static memory allocation (Yao et al., 2023), struggling with issues related to scalability, reusability and configurability.
frequently encountered hidden states. Consequently, this specialization comes at the cost of reduced generalization to semantically similar but representationally divergent queries (see Figure 6 and Section 4.4.1).
To overcome limitations of the direct memory querying, we are now actively developing a memory management module inspired by the Memory Management Unit (MMU) in oper- ating systems. This module serves as an abstraction module between the LLM and the peripheral memory, effectively decoupling semantic alignment from storage operations. By doing so, the peripheral memory can specialize in storage and retrieval, while the MMU is responsible for query nor- malization and semantic mapping.
Acknowledgments
This work is partially supported by National Nature Science Foundation of China under No. 62476058. We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper.
Impact Statement
6. Conclusion
This paper presents work that aims to advance the field of aemore-angmented LLMs, with the potential to signif- icantly advance the field of machine learning. There are many potential consequences of our work:
In this paper, we introduced peripheral memory, a novel memory augmentation framework inspired by the decoupled architecture of physical RAM and CPU. By conceptualiz- ing memory as an independent module separate from the LLM, method addressed critical limitations in scala- bility, reusability, and configurability inherent to existing memories. At the core, the Kolmogorov- were used to model memory banks, enabling smooth state transitions and sequential interactions between storage units. Additionally, a dedicated confidence mechanism further en- sured reliable retrieval by filtering irrelevant or noisy data, preserving the LLM's original knowledge integrity. Exten- sive experiments knowledge editing demonstrated its superior performance while maintaining efficiency.
Peripheral memory has the potential to democratize access to LLM capabilities by enabling effi- cient, modular memory systems. However, its deployment must be accompanied by rigorous ethical guidelines and regulatory oversight to ensure responsible use.
Societal Implications:
Networks
The modular design of peripheral memory enhances interpretability, as memory operations are decou- pled from the LLM's core processing. This transparency crucial for auditing and debugging memory systems.
Ethical Impacts:
The reusability and read& efficiency of peripheral memory significantly reduce redun- dant storage of identical knowledge, enhancing the overall energy efficiency of the memory system. By minimizing repetitive operations, our framework lowers the computa- tional resources required for memory management, thereby reducing the carbon footprint of memory-augmenteed LLMss. This aligns with broader sustainability goals in AI research, promoting environmentally responsible development of ad- vanced language models.
Environmental Impact:
Limitations
In the proposed framework, memory is queried directly us- ing the hidden state of the last input token. While this design facilitates efficient retrieval, it faces a well-known challenge in representation learning: hypersensitivity to minor in- put variations (Jiang et al., 2020). Semantically equivalent queries (despite conveying the same meaning) may be pro- jected to distinct regions of the memory space due to subtle differences in token-level hidden representations. Moreover, as the volume of stored data increases, the memory module tends to overfit to the original query distribution, becom- ing increasingly specialized in retrieving exact matches for
Storing and retrieving dynamic knowledge Potential Risks:
raises privacy risks, particularly if sensitive or personal data is inadvertently memorized. Techniques like differential privacy memory anonymization should be investigated.
9

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
References
Guo, X., Pan, J., Wang. X., Chen, B., Jiang, J., and Long,
M. On the embedding collapse when sealing up recom- mendation models. In ICML. JMLR.org, 2024.
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.1, knowledge storage and extraction. In ICML. OpenRe- view.net, 2024. URL httpttps///penneeier... Forum?id=5x B8rqbej.
Gupta, A., Baskaran, S., and Anumanchipalli, G. Re- building ROME Resolving model collapse dur- ing sequential model editing. In EMNLP, pp. 21738-21744, Association for Computational Linguis- tics, 2024a. URL https: aaclanthoooooyyyyyo org/ 2024 emn ponnlp-mar 1210.
Azaria, A., Azoulay, R., and Reches, Chatgpt a remark- able tool-for experts. Data Intelligence, 6:240-296, 2024. URL http::/ppp/.... /ww. sci enciengine / 10.1162/dints 00235.
Gupta, A., Sajnani, D., and Anumanchipalli, G. uni- fied framework for model editing. Al-Onaizan, Y., Bansal, M., and Chen, Y. (eds.), EMNLP, pp. 15403-15418 Association for Computational Linguis- tics, 2024b. URL https : acclllllll 2024. findings emmlp. canlp.903.. 903.
Aziznejad, S. and Unser, M. Deep spline networks with control of lipschitz regularity. In IEEE ICASSP, pp. 3242- 3246, 2019. URL
ieee.
otgg/docueeeee httg/document/B6622
org/
Blum, B. I. History of Computers, pp. 3-32.
Springer Berlin Heidelberg. Berlin, Heidel- berg. 1986. ISBN 778-3-662-26537-6
Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y., and Ghassemi, M. Aging with GRACE: lifelong model editing with discrete key-value adaptors. In NeurIPS, volume 36, pp. 47934-47959. Curran Asso- ciates, Inc., 2023. URL https: /dl. acm. orq/ doi / 5555 22155366666222.6666626 3668201.
doi:
10. 10.11077998833-62-2663-- doi. wtt./////..../991.001
URL https:// 262-2263776_1.
Bohra, P., Campos, J., Gupta, H., Aziznejad, S., and Unser, M. Learning activation functions in deep (spline) neural networks, IEEE Open Journal of Signal Processing, 295-309, 2020. doi: 10. 10.1009/11S.22203000999
Hu, E. J.. yelong shen, Wallis, P., Allen-Zhu. Z., Li, Y.. Wang, S., Wang, L.., and Chen, W. LoRA: Low-rank adaptation of large language models. In ICLR, Virtual, 2022. OpenReview.net. URL net /
Chen, H. Large knowledge model: Perspectives and challenges. Data Intelligence, 6:587-620,
2024. URL http::/tt//// //wttp:/www.... wwwww.sciin. siciennine.... wwm/coo//ooooooo /ooi/ 10. 3794/2099-9 7004.di. 2024. 0001.
Jacob, B.. Ng. S., and Wang, D. Memory Systems: Cache, DRAM, Disk. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2007. ISBN 0123797519.
Dai, D., Dong, L., Hao, Y. Sui, Z., Chang, B., and Wei, Knowledge neurons in pretrained transformers. In ACI,
pp. 8493-8502. Association for Computational Linguis- tics, 2022. URL https:: / // /ooi. org/10.1863/// v1 21/2022. acl acl 1oon..5..55 581.
Jiang, W. X., Nelson, B. L., and Hong, J. Estimating sensi- tivity to input model variance. In WSC, pp.
IEEE Press, 2020. ISBN 9781728132833 URL https: / //informs sim. org www./wwcc.....c..c......cc... 9papers 395. pdf.
Dai, D., Jiang, W., Dong, Q., Lyu, Y., and Sui, Z. Neural knowledge bank for pretrained transformers. NLPCC, volume 14303 of Lecture Notes in Computer Science, pp. 772-783. Springer, 2023. URL wwwpthttt/h//..... http/oiooooooooooo/oooooooooooo. org/ 10.1007 978 031 44696-422222 2_60.
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., and McHardy, R. Challenges and applications large language models. ArXiv, abs/2307.101661 2023. URL https: https//pap......aecee..o.. semanticscholar orq/CorpusID:220 :D:
Dai. Z., Yang, Z., Yang, Y., Carbonell, J., Lc, Q., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond fixed-length context. In ACL, pp. 1978-2988. Association for Computational Linguis- tics, 2019. URL https://aclantholl P19 P19-1285/.
Kolmogorov, A. N. On the representation of continuous functions of many variables by superposition of con- tinuous functions of one variable and addition. Dok- lady Akademii Nauk, 114:953-956, 1957. URL http: //mi wat hnet. . ru/ dan22050.
org/
Fang, J., Jiang, H., Wang, K., Ma, Y., Wang, X., He, X., and Chua, T. Alphaedit: Null-space constrained knowledge editing for language models. In ICLR, 2025. URL https:// http::::///peeeeeee.eeeee. 1dHHVSytv33.h.
Levy, O., Seo, M., Choi, E., and Zettlemoyer, L. Zero- shot relation extraction via reading comprchension. In Levy, R. and Specia, L. (eds.), CoNLL, pp. 333-342. Association for Computational Linguistics, 2017. URL https:// http:///doo.oor////// 11tg//0..165///////1111--00000
10

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
Li, X., Li, S., Song. S., Yang, J., Ma, J., and Yu, J. PMET: precise model editing in transformer. In AAAI, pp- 18564-18572. AAAI Press, 2024, URL https:: /doi. org/10. aaai. v38117.... 29818.
Raunak, V. and Menezes, A. Rank-one editing of encoder- decoder models. In The Second Workshop InterNLP of NeurIPS, 2022. URL https:/ /tt/ org/orr//0. 18550/arXiz wttpaaiwwi.....111.1 13317.
Li, X., Li, S., Song, S., Liu, H., Ji, B., Wang, X., Ma, J. Yu, J., Liu, X., Wang, J., and Zhang, W. Swea: Updating factual knowledge in large language mod- els via subject word embedding altering. AAAI, (23):24494-24502, 2025. URL https://ojs...o.o org/indess php/ AAAI/AAAAAAAAAAAA Att./aaaaceeeeeee////aaea..
Rojas, R. and Hashagen, U. (eds.). The first computers: history and architectures. MIT Press, Cambridge, MA, USA, 2000. ISBN 0262181975.
Singh, A. Exploring language models: A comprehensive survey and analysis. In RMKMATE, pp. 1-4, 2023. doi: 10. 1011009RMMEATT555444.00....... 10334994
Liu, Z., Wang, Y., Vaidya, S., Ruchle, F., Halverson, J.. Soljačić, M., Hou, T. Y., and Tegmark, M. Kan: Kolmogorov-armold networks, 2024. URL https:// arxiv. org/ abs/ / 2404 19756.
Wang. B. and Komatsuzaki, A. GPT-J-6B: A
6 Billion Parameter Autoregressive Language Model. https://github: mesh-transformer- 2021.
Llama Team, A. M. The llama herd of mod- els. CoRR, abs/2407.21183, 2024. URL https:// github. www./o///eeeeeee met 11 / 11ama3/.
Wang, P., Li, Z., Zhang, N., Xu, Z.,, Yao, Y., Jiang, Y., Xie, P., Huang, F,, and Chen, H. WISE: Rethinking the knowledge memory for lifelong model editing of large language models. In NeurIPS, 2024a. URL https: / //openreview.. net/ forum? diw.mm/iiii...
Luo, M., Xu, X., Dai, 7.,, Pasupat, P., Kazemi, M., Baral, C., Imbrasaite, V., and Zhao, V. Y. Dricl: Demonstration- retrieved in-context learning. Data Intelligence, 6:909- 922, 2024. URL http://sttt /wwww sciengine. com/ doi/10. 3024/209622222 7004. di. d..44.4..2224.... 2024. 0012.
Wang, R. and Li, P. Memoe: Enhancing model editing with mixture of experts adaptors. CoRR, abs/2405. 19086, 2024. URL https: 48550/arXiv. 2405 19086.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in GPT. In NeurlPS, pp. 17359-17372 Curran Associates, Inc., 2022. URL https: rome munlabbinfo..
Wang. Y., Gao, Y., Chen, X., Jiang, H., Li, S., Yang. J., Yin, Q., Li, Z.,, Li, X., Yin, B., Shang, J., and McAuley, J. J. MEMORYLLA YLLM: towards self-updatable large language models. In ICML. OpenReview.net, 2024b. URL https: / //openreview. net forum? wtw.rrmmmmppooo//iiiii..
Meng, K., Sharma, A. S., Andonian, A. J., Belinkov, Y., and Bau, D. Mass-editing memory in transformer. In ICLR, Virtual, 2023. OpenReview.net. URL https: // //openreview.oo wtt.:eeetdpd.....eoaaa..t..
Wu, Q., Lan, Z., Qian, K., Gu, J., Geramifard, A., and Yu, Memformer: A augmented transformer for sequence modeling. In Findings of AACL-IJCNLP, pp. 308-318. Association for Computational Linguis- tics, 2022a. URL https: hclonthhlooyy horgg 2022 f wwa.ininaaaaaal..22..
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Man- ning, C. D. Fast model editing at scale. In ICLR, Virtual, 2022a. OpenReview.net. URL https:// openreview net net//forummlld-00ezxeeei
Mitchell, E., Lin, C., Bosselut, A., Manning, C. D., and Finn, C. Memory-based model editing at scale. In ICML, volume 162, pp. 15817-15831. PMLR, 2022b. htttttts///prceeeeeee press/ / vv1 wttte//cmeeeeee/eeeeeeeeeeeeeeee html.
Wu, Y. Rabe, M. N., Hutchins, D., and Szegedy, C. Memorizing transformers. In ICLR. OpenReview.net, 2022b. htRp:: //ppereeie//eeeeew net/ forum? j-xzReff-
mlrr.
Modarressi, A., Köksal, A., Imani, A., Fayyaz, M., and Schuetze, H. MemLLM: Finetuning LLMs to use explicit read-write memory. Transactions on Machine Learning Research, 2025. URL https:t///// hnttps://opereeeee /own/eveeeeeeeeeeeeeeeeeeee net./ forum? hdtmmmdddddgdhddddn
Yang. H., Lin, Z., Wang, W., Wu, H., Li, Z.,, Tang, B., Wei, W., Wang, J. Tang, Z., Song, S., Xi, C., Yu, Y., Chen, K., Xiong, F., Tang, L., and E, W. Memory Language modeling with explicit mem- ory. Journal of Machine Learning, 3(3):300-346,
2024. URL http:://global-eee www.ssii.....eooe.e..oe.oeeeee.e wwt./itrooooooooooooooooooo article www.ddeeaaaaoooo hmlmmm111/19 htmll
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Barnes, N., and Mian, A. A com- prehensive overview of large language models. CORR, abs/2307.00435,, 2024. URL https::///// htttp::/arxiv..rr waaaxiivvoorr./........../.. pdf/2307.0643
Yao, Y., Wang, P., Tian, B., Cheng, S., Li, Z., Deng, S., Chen, H., and Zhang, N. Editing large language models: Problems, methods, and opportunities. In EMNLP, pp-

---

Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying
10222-10240. Association for Computational Linguistics, 2023. URL.ttps: http//aaaantttt wwwttholoy..org////..oooo Homlp-maing 632/.
Yu, L., Chen, Q. Zhou, J., and He, MELO: en- hancing model editing with neuron-indexed dynamic lora. In AAAI, pp. 19449-19457. AAAI Press, 2024. URL /ttp::///oo...////./// 138117.29918
aaai.
Zhang, N., Yao, Y., Tian, B., Wang, P. Deng. Wang, M., Xi, Z., Mao, S., Zhang, J., Ni, Y., Cheng, S., Xu, Z., Xu,
X., Gu, J., Jiang, Y., Xie, P.. Huang, F., Liang, L., Zhang,
Z., Zhu, X., Zhou, J., and Chen, H. A comprehensive study of knowledge editing for large language models. CoRR, fbss/2001.0186,, 2024. URL https://cee org/10. 48550/8rXiv 6128.
Zheng, C., Li, I.., Dong, Q. Fan, Y., Wu, Z., Xu, J., and Chang, B. Can we edit factual knowledge by in-context learning? In EMNLP, pp. 3862-4876. Association for Computational Linguistics, 2023. URL /attpp:///ci. org/10. 011/1...166331111 vv1 /223.. emnlp-main. 296.
Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, X., and Kumar, S. Modifying memories in transformer models. CoRR, abs/2012.00363,, 2020. URL https:/ http://arvvvvo///e httpa/awwww.gg///////////./// 2002 00363.
12


Reviews:
[{"review_id": "OWVGQ7WV9v", "reviewer": "Reviewer_gZ6v", "summary": "This work proposes Peripheral Memory for LLMs, in which the sequence modeling and the memory updates interleaves in the language modeling process. The experimental results on knowledge-based model editing and long-context QA demonstrate the effectiveness of such method.\n\n## update after rebuttal\nincreased by 1 score due to the helpful discussions from authors on the generality of the work", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.Question about $\\mathbf{W}_0$ and $\\mathbf{W}_1$**\n\nThank you for raising this important concern. We clarify the training protocol and evaluation fairness as follows:\n\n*(1) Role and Training of Convertors*\n\nThe mapping matrices $\\mathbf{W}_0$ and $\\mathbf{W}_1$ (see Section 3.2) serve solely as feature space adapters, analogous to \"plug-and-play\" connectors between the LLM’s hidden states and the memory module. Their sole purpose is to align dimensionality between the LLM’s hidden space and the memory’s representation space, not to encode task-specific knowledge. `Specifically, the convertors were trained solely to memorize contexts into memory (See Line-986, 587, 273 of code), with no exposure to query data or ground-truth answers during training.` This ensures the convertors do not learn task-specific knowledge.\n\n*(2) Ensuring No Data Memorization*\n\nTo rigorously verify that convertors do not memorize evaluation data, we also implemented an `empty_fn` (similar to the `memory_fn` in Line-508 of supplementary code):\n\n```\ndef empty_fn(self, layer_idx, hidden_states, causal_masks, **kwargs):\n    if layer_idx == self.merge_layer_idx:\n        empty_feats = self.convertor(self.query_signal.to(self.convertor.device))\n        hidden_states[:,self.replace_idx,:] = empty_feats.to(hidden_states.device)\n        causal_masks = self._replace_attn_fn(causal_masks, 1., self.replace_idx)\n    if layer_idx > self.merge_layer_idx:\n        causal_masks = self._replace_attn_fn(causal_masks, 1., self.replace_idx)\n    return hidden_states, causal_masks\n```\n\nThis function skips memory retrieval and aligns the LLM’s output with its original predictions. However, extensive experiments show no statistically significant difference in model performance ($\\Delta$ < 0.3%) between models with and without `empty_fn`, confirming that convertors introduce no hidden memorization. *In consideration of efficiency and simplicity, we omitted this part directly and did not provide it in the supplementary materials.*\n\n**2.Question about the variable l in Section 2.1**\n\nThank you for highlighting this important point. The variable $l$ in Section 2.1 denotes the index of the hidden layer from which query features are extracted for memory retrieval. While $l$ is indeed a critical hyper-parameter, we relegated its analysis to Appendix D.3 due to space constraints.\nFindings can also be found in `Response#2 of Reviewer UVbc`.\n\n\n\n**3. Question about using KAN**\n\nWe appreciate your thoughtful critique and address the motivation for using KANs (Kolmogorov-Arnold Networks) as follows:\n\n**(1) Motivation for KANs**\n\nAs formalized in Eq.2, the memory query process can be abstracted as a smooth mapping from query signals to memory data. Physical RAM approximates this mapping with an indicator function, which is ill-suited for neural memory systems requiring gradual state transitions. KAN was chosen over MLP due to its superior symbolic regression capabilities:\n\n- `Superior Function Approximation`: KANs achieve higher precision with fewer parameters by leveraging spline-based nonlinearities, avoiding MLPs’ reliance on rigid activation functions (e.g., ReLU).\n- `Smoothness`: KANs’ piecewise polynomial basis functions enable $C^2$-continuous mappings, critical for stable gradient propagation during sequential memory interactions.\nWe will revise the title of Section 2.2 in future editions and provide further clarification in this section.\n\n**(2) Empirical Comparison with MLPs**\n\nWe compared KANs against MLPs with varying parameter counts:\n|Type||ZsRE||||||CF|||\n|:-----|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|-----:|:-----:|-----:|-----:|\n||Efficacy|Generality|Locality|Score|||Efficacy|Generality|Locality|Score|\n|**KAN**$\\approx$0.1932M|**0.9774**|**0.6432**|**1.0000**|**0.8735**|||**0.9915**|**0.3108**|**1.0000**|**0.7674** |\n|**MLP**(\\#KAN-param $\\times 1$)$\\approx$0.0219M|0.8672|0.5954|0.3649|0.6092|||0.8675|0.2410|0.0240|0.3775|\n|**MLP** (\\#KAN-param $\\times 2$)$\\approx$0.0434M|0.8835|0.5883|0.3652|0.6123|||0.8742|0.2370|0.0315|0.3809|\n|**MLP** (\\#KAN-param$\\times 4$)$\\approx$0.0864M|0.8973|0.5785|0.3747|0.6168|||0.8835|0.2295|0.0330|0.3820|\n|**MLP** (\\#KAN-param$\\times 5$)$\\approx$0.1079M|0. 3989|0.3852|0.0000|0.2614|||0.8330|0.2104|0.0230|0.3555|\n|**MLP** (\\#KAN-param$\\times 8$)$\\approx$0.1723M|0. 0000|0.0000|0.0000|0.0000|||0.0000|0.0000|0.0000|0.0000|\n\nNote: Where the MLP is used to model a specific memory bank. Each MLP memory bank mirrors KAN’s structure:\n- First&Confidence bank: torch.nn.Linear(1,9) $\\to$ torch.nn.Linear(9,1)\n- Other banks: torch.nn.Linear(2,9) $\\to$ torch.nn.Linear(9,1)\n- Hidden dimensions scaled via torch.nn.Linear(1,9$\\times X$) $\\to$ torch.nn.Linear(9$\\times X$,1), #Epoch=10.\n\n*Results indicate that the KAN-based memory bank surpasses the MLP-based counterpart within our peripheral memory.* \n\n*For a more detailed analysis, please refer to `Response #3 of Reviewer TWi2` due to chararacter limitation of current response.*", "claims_and_evidence": "No, check evaluations and weakness.", "methods_and_evaluation_criteria": "I checked the code in supplementary material. I found the functions for training on KME, TME and LongBench. The author do not mention the details in the paper. But I believe this method requires data-specific fine-tuning to train its \"convetor\" module. Thus, most of the comparisons lead to unfair comparisons because your model is trained on LongBench but other long-context models are not trained on LongBench data.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "Check evaluations.", "relation_to_broader_scientific_literature": "This method is quite impressive because the designs are very lightweight. Compared with this method, MemoryLLM (Towards Self-Updatable Large Language Models) requires the parametric memory at each layer, which consumes quite a lot of GPU memory.", "essential_references_not_discussed": "1. M+: Extending MemoryLLM with Scalable Long-Term Memory\n\n2. CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory\n\n3. Augmenting Language Models with Long-Term Memory", "other_strengths_and_weaknesses": "Strength:\n1. The designs of MemoryBank is motivated by the RAM and make a lot sense. The working flow is clear and can captures the global memory information.\n\n\nWeakness:\n\n1. The motivation of Kolmogorov–Arnold Networks design are invisible and confused. I understand it is a novel architecture which can arouse the attention from the community. But I believe using MLP here rather than KAN will make the design clearer and more intuitive. Additionally, using MLP is very likely to yield better performance in your case.\n\n2. If I understand the method clearly, this method is not a training-free method. The W_0 and W_1 mapping matrices in Section 3.2 are newly initialized and they require training for alignment. But I am sure that I did not find any details about the training for such mapping weights as well as the KAN network. If you use a general SFT dataset to adapt your foundational LLM into a LLM with memory, then please introduce your training and dataset details. If you perform fine-tuning on the training data on each downstream task, your evaluations are completely unfair.\n\n3. The variable l in Section 2.1 should be a hyper-parameter, but I did not find any value about this hyper-parameter as well as the ablation studies on that.", "other_comments_or_suggestions": "Check weakness", "questions_for_authors": "Check weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HCdSadFhDE", "reviewer": "Reviewer_UVbc", "summary": "This paper proposes a novel memory augmentation technique for LLMs by decoupling memory from the model architecture, analogous to a CPU and RAM architecture. The proposed peripheral memory consists of sequential memory banks modeled by Kolmogorov-Arnold Networks (KAN) to have smooth and adaptive memory read/write operations controlled by internal LLM states. The framework integrates retrieved memory content with an adaptive confidence masking mechanism. The experiments demonstrate effectiveness in knowledge-based model editing and long-context question answering.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.Question about the semantic drift**\n\nThank you for this insightful question. We acknowledge that semantic drift and retrieval degradation at extreme scales (>10K updates) for semantically equivalent queries remain challenges. Below, we summarize our empirical observations and outline explicit mitigation strategies under development:\n\nAs discussed in Appendix-C, our memory is directly queried using the hidden state of the final input token. While efficient, this design inherits a challenge in representation learning: geometric misalignment in high-dimensional spaces. That is, small differences in inputs can lead to disproportionate shifts in representations. Thus, semantically equivalent queries may map to distinct regions of the memory space due to minor differences in token-level representations. This reduces the robustness of rephrased inputs, as shown in Table 2. Additionally, as memory utilization scales, the module becomes overly specialized to the original query distribution (stability), sacrificing generalization to representationally divergent queries (plasticity). \n\nTo address the limitations of the direct memory querying strategy, we are now actively developing a memory management module inspired by Memory Management Unit (MMU) in operating systems. The module acts as an abstraction layer between the LLM and peripheral memory, decoupling semantic alignment from storage operations. This allows the peripheral memory to specialize in efficient storage/retrieval, while the MMU handles query normalization, introducing three key innovations:\n\n*(1) Semantic-Aware Querying*\n\nThe MMU parses and refines raw query signals (e.g., token-level hidden states) into semantically enriched descriptors, mitigating hypersensitivity to input variations. For example, paraphrased queries like \"What is the capital of France?\" and \"Name France’s capital city\" would be mapped to unified descriptors, enabling robust retrieval regardless of surface-form differences.\n\n*(2) Optimized Memory Operations*\n\nThe MMU supports bulk memory read/write operations, reducing overhead for large-scale edits. Inspired by virtual memory paging, it dynamically groups related memory entries (e.g., knowledge about a specific entity) into contiguous blocks, improving cache utilization.\n\n*(3) Adaptive Scheduling Policies*\n\nLeveraging reinforcement learning, the MMU learns optimal policies for memory allocation and eviction. This balances hot (frequently accessed) and cold (rarely used) memory regions, addressing the stability-plasticity trade-off while minimizing fragmentation.\n\nThis MMU introduces OS-inspired abstractions (e.g., memory pages) to LLMs, enabling systematic memory control. We are currently refining this architecture and will introduce it in future work.\n\n**2.Question about sensibility of query**\nThe performance of our framework exhibits moderate sensitivity to the choice of query representation. Below, we analyze two key dimensions of this sensitivity:\n\n- Token Position Sensitivity: Using the last token hidden state as the query signal is an intuitive choice since it aggregates semantic information from the entire input. In addition, we also do an experiment using the average of all hidden token features as query, and find similar results. This shows that the selection of token features is relatively stable.\n|Type||ZsRE||||||CF|||\n|:-----|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|-----:|:-----:|-----:|-----:|\n||Efficacy|Generality|Locality|Score|||Efficacy|Generality|Locality|Score|\n|**Last Token**|0.9774|0.6432|1.0000|0.8735|||0.9915|0.3108|1.0000|0.7674|\n|**Average**|0.9698|0.6399|1.0000|0.8699|||0.9800|0.2520|1.0000|0.7440|\n\n- Layer-Wise Variability: Deeper layers (e.g., layer 8-31 in Llama3-8B) yield more stable query representations due to their focus on high-level semantics, while shallower layers (e.g., layer <8) exhibit higher variance (See Figure 12 in Appendix D.3). This means that query performance is relatively stable as long as it is within reasonable limits. \n\nAdditionally, as noted in prior responses, the improved query mechanism (MMU) will decouple semantic alignment from query representation by aggregating hidden states across multiple layers at the same time. This could balance low-level syntactic and high-level semantic features, reducing positional bias and improving robustness.\n\n**3.Comparison with Recent RAG**\nThank you for emphasizing the importance of comparison with recent RAG. We have provided experimental analysis in Appendix D.1. Briefly, we evaluate our method against recent LongRAG:\n||Qasper|MutilfieldQA-en|\n|:-----|:-----:|-----:|\n|C-500$\\times$3|22.5|39.5|\n|B-500$\\times$3|20.4|26.2|\n|LongRAG|15.5/26.3$^*$|38.9/49.4$^*$|\n|Ours|30.1|43.3|\n\nLongRAG: Results with * use Document-level retrieval; others use Passage-level (1 retrieval unit).\n\n`4.​Limitation discussion:see Response #1.`\n\n`5.​Additional results on diverse LLM architecture will be provided later due to current character constraints.`", "claims_and_evidence": "The authors claim improvements in scalability, reusability, and configurability of memory augmentation. These claims are convincingly supported by extensive experiments. It would be better if the experiments involve more diverse LLM architectures.", "methods_and_evaluation_criteria": "The method is novel and well-justified. The evaluation criteria, including benchmarks such as ZSRE, COUNTERFACT, Qasper, and HotpotQA, are appropriate and widely recognized. One minor weakness is the absence of comparison to retrieval-augmented generation (RAG) techniques, particularly in the QA tasks, which are directly related.", "theoretical_claims": "The theoretical explanation is clear and correctly aligns with established literature on smooth nonlinear mapping networks.", "experimental_designs_or_analyses": "The experimental design is sound and clearly articulated, providing a fair and extensive comparison against state-of-the-art baselines. A potential bias arises in the choice of baselines, as recent RAG methods  were not explicitly compared.", "relation_to_broader_scientific_literature": "The paper situates itself clearly within the existing memory-augmentation literature for LLMs.", "essential_references_not_discussed": "[1] Jiang, Ziyan, Xueguang Ma, and Wenhu Chen. \"Longrag: Enhancing retrieval-augmented generation with long-context llms.\" arXiv preprint arXiv:2406.15319 (2024).\n\n[2] de Jong, Michiel, et al. \"Fido: Fusion-in-decoder optimized for stronger performance and faster inference.\" arXiv preprint arXiv:2212.08153 (2022).", "other_strengths_and_weaknesses": "Strengths:\n\nThe idea that in conceptualizing memory architecture analogous to RAM-CPU structure is novel and interesting. \nThe empirical performance outperforms current state-of-the-art methods significantly.\nComprehensive analysis of scalability and configurability, clearly showing real-world applicability.\n\nWeaknesses:\n\nLimited discussion regarding limitations or failure cases of the proposed approach, particularly in cases where retrieval signals might degrade or when memory banks encounter interference at extremely high capacities.\nLack of comparison with advanced RAG methods.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could you provide a detailed analysis or empirical investigation into how semantic drift or retrieval quality degradation is mitigated (or worsens) at extremely large storage capacities (e.g. >10K updates)? Have you considered explicit drift-mitigation methods such as periodic re-indexing or semantic clustering?\n2. How sensitive is the performance to the specific choice of query representation?\n3. Could you discuss comparison to recent advanced retrieval-augmented generation methods, particularly on long-context question answering tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HpkeAZ1Zox", "reviewer": "Reviewer_TWi2", "summary": "The paper introduces a \"peripheral memory\" memory augmentation method for LLMs. The paper views memory as a separate ram-like component that interfaces with an LLM. The memory is designed as a sequence of memory banks, each modeled using KANs. Memory operations are controlled by query signals from the LLM internal states and dedicated memory bank generates mask values indicating relevance of retrieved data. The paper claims to improve over limitations of prior methods by enhancing scalability, reusability and configurability.  The method is evaluated on knowledge-based model editing and long-context Q&A.\n\n## Update after rebuttal: I maintain my score as my main questions are addressed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Handling conflict**\n\nThank you for raising this critical point. Below, we clarify our method’s current behavior and outline potential enhancements:\n\n**(1) Last-Write-Wins Policy**\n\nIn sequential editing scenarios, our memory defaults to a temporal priority strategy: the most recent edit overwrites previous entries for the same fact. This is reflected in the confidence mechanism, where newer entries are assigned higher trust values due to their proximity in the query distribution.\nWhile this ensures consistency in retrieval (only the latest edit is returned), it does not explicitly resolve semantic contradictions. However, the confidence bank can be easily configured into detect contradictions using off-the-shelf entailment classifiers. $\\Downarrow$\n\n**(2) Conflict-Aware Confidence**\nIntegrate semantic similarity checks between new edits and existing memory entries. For example, if a new edit contradicts a stored fact (measured via entailment models, the confidence bank could trigger a conflict resolution protocol (e.g., human-in-the-loop verification or probabilistic truth maintenance). Notably, this enhancement can be easily implemented as a configurable extension to our peripheral memory without altering the core LLM.\n\n*Actually, we are now actively studying a version-controlled peripheral memory inspired by database systems, where conflicting edits are stored as alternate branches. This could enable users to query the memory with temporal constraints (e.g., “What was believed about xxx in 2023?”).*\n\n**2. Scale to even more edits**\n\nOur framework currently employs a direct memory querying strategy, which achieves reliable storage accuracy for up to 10K edits (see Figure 7). However, as noted in Section 5.4, while the memory retains high storage fidelity at scale, its generalization degrades for semantically equivalent queries due to geometric misalignment in hidden feature spaces (see Appendix C). To balance robustness and scalability, we adopt a memory archival protocol: when the active memory reaches 1K entries (optimal for generalization), it can be archived, and a fresh memory will be initialized. This approach enables theoretically unbounded storage capacity while maintaining generalization performance thanks to the unique properties of our **peripheral** memory component. Notably, even with a 1K entry limit, our memory achieves high effective storage density (4.95, see Table 4).\n\nWhile this archival strategy significantly improves scalability, it trades off per-bank memory utilization. To address this, we are developing a memory management component (MMC), inspired by the Memory Management Unit (MMU) in operating systems. The module maps semantically equivalent inputs to unified memory space using contrastive learning, decoupling semantic alignment from storage operations. This allows the peripheral memory to specialize in efficient storage/retrieval, while the MMC handles query normalization and address translation.\n\n**3. Importance of KAN**\n\n`Response #3 of Reviewer gZ6v` provides detailed experimental results.\nThese indicate that the KAN-based memory bank surpasses the MLP-based counterpart within our peripheral memory. \n\n*(1) Accuracy vs. Generalization Trade-off*\n\nKAN demonstrates exceptional performance in both accuracy and generalization, while MLPs (even at 4$\\times$ params) plateau at 89.7% accuracy and 57.8% generalization.\nMLPs’ noisy memory outputs degrade Locality, as seen in CounterFact: KANs retain 100% Locality vs. MLPs’ 2.3%-3.3%.\n\n*(2) Scaling MLPs Fails to Close the Gap*\n\nIncreasing MLP hidden dimensions marginally improves storage accuracy (e.g., 89.7% at 4× params vs. 86.7% at 1× on ZsRE), but harms generalization due to overfitting on memorizing data. This reflects the inability of MLPs to learn smooth mappings.\n\n*(3) Catastrophic Collapse at Extreme Scaling*\n\nAt 8$\\times$ parameters, MLP performance collapses to 0 across all metrics. We attribute this to:\n- Optimization Instability: Overparameterized MLPs suffer from vanishing/exploding gradients, exacerbated by the memory’s sequential architecture (sequence length = 512).\n- Loss Landscape Degradation: High-dimensional MLP weights create chaotic loss surfaces, preventing convergence. In contrast, KANs’ spline parameterization inherently regularizes the optimization landscape.\n\n**(3). Why MLPs Underperform**\n\n- Approximation Theoretic Limitations: MLPs struggle to model the compositional structure of memory mappings (query→key→value), which KANs explicitly encode via Kolmogorov-Arnold superposition[1].\n- Noise Amplification: MLPs’ fixed activations amplify high-frequency noise in memory queries, degrading generalization. KANs’ adaptive splines act as low-pass filters, suppressing noise[2].\n\n**References**\n\n[1] Liu et al. (2024). Kan: Kolmogorov-arnold networks.In ICLR.\n\n[2] Prautzsch et al. (2002). Bézier and B-Spline Techniques. Mathematics and Visualization. Springer Science & Business Media. ISBN 978-3-540-43761-1.", "claims_and_evidence": "Yes, the claims around scalability, reusability and configurability generally seem pretty well supported by the strong performance on the knowledge editing, long context QA, shared memory across LLM experiments, and experiments on memory bandwidth, depth and allocation experiments.", "methods_and_evaluation_criteria": "Yes, the paper appears to evaluate the method on standard benchmarks for measuring memory effectiveness and compare to a variety of baselines.", "theoretical_claims": "The paper is empirical based without any major theoretical claims.", "experimental_designs_or_analyses": "Yes the experiments seem to be well designed and take into account relevant baseline methods, different LLM backbones and varied settings.", "relation_to_broader_scientific_literature": "The paper positions itself clearly in relation to existing memory augmentation approaches to LLMs, working memory, implicit memory and explicit memory and their limitations.", "essential_references_not_discussed": "In general it seems fine, however a more detailed discussion and comparison with RAG approaches would strengthen the paper. There is a brief discussion and comparison in the appendix, but this seems relevant enough to be included in the main paper.", "other_strengths_and_weaknesses": "Strengths:\n- The overall approach appears novel and creative\n- The comparison with physical RAM is intuitive\n- The empirical results seem convincing\n- The ability to share memory across different LLMs seems promising for practical deployments\n\nWeaknesses:\n- As mentioned above, a more detailed comparison and discussion of RAG would strengthen the paper\n- While some motivation of KANs is introduced, no empirical comparisons of this are made, leaving the reader to wonder how important they actually are to the method\n- In addition, while sprinkled throughout the paper, a more explicit discussion of the limitations of this work and the proposed methods and future improvements would strengthen the paper.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How does the approach handle conflicting information or multiple edits about the same fact? The paper demonstrates how the confidence bank can mask out irrelevant knowledge, but I do not believe it addresses the case of contradictory knowledge?\n2. How will the approach scale to even more facts or edits, e.g. 100K or 1M or more?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n6DI7xkLYL", "reviewer": "Reviewer_Eb4F", "summary": "This paper proposes peripheral memory, which is inspired by RAM architecture. It focuses on the task of model editing, and significantly outperforming previous methods. The peripheral memory seems to add external memory into the process of LLM inference, editing some layers in the foundation model. Although the proposed method seems interesting, I'm still confused by some details.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Question about baselines**\n\nThank you for raising this important concern. We provide detailed clarifications regarding the baseline implementation:\n\n**(1) Baseline Implementation**\n\nAll baselines for Knowledge-based Model Editing (KME) were implemented using the widely adopted toolkit EasyEdit (https://github.com/zjunlp/EasyEdit), ensuring reproducibility, reliability and alignment with established practices. For instance, in the case of WISE, we strictly followed the official implementation guidelines, with our editing code closely mirroring the provided examples:\n\n```\nfrom easyeditor import WISEHyperParams\nhparams = WISEHyperParams.from_hparams('./hparams/WISE/llama-3-8b.yaml')\neditor = BaseEditor.from_hparams(hparams)\nmetrics, edited_model, _ = editor.edit(\n        prompts=prompts,\n        rephrase_prompts=rephrase_prompts,\n        target_new=target_new,\n        subject=subject,\n        sequential_edit = True,\n        locality_inputs=locality_inputs,\n)\n```\n\nThe hyperparameters are suggested by the EasyEdit.\n\n**(2) Low Baseline Performance**\n\nWe observed that several baselines (e.g., PMET, MEMIT) exhibit zero performance in consecutive editing scenario. Initially surprised by these results, we rigorously repeated the experiments 10+ times and consistently observed the same outcome.\n\nKey Insight: When the number of sequential edits exceeds 1K, these localization-based editing methods suffer from catastrophic performance degradation, mainly arising from the repeated modifications progressively destabilize the LLM’s parameters. This phenomenon aligns with observations in recent works [1,2], which highlights the fragility of localized edits under high edit counts.\n\n**(3) Reproducibility of our method**\n\nFull implementation details and hyperparameters are provided in the supplementary material. For instance, refer to `memory.py` (Lines 1073–1075) for key parameters such as memory depth and grid size. Additionally, they have also been discussed in Section 4.3 for clear description. We welcome further discussion or code review to address any remaining concerns.\n\n**2.Questions about Llama3.1-8B as backbone**\n\nThank you for your insightful feedback. We address your concerns as follows:\n\n**(1) Token Limit of Llama3-8B**\n\nThe token limitation of Llama3-8B (8k tokens) does not impact our experiments, as the input sequences in KME datasets (ZsRE and CounterFact) are well within this constraint: ZsRE: Maximum input length = 36 tokens and CounterFact: Maximum input length = 56 tokens. Thus, even for 3,000 sequential edits, the total token count remains far below the 8k limit, impossibly impacting model performance. Thus, the observed performance limitations of baseline methods (e.g., MEMIT, PMET) are not attributable to token constraints but to inherent challenges in parameter-localized editing [2].\n\n**(2) Performance under longer token limit**\n\nWe acknowledge that token limits could theoretically hinder prompt-based methods. For comprehensive comparison, we conducted additional experiments on Llama3.1-8B (128k token limit) and compared our method with IKE, a state-of-the-art prompt-based editing method that uses in-context learning without parameter updates.\n| Type | | ZsRE | | | | | | CF | | |\n| :----- | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | -----: | :-----: | -----: | -----: |\n|| Efficacy | Generality | Locality | Score | | | Efficacy | Generality | Locality | Score |\n| **Original** | 0.2287 | 0.5211 | 1.0000 | 0.5832 | | |    0.0043 | 0.0040 | 1.0000 | 0.3361 |\n| **IKE** | *0.5232* | 0.5231 | 0.5190 | 0.5218 | | |    0.0055 | 0.0040 | 0.6725 | 0.2273 |\n| **Ours** | *0.9919* | 0.6010 | 1.0000   | 0.8643 | | |    1.0000 | 0.2875 | 1.0000 | 0.7625 |\n\nOur method achieves superior performance even with extended context lengths, confirming that token limitations are not the bottleneck in our setup. This also confirms that our framework’s advantages are architecture-agnostic and not contingent on context length. \n\n**3. Explanation of Figure 1 and Figure 2** and **The function $g$ and $h$**\n\nFigure 1 provides an overview of our framework, while Figure 2 illustrates the peripheral memory. \n\nThe functions $g_i^k$ and $h_{i1}^k$ are learnable univariate functions, parameterized as B-spline curves. For a detailed understanding, please refer to Chapters 5.1 and 5.2 [3] for more details.\n\n`Due to current character constraints, we are unable to provide a more detailed explanation here but will offer a comprehensive discussion in later response.`\n\n**References**\n\n[1] Wang et al. 2024. WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models. In NeurIPS.\n\n[2] Li et al. 2024. Consecutive Batch Model Editing with HooK Layers. In EMNLP, 13817–13833. Association for Computational Linguistics.\n\n[3] Prautzsch et al. 2002. Bézier and B-Spline Techniques. Mathematics and Visualization. Springer Science & Business Media. doi:10.1007/978-3-662-04919-8. ISBN 978-3-540-43761-1.", "claims_and_evidence": "Good. The author claims the proposed method can greatly address the task of model editing, and the experiments have verified its effectiveness.", "methods_and_evaluation_criteria": "I'm still confused by some details in terms of the methods and evaluations.\n1. Figure 1 and Figure 2 need more explanations. It is a bit hard to understand.\n2. What is the meaning of the function g and h? Could you please explain with some detailed examples, such as Llama architecture?\n3. I think Llama3-8B is not suitable as the backbone to evaluate consecutive editing. I notice that the comparisons are under 3k updates. However, the maximum number of tokens of Llama3-8B is limited to 8192 tokens, which is obviously less than 3k updates (it seems more than 300k tokens, assuming 100 tokens for each update). Therefore, most prompt-based editing methods may fail, due to the token limitation. I think the author should use Llama3.1-8B with the maximum length of 128k tokens, or GPT-4o with 128k tokens as well.\n4. How do you implement the baselines in your experiments? The results of baselines are too low (even with 0.00), and the proposed method seems significantly high. I would prefer to request more details about the experimental results. For example, how many times the experiments are repeated?", "theoretical_claims": "No theoretical claims in this paper.", "experimental_designs_or_analyses": "See \"Methods And Evaluation Criteria\".", "relation_to_broader_scientific_literature": "Yes, very relevant.", "essential_references_not_discussed": "[1] Zhong, Wanjun, et al. \"Memorybank: Enhancing large language models with long-term memory.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.\n\n[2] Packer, Charles, et al. \"Memgpt: Towards llms as operating systems.\" arXiv preprint arXiv:2310.08560 (2023).", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "I think the author should respond to the above questions and concerns. I'm willing to raise my score if the response can address my concerns.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel memory augmentation framework for LLMs that treats memory as a peripheral module, inspired by RAM-CPU architecture, to enable efficient long-term knowledge updates. Using Kolmogorov-Arnold Networks (KAN) for smooth memory transitions and dynamic read/write operations, the system enhances model predictions without degrading performance. Experiments show strong results in knowledge editing and long-context question answering.

The paper was reviewed by four experts with final decisions of two Accepts, one Weak Accept and one Weak Reject. All reviewers appreciated the premise of the work, especially its motivation by RAM to use a peripheral memory with contemporary deep learning models, as well as its strong results and comprehensive analysis. There were initial concerns around the motivation for KANs (R#gZ6v, R#TWi2), comparison with RAG (R#UVbc, R#TWi2), need for discussion of limitations and failure cases (R#UVbc, R#TWi2), and some clarification questions (R#Eb4F). The authors' responses addressed most concerns, and R#gZ6v increased their score considering the authors' clarifications on the generality of their method. While there is one reviewer with a borderline negative view, their concerns were more around clarifications which are largely addressed in the rebuttal. Given these considerations, the paper is recommended for acceptance. The authors are however recommended to integrate their rebuttal responses in the final version.
Decision: Accept (poster)