MemLLM: Finetuning LLMs to Use Explicit Read-Write Memory
Authors: Ali Modarressi
Abdullatif Köksal
Ayyoob Imani
Mohsen Fayyaz
Hinrich Schuetze
Venue: Accepted by TMLR ()

Abstract: While current large language models (LLMs) perform well on many knowledge-related tasks, they are limited by relying on their parameters as an implicit storage mechanism.  As a result, they struggle with memorizing rare events and with updating their memory as facts change over time. In addition, the uninterpretable nature of parametric memory makes it challenging to prevent hallucination. Model editing and augmenting LLMs with parameters specialized for memory are only partial solutions. In this paper, we introduce MemLLM, a novel method of enhancing LLMs by integrating a structured and explicit read-and-write memory module. MemLLM tackles the aforementioned challenges by enabling dynamic interaction with the memory and improving the LLM's capabilities in using stored knowledge. Our experiments indicate that MemLLM enhances the LLM's performance and interpretability, in language modeling in general and knowledge-intensive tasks in particular. We see MemLLM as an important step towards making LLMs more grounded and factual through memory augmentation. The project repository is publicly available at: https://github.com/amodaresi/MemLLM

Published in Transactions on Machine Learning Research (04/2025)
MemLLM: Finetuning Memory
LLMs to Use Explicit Read-Write
Ali Modarressi A Abdullatif Köksal Ayyoob Imani Mohsen Fayyaz', Hinrich Schüitzel.2 1.2 amodaresi@cis. lis...e
Center for Information and Language Processing, LMU Munich, Germany Municc Center for Machine Learning, Germany Micrrssft, Berli, Cermany
Reviewed on OpenReview: https: // optpreveew.ewe.... net/ forum? id= dghh7sOudh
Abstract
While current. large language models (LLMs) perform well on many knowledge-related tasks, they are limited by relying on their parameters an implicit storage mechanism. As result, they struggle with memorizing rare events and with updating their memory as facts change over time. In addition, the uninterpretable nature of parametric memory makes it challenging to prevent hallucination. Model editing and augmenting LLMs with parameters specialized for memory are only partial solutions. In this paper, we introduce MEMLLM, a novel method of enhancing LLMs by integrating a structured and explicit read- and-write memory module. MEMLLM tackles the aforementioned challenges by enabling dynamic interaction with the memory and improving the LLM's capabilities in using stored knowledge. Our experiments indicate that MEMLLM enhances the LLM's performance and interpretability, in language modeling in general and knowledge-intenan intensive tasks in particular. We see MEMLLM as an important step towards making LLMs more grounded and factual through memory augmentation. The project repository is publicly available al: https: htggthuu..co//mmoo///ae.
//github. ctt/:/cccaaaaaaaa aww.aresddees//////mmm MMemLLM
1 Introduction
State-of-tho-art large language models (LLMs) perform well in knowledge-intensive tasks Yuu et al., 2023 Chowdhery et al. 2023). They solve these lasks utilizing the information memorized in their vast array of parameters (Roberts al. 2020). However, the effectiveness of parameter-basee memorization is limited for infrequent entities and concepts (Kandpal et al. 2023 Mallen al. 2023) and prone temporal degradation (Kasai al. 2023 Jang et al. 2022). Parametric model editing may address some of these
issues (Sinissin et al.] 2020 De Cao et al.
2021 Mitchell et. al. 2022 but struggles with maintaining
locality possibly damaging model performance in unrelated areas (Yao et al. 2023c Gu et al., 2024
Moreover, model editing often deteriorates performance when applying sequential editing or batch updates. This is because it primarily focuses on applying (and evaluating) single edits one mu-bby-one (Huang et al. 2023 Finally, model editing may struggle to generalize and maintain previous edits when updating multiple facts simultaneously (Yao al.) 2023c].
Other parametric solutions, like augmenting LLMs with extra parameters such memory pools can preserve knowledge for subsequent utilization (Wang et al. 2023a. 2024b). However, parametric memorization is prone to distortion and hallucinated nonfactual output. In addition, parametric mechanisms like memory pools
have limited capacity and lack interpretability (Maynez et. al. 2020 Ji et al. 2023
augment LLMs with a non-parametric memory component that interacts with the LLM either through natural language text or a formalized API (Wang et al., (023b). Although prior work has demonstrated enhanced abilities in extended dialogs, long-text generation and question answering
Another approach

---

Published in Transactions on Machine Learning Research (04/2025)
Packer et al. 2023 Hu et al 2023; Zhou et al., 2023 these methods are primarily prompt-dependent and necessitate customization for each specific task and model. They also suffer from the lack of structured memory. This undermines interpretability and interoperability (Wang et al.) 2023b). While Retrieval- Augmented Generation (RAG) methods (Lewis et al.) 2020) can provide updated facts, their unstructured storage complicates fact editing. Updating an atomic lact requires modifying all related instances to prevent contradictions when conflicting facts are retrieved together (Shi et al. 2023).
In this paper, we introduce MEMLLM, an LLM endowed with an explicit memory component. This archi- tecture has the following key characteristics.
This explicil memory component has the general advantages of some of the memory-focused work we discussed: we can keep information accessible indefinitely, beyond the context. window, including infrequent information that standard LLMs struggle with.
The LLM has both read and write access to the explicit memory component, i.e., it can store information in the memory as it processes text (or interacts with a user) and retrieve it when it needs it.
We adopt finetuning to train the model to access the explicit memory component through read and write commands. To this end, we specify an API for read and write access. Based on the APT specification, we create a dataset with training examples of API read and write commands and finetune the LLM on it. Our published training dataset can be used to finetune any language model, endowing it with an explicit memory component without requiring architectural changes.
The memory component has an explicit structured schema, similar to a database schema. Therefore, is interpretable and inspectable for humans; is editable by humans; it is scalable since databases have excellent scalability properties; and it interoperable since the contents of the memory can be exported (e.g., to different LLM supporting explicit memory) and contents can be imported from data resources (e.g., from Wikidata).
Our evaluation on Ro-DocRED TTnn al. 2022 demonstrates that MEMLLM achieves better perplexity compared to baselines without memory components, with strong gains named entities. We also show that MEMLLM outperforms cet-memory-baas based methods on knowledge editing.
2 Related work
External memory augmentation. Augmenting an LLM with memory as an external component can enhance its ability to process larger contexts and maintain reliable records by storing facts and knowledge. Such components include databases, knowledge bases and knowledge graphs that LLMs interact with via
natural or formal language (Guu et al.) 2020 Lewis et al. 2020 Liu al.) 2022 Yao al. 2023a Park al.,
2023
Zhou et al.)
2023 Schick et al.] 2023 Hu ct al.) 2023). For instance, Retrieval Augmented Generation
RAAG
retrieves relevant text snippets from large document databases, to improve factuality (Guu et all
2020
2020). Other recent solutions store summarized information from previous contexts for future retrieval, improving performance in long-form generation, summarization, question answering and
[ Lewis et al.
dialog coherence (Park et al. 2023 Zhou et al. 2023 Packer et al. 2023 Chen et al.] 2023 Liang et al.
2023
In general, our framework aligns with external memory methodologies but stands out with its structured format for storing information. This explicit memory facilitates large-scale knowledge editing and makes the model's output generation process more interpretable. While similar structured storage approaches exist, they are often task-specific. For example, Hu et al. (2023) introduce ChatDB, which takes databaso structure as input for a data record management lask. This requires prior knowledge of the task-specific database schema. (e.g., a sales table), which must be defined and provided as a prompt to the model. In contrast, our method designed for generic language modeling, making it adaptable to a variety of tasks without extensive prompt engincering. Knowledge graphs are another structured format that can be interpreted memory of triples, similar to memory. However, while there is work extracting triples
2

---

Published in Transactions on Machine Learning Research (04/2025)
from text into knowledge graphs (Zhang & Soh, 2024) using LLMs and on LLMs "intcractively" querying knowledge graphs to answer questions (Back et al. 2023). our approach is distinguished by teaching the LLM memory-write and memory-read functionality through finetuming, This makes memory an integral component of the modcl's text processing and at the same time enables the model to fiexibly handle new and updated knowledge. Similar integrated systems have also been considered the knowledge graph literature (e.g.: AutoKG (Zhu et al., 2024)), ) but to the best of our knowledge, our system is the first that goes beyond conceptual proposal and demonstrates empirical success in both language modeling and knowledge editing, Additionally, our published training dataset can be used to endow any trainable language model with explicit memory without requiring architectural changes.
Memory as a state. The term memory can refer to recurrent architectures that represent past context 2014). Transformer-basee models use similar mechanisms with memory tokens (to transfer context across segments) and memory pools (to share informa- tion across multiple contexts) (Burtsev et al.
with vectors (Hochreiter & Schmidhuber 1997 Cho et al.
2022 Wang et al.) 2024b). While recent advances use veclor- or parameler-baser memory systems for long-range dependencies (arrtiss et al., 2022; Wu et al. 2022a b) Cheng et al. 2023 Wang al.
2020
Bulatov ct al.
2024). they are limited by memory vec- tor capacity (Jelassi et al. 2024 5. In contrast, MEMLLM has no such architectural limitations and fcatures explicit, interpretable and editable memory.
2023a He et al.
Knowledge editing. The goal of knowledge editing is to apply data- changes to a model's behavior for a set of edits while keeping other knowledge unaffected (Yao et al.) 2023c Gu et al. 2024 Zhang et al. 2024). Meta-learning and locate-then-ee then-edit are two classes of parametric methods that modify model weights. In metaalearning, hypernetwork is trained and applied to the model weights during test time (De Cao et al. 2021 Mitchell et al. 2021 In theeeet,,,,,,it, the weights triggered by knowledge expression are located and modified (Dai et al. 2022 Meng et al. 2022). There are also nemory-based methods that do not alter the original model weights but use an external memory (Gu et al., 2024). E.g., methods like SERAC, GRACE and DEFER use retrioval-based memory to fetch previously given edits and apply them to new inputs (Mitchell et al. 2022: Hartvigsen el al.] (024)). In WISE (Wang et al.) 2024a), in addition to the LLM, two additional parametric models are trained: a side memory and a routing network. Based on the query, the routing network decides which memory to use, the side memory or the main LLM. Evaluations show that multiple edits at time (batch editing) or successive edits (sequential editing) are challenging tasks - but certainly critical for the intended application of knowledge editing. While most methods can handle a few edits at time, their performance drops when applying more (Yao et al.) [023c] Wang et al., 2024a). Due to the explicit memory structure of MEMLLM, it can handle a large number of edits while maintaining performance.
3 Methodology
Our approach to endowing an LLM with an explicit memory is finetuning with the standard language modeling objective. We now present a finetuning regime that teaches the LLM (1) to extract knowledge from text and write it to the memory and (2) to read knowledge from the memory and leverage it for better language modeling. Following (Schick al. (2023) and Modarressi et al. (2023), we define an APT through which the LLM initiates memory writes and memory reads.
3.1 Memory structure
The memory stores information in relation triples. Each triple has the form = t,,,t, where e, is the first entity subject, e, the second entity or object, and t the relation. Example: (Washington D.C., capital of, United States). The entities and relations are stored each in scparate tables. As shown in Figure [ the facts are stored in the main table "Triple Memory" using three idintifiers linked to two other tables: one for subject and object entities, and one for relations. The entities and relations tables are indexed by unique names. We enforce uniqueness across the three linked iden- tifiers in the main table "Triple Memory", i..,, a specific combination (Eatity_ Relation IDaiiio________________ Entity ID2) can only occur once in the table. Because these identifiers indirectly refer to the names of the entities or relations, different identifiers imply different entity or relation names, thereby preventing redundant storage
text and vectors,
3

---

Published in Transactions on Machine Learning Research (04/2025)
Relations
Triple Memory (Main)
Entities
PK
Relation_ID
PK
Triple_ID
PK
Entity_ID
Relation_ _nmee (str)
Subj_Entity..
Entity Baailyy__eeeeee (str)
IDX
Embedding (vector)
Relation_ID
IDX
Embedding (vector)
Obj_ En
Figure 1: MEMLLM memory schema. Each triple stored in the "Triple Memory" using its subject entity ID, relation ID and object entity ID, along with a designated triple ID. Entitics and relations are stored in separate tables, each containing their designated ID, name and corresponding vector embedding. Both tables are indexed based on their vector embeddings.
of identical facts. The vector representations (created with Contriever (Izacard al. 2022) abstract away from different surface forms of the same entity, e.g., "US" vs "USA". In the interest of brevity, use the symbols and for both the entity/relae /relation itself and for its vector.
Our query format for querying the memory is:
q {(eg, 19, +), ()))
where eg, eg, 1 are subject entity, object entity, and relation. indicates the position in the triple of the entity
querying for. These two templates give us sufficiently specific querics (as opposed to, c.g.: queries with two variables) that are likely to return useful entity information.
We want to retrieve triples from the memory that match the query. Given that the surface form of an entity (and also the relation) can vary (e.g., "US" vs "USA"), our match criterion is not exact match, but rather vector similarity. We refer to entities candidate entitics/relations.
that are similar to the query entity and the query relation as
For retrieval, we first determine set of candidate entities:
C {E'l cos(e", Te)
That is, all entities with an above-threshold similarity are considered candidate entities. Similarly, we determine a set of candidate relations:
T {t'l (ts, 2 Te)
If the query query for object, i.c., q {cg, 19, *), then retrieve the following final set & of entities from the memory:
(c.13(e,t,e.)) E M: T,55 cos(t, (t) 2 Tp}
where M is the memory. That is, we retrieve all triples with eatities/relationn from the candidate sets such that their average similarity to query subject and relation is above the threshold. Subject queries are handled analogously
3.2 Memory-APi and Inference
We now describe the API that specifies how the LLM initiates memory writes and memory reads.
Memory writes. We process the input sentences by See the example given in Figure 2a For sentence 8; the input MW to the LLM is formatted follows:
x = + ((USER_ST))++s; sT))+ (CUSER_END)>
where Sey are the preceding sentences and s; bracketed by tags to mark it as the focus sentence. The LLM's task is then extract all relations occurring in the focus sentence and generate write command
We discuss how we set the thresholds and other hyperparameters in Appendix

---

Published in Transactions on Machine Learning Research (04/2025)
Memory Write
Regalo Plù Grande" (English: "The Greatest Gift") song by Italian singer Tiziano Ferro. (USER__TT) song was written by Ferro for his fourth studio album, Alla Mia ELÖ.(USER__NN (MEM_ WRITE-~AAla Mia Btbxxpaforrer>>>zziann Ferro:ll Regalo Più
Grande>>part of>>Alla Mia Etä))

LLM Ousput
(a) For memory writes, the input is given in two parts. (i) The pretext provides context for the model (e.g., an- tecedents for pronouns). (ii) The focus sentence the span of text (bracketed by ((USER_ST)) and ((USER_ ENDE)) from which the model is tasked to extract all relations. The model calls the API starting with the ((MEM_WRITE- WRITE >> command followed by extracted relations. })
 closes the API call. each document, MEMLLM scans the sentences one by one.
Memory Read
ELM Ouppo
"Il Regalo Più Grande" (English: 'The Greatest Gift') by Italian singer Tiziano Ferro. The song written by Ferro for his
fourth studio album, ((MEM_ READI>>performer>>>> Tiziano Ferro:ll Regalo Grande>>part of}-> Alla Mia Etā)) Alla Mia Eta. The
track was released as the album 's second single
Memory Output
Newr menory Incoming-
Remove previuu cal text new new
Previousty generated text as loput
"Il Regalo Più Grande" (English: 'The Greatest Gift') is by Italian singer Tiziano Ferro. The song written by Ferro for his fourth studio album, Alla The track was released as the album 's second single ((MEM_ [IMEM_READ....
LLM Outpor
(b) The model decodes one token at a time, as in standard language modeling. It also trained to generate memory read commands points when they retrieve useful information. In the example, after decoding some tokens, the model generates a ({MEM_RES) READ command followed by queries. --> triggers execution of the queries. Returned results appended. The model then the retrieved results for decoding the posttext. Whenever, during further decoding, the model initiates a new memory read by emitting (L we remove the previous one because it is unlikely still be useful.
Figure 2: MemLLM inference with memory read and memory write
that stores them the memory:
y/ M|x (MEM_NRITE-->ee sed; ef
Context Sei is necessary to extract relations from the focus sentence, e,g., if the focus sentence refers to a previously introduced entity with pronoun. We finetune the LLM to only extract relations from the focus sentence (not from the preceding context); see $3.3 for details.
To extract all relations from a document and write them to the memory, we iterate over the sentences of a document one by one.
Memory reads. The LLM can at each point in time either cmit regular token initiate API MEM READ call by generating:
(OMEM_READ(
It then continues by generating subject or object queries as introduced above: q E (leg, (M, eg)) The syntax for the memory-read API call is:
(<MEM_ (MEE_REAA(ee") > 3/222eg2...))
The entity sets & are then retrieved from the memory 3.1 merged and appended to the API call:
.;. 62, ......))
The LLM then continues decoding. Figure gives an example. The LLM starts generating a sentence that refers to an album by the Italian singer Tiziano Ferro. It has learned that just before naming the album a good point which initiate memory read. Two queries are generated (including: "What the song "Il Regalo Più Grande" part of?"). One entity is returned by the memory ("--> Alla Mia Etå])") and written
5

---

Published in Transactions on Machine Learning Research (04/2025)
MemLLM Training Data
Mertoy-Wrrite Training Examples
Memoory-ritt Model
Wikipedia Relations
2
3
Documents
Relations
Memory- Read Training Examples
Memory
Figure 3: MEMLLM training data pipeline.
to the buffer. The LLM then generates the name of the correct album (*Alla Mia Età"). This example illustrates that our memory has the potential of reducing hallucinations because through the memory an explicit representation is available of the fact that. "Il Regalo Più Grande" is part of the album "Alla Mia Età".
We remove memory-read API calls from the context if they are no longer useful. This happens in three cases: (i) The returned set of entities is emply. (ii) The number of retrieved entities exceeds threshold Qeh (Quhr 30). Such large retrieval results are unlikely to be helpful. (iii) The model emits " "((", initiating memory-read API call.
For (ii)), our motivation for removing the API call is as follows. Omitting API verbiage preserves the text's natural flow and reduces the context to those parts of the input that are still informative for high-quality generation.
3.3 Finetuning the LLM
We now describe how we create the dataset for finetuning the model to generate memory-write and memory- read API calls. One innovation of our work is that we create these API training data from corpora annotated with entities and relations, including Re-DocRED (Tan et al., 2022 a Wikipedia corpus annotated in Wikidata format with named entity mentions and relations (see also $4.1 that we will use as an example below.
Memory- data. Figure 3 shows how we use Re-DocRED annotated relations. For each sentence 8;, we retrieve from Re- DocREDD all relation triples such that one entity has a full mention (i.e., not pronoun) in 8; and the other entity has full mention either in or in the pretext (Sxx). The memory-write training example consists of the context z; .WW and the memory-write command y[mM see /3.2land Figure 2a Since we want to teach the LLM to generate memory-write API calls, we compute the training loss on y(a MW) only.
The set of relation triples can be empty for sentence st. In that in M|n that contains no relations. This encourages the LLM not to generale spurious relations for such "empty" sentences.
generate memory-write command
Memory-read data. For effective memory reads, the LLM has to learn (i) to identify where to initiate a query to the memory, (ii) to generate queries that retrieve helpful information from the memory and (iii) make good use of the information that is returned by the memory. We now describe how we generate our training data with all three capabilities in mind.
Given Re-DocRED document d, we generate different training instance d' for each memory-read API call. To produce d', we scan d's annotated entity mentions from the beginning to the end of the document. For each entity mention étarget, we collect all relation triples in which il participates. Such triples are a good basis for memory-read API calls that when issued before étarget first appears - will help the LLM correctly generate étargtti this why we refer to Starget as the target entity. We keep only that subset of the triples in which the mention of the other entity eg that the triple refers (the query entity) has already occurred. (The LLM will in general not be able to generate query containing eg if e, has not yet

---

Published in Transactions on Machine Learning Research (04/2025)
occurred.) We also discard all triples that we previously encountered during our scan. (These are already known at this point, so there is little utility initiating query for them.) We then generate query for each remaining triple: either (eg, x) (Etarget = object) or (*, f, eq) (Erarget = subject). The memory-read API call for the queries generated for Cearget is placed immodiately precoding Starget. This will retriove Ctarget from the memory many cases and then make it easy for the LLM to correctly predict Crarget al. the next position.
Next we retrieve results for the query from the memory (see Figure D The memory we use here the one that is populated from Wikipedia by the trained memory-write model (as described earlier in this section and in Figure 2ah. The memory write model misses some relations and incorrectly identifies others, resulting in an imperfect memory. We intentionally use this imperfect memory because it aligns the training data with the ultimate inference conditions.
If the query returns& a large number of results from memory (more than Quh = 30), we discard it as unlikely to be helpful. (Soc Appendix B for details.) An example is the query (+, country, United States) where Wikidata defines the relation "counlry" as "sovereigu state that this item in". There are thousands of entities that satisfy this query. Such an unspecific result is not useful. Otherwise we add the queries and the query result to d'; see Figure 2b and $3.2
Finally, we add the rest of d to d' up to the next memory read (indicated by "((") or (if there isn't one) the entire remainder of d.
To summarize, each training example d' a concatenation of (i) the pretext, including the first two letters the query result from the "2, (3,-...) })" and (iv) the posttext. The posttext consists of the rest of the following text until the next memory read or (if there isn't one) the entire rest of d.
("(E") of the API call, (ii) the API call proper "MEM_READ" (es² >t?)), ) (iii) memory e2,
The loss is applied to the API call (ii) - this teaches the model to generate the correct API call. The loss is also applied to the posttext - this teaches the model (a) to make good use of the information provided in the query result for predicting entities and (b) to predict the next memory read (as indicated by " "(("). (iii) is not subject to the loss since the query results are generated by the memory, not by the LLM. For the training example d' that contains the very first " ({MEM_READC READ(" ( in the document (and only for this d'), the loss is also applied to the pretext because the LLM needs to learn where to generate this first " "((MEM_REE)) READ (". For more comprehensive overview of the memory-read data generation process, refer to Appendix C which outlines the detailed algorithm.
4 Experiments
4.1
Setup
To train and evaluate MEMLLM, we construct training and evaluation datasets as described in $3.3 We roquire datascts annotated with entities and relations. We use three such datascts. (i) Re-DocRED Tan et al. 2022]: Wikipedia texts annotated (in a Wikidata format) with named entity mentions, coreference information and 96 relations (occurring intra- and inter-sentence). Re-DocRFD includes many relation instances missing in DocRED Yao et al. (2019). (ii) DocRED's distant supervised training set. It includes >10K documents but fewer relations per document. The size of this dataset makes the training more effective and robust. (iii) set of "counterfactual" variations of Re-DocRED (Modarressi et al., 2024). Modarressi et al. (2024) introduces an entity replacement strategy to find and apply suitable replacements over Re-DoeRED. In our initial tests, we found that teaching the model to produce counterfactual answers (which often contradict its parametric memory) increases robustness against pretrained knowledge bias. This is described in detail in 14.1.1
We finetune two Mistral-7B (Jiang el al. 2023) models using LoRA (uu et al. 2022 a memory-wrile model and a memory-read model. See Appendix D for details on finetuning and hyperparameters. Our baselines are the original Mistral-7B and the memory-read model with its memory capabilities disabled. The latter bascline lets us ascertain what extent improvements are due to in-domain finetuning (as opposed to the memory).
7

---

Published in Transactions on Machine Learning Research (04/2025)
Filtering Approach
Prec.
Rec.
F1
Acc.
Bascline
0.58
0.83
0.68
0.61
Justification
0.56
0.82
0.66
0.59
Reasoning
0.78
0.84
0.80
0.80
Table 1: Comparing performance of different prompting strategies for filtering distant supervision data. The reasoning approach similar to chain-of-thought prompting performs best among the strategies.
4.1.1 Filtering Distant Supervision Relations
Re-DocRED is human-an comaan-nnnotted and mostly consists of relations with explicit evidence. In contrast, the distant supervised DocRED training set lacks explicit evidence and contains many false positives due to its automated annotation method. To address this, we implement few-shot-based filtering approach to remove false-positive relations. We also apply this filtering to Re-DocRED relations that lack explicit evidence.
To increase the number of training examples, we also include examples from the distant supervision subset of DocRED, Distant supervision (Mintz et al. 2009) assumes that relation r exists between two entities (es, Co) in text if the text. includes both entities and the r = (es, f, co) relation triple exists a knowledge base. While this method is valuable for relation extraction, may introduce noisy examples without any evidence of the relation in the text. This noise could adversely affect our training pipeline.
The experimental setup is follows: We start with partial document (S {s1, 8 8:)) mentioning two entities (E1: 62), with at least one of them present in the last sentence (i.e., the focus sentence), 8;. Our aim is to determine whether the potential relation r between and has any evidence in the last sentence.
To filter out negative examples, we use large language models (i.e., Mixtral). We design 8-shot in-context learning examples to detect if there evidence of relation the focus sentence. We curate a test set to evaluate the performance of this filtering mechanism as follows. We select 1000 examples from the human- annotated split of DocRED as positive examples where the focus sentence is annotated evidence. For negative examples, we choose 1000 examples where the focus sentence contains at least entity but there is no evidence for the relation in the focus sentence.
For prompting, we apply three different strategies. In the first approach (baseline), we expect the LLM to answer with "Yes" or "No" to report whether the focus sentence contains evidence. With the second approach (justification), we expect the LLM to provide justification after giving the answer. In the final approach (reasoning), we expect the LLM generate a natural sentence representing the relation, then provide reasoning, and finally generate the answer with "Yes" or "No" in the last sentence, similar to chain- of-thought prompting.
We present the results in Table 1 These results suggest that the reasoning approach outperforms the other two approaches by a large margin. Also, it suggests that the filtering would lead to higher quality based on the high recall score, 0.84. We demonstrate the best-performing prompt in Appendix [E]
After applying this method, we use the filtered distant dataset alongside 10 counterfactual variations of Re-DocRED to generate data for the initial fine-tuning phase. We then continue the finetuning process with data generated from the supervised set of Re-DocRED.
4.2 Perplexity evaluation
To evaluate how the memory component. would improve language modeling, we perform perplexity evalu- ation. For this, we need corpus to extract facts, do memory writes, and store them in structured memory. Our primary source is full dump of English Wikipedia but we also evaluate using Wikipedia abstracts and Re- DocRED texts for further analyses. The language modeling evaluation then examines how the model performs in terms of perplexity once the memory component has been filled. Following Liu et al. (2022) we report: (1) OVERALL PPL (PPL on the entire input text), (2) TARGET PPL (PPL on the target enti-
Dump date: 2023-11 11-01, available http:://wwhtgggee....////aaaaaaa

---

Published in Transactions on Machine Learning Research (04/2025)
PPL TARGET ENTITY
Memory
OVERALL
Baseline #1 (Mistral-7b)
5.823
3.550
4.666
(no memory)
Baseline #2 (Memory Disabled)
4.997
3.510
4.353
MemLLM
MW[ (Fall)
4.905
2.986
4.187
MEMLLM
MW Wikipediaa (Abs.))
4.898
2.955
4.170
MEMLLM
4.863
2.821
4.102
MWRRe-DoRRED (Test))
4
Gold MR Pos. & Queries
4.634
1.938
3.548
MEMLLM
4.811
2.596
3.993
Gold MR Position
4.674
2.232
3.728
Gold Queries
Re-DocRED (Test)
4.431
1.364
3.149
Gold Target
4.426
1.357
3.142
Gold Target Only
4.385
1.194
3.026
Table 2: MEMLLM performance OVERALL PPL (all text), TARGET PPL (target entities) and ENTITY PPL (all entities). We show the effect of memory content ("Memory")) "MW[X]": the memory is populated with triples generated by wemory-writes with MEMLLM run on X. 5 0-9) the triples are from Re-DocRED (Test), Re-DocRED's validation set.
ties) and (3) ENTITY PPI. (PPL all named entities). The model produces token wy with probability (e))w))::
p(w(lwe)) = p(w; Wei, MR)p(MRR))) wez))
+ p (w - PMMR|
where ((MR[we)) is the probability of initiating memory read (MR) with the "({" token
In case of MR, wy is conditioned both MR (including the MR call and the returned result, see Figure 2b and the pretext Wes. If there is no MR, then Wy is only conditioned on wes.
Table 2 gives perplexity results on Re- Do-RDD test. MEMLLM outperforms the two baselines on all three PPL measures (①). This increase for triples appearing for the first time in the text suggests that memory- reads successfully recall relevant information for language modeling. This improvement benefits not just entities the text (ENTITY PPL) but the entire text (OVERALL PPL). TARGET PPL. (the focus of this work) improves by .524 (2.986 3.510). This substantial improvement demonstrates the effective- ness of MEMLLM for target entities. This capability is crucial for generating factual text and preventing hallucinations.
For our memory-read analysis, instead of using the LLM to write to the memory, we directly populate the memory with the relations from the validation set (indicated as "Re-DoeRED (Test)" in column "Memory"). This lets us investigate what would happen if the memory-write process were error-free, i.e., all remaining errors are due to the memory-read process. We look at four potential sources of crror memory reads and present in each case an ablation in which this source of error is eliminated: the position of the memory read is the gold position immediately before the target entity (6 "Gold MR Position"), the query to the memory is the gold query (7 "Gold Queries"), the correct target entity is returned by the memory "Gold Target"), the correct target entity is returned by the memory and no other entities (9 "Gold Target Only"). (g) is the lower bound perplexity for perfect memory reads (and perfect memory writes).
We evaluate p(w,|wed by setting p( [we, MR) to zero for all positions except for positions where memory reads actually The reason is that laking into account MR call at each position results tree with 2 leaves at position in the expensive compute. As result, evaluate with smaller values of p(w/wwe than the true p(w; wex) estimated by the model and, consequently, with higher perplexitics, thus unfairly penalizing MEMLLM. Note that this is problem for fairness of our perplexity evaluation, but for application (where we only pursue single path each point).
text, each requiring memory call. This
9

---

Published in Transactions on Machine Learning Research (04/2025)
Method
REL
GEN
LOC
AVG
DEFER
0.02
0.02
0.67
0.24
GRACE
1.00
0.02
1.00
0.67
WISE
0.70
0.67
1.00
0.79
MemLLM
0.78
0.76
0.97
0.84
Table 3: Knowledge editing results on ZsRE with 1000 sequential edits. AVG: mean of reliability (REL), generalization (GEN), locality (LOC).. Baseline results (using the same model, Mistral-7B, and edit set) are from Wang et al. 2024a). Bold (italics): (second) best result.
Comparing 9 and ( on TARGET PPL (1.194 vs 1.357) shows the effect of "ambiguity". The +.343 gap is due to the memory returning more targets than just the gold target.
Moving from 8 to ① (1.357 to 1.364) indicates the impact of the memory retrieval process. In 7 we use the gold queries, but without ensuring the inclusion of the gold target the results. The next comparison highlights the impact observed when the LLM itself generates queries (G) vs when only gold queries are issued (7). Finally, the effect of the model itself selecting the position for the memory read (6) versus
predetermining that position ( is shown in the rise from 2.232 to 2.596.
To isolate the factor memory-write performance, we fix (i) gold memory-read positions and queries and (ii) the input corpus for extracting relations (we use Re-DocRED test). We only vary the method by which the memory populated: running the memory-write model on the input corpus (④) vs reading out the relations from the gold data and directly storing them in memory (7). As expected, PPL improves when directly stored (i.e., 100% precision and recall) triples used (7) when MEMLLM extracts and writes triples to memory (+). This indicates that there is room for improvement by training MEMLLM to do a better job at information extraction.
Scaling the stored triples. In a real-world scenario, the size of the memory and, consequently, the size of query results will get large. This increases the risk of unhelpful information being returned from the memory. To investigate this, we compare our main experiment (Q) using the full Wikipedia, 111M triples) with two ablations that use only Wikipedia abstracts (2) 38M triples) and only Re-DocRED test (3). Table 2 shows that there is relatively small negative effect of memory size: 3 (memory stripped down to the relations generated from Re- Re-DoD test) is only slightly better than ! (full memory). This suggests good scaling properties of our approach.
Memory redundancy reduction benefits. In- stead of storing facts in structured triple format, each fact could be stored as own proposition
100
(Chen al. 2024). For instance, one can store the triple (Washington D.C., capital of, United States)
Memory Usage Relative Fact-Based Storage (%)
80
as the proposition: "Washington D.C. is the cap-
ital of the United States." Based on the memory
60
populated with full Wikipedia (111M triples), an
equivalent RAG system would need to store and in-
40
dex 111M proposition sentences. In a vector-based (dense) retrieval setting, this means storing and in- dexing 111M vectors alongside their text counter- parts. However, in MEMLLM, store these facts in structured triples with identifiers (cf. Section 3.1. and the only veclor-based indices are entity and relation embeddings. After storing the 111M triples, the entity/rclattin tables contain roughly 21M unique entity/relation records. Thus, the number of extracted facts increases, our structured
20
0
102
10a
104
10a
10
107
10a
No. Extracted Facts (Triplets)
Figure 4: Memory efficiency of storing structured triples vs. proposition-based storage. The y-axis rep- resents the fraction memory required compared to a RAG system that stores an embedding per fact.
10

---

Published in Transactions on Machine Learning Research (04/2025)
triple approach results in significantly lower mem- ory usage and, at large scales, it requires less than 20% of the memory (21M) needed for direct proposition storage (111M), demonstrating its efficiency in roducing rodundancy (Figure 4). Morcover, having only entitics and relations encoded as vectors reduces ambiguity and improves recall by introducing less noise compared to encoding entire sentences.
4.3 Knowledge Editing Evaluation
To test whether MEMLLM facilitates knowledge editing, we evaluate prompt-based knowledge editing. Fol- lowing Hartvigsen et al. (2023) and Yao et al. (2023b), we measure reliability (REL), generalization (GEN) and locality (LOC). Each example includes prompt, edit, generalization test prompt and locality test prompt. The task is to apply the edit on the original prompt to the model. The goal is for the model to respond to original and generalization test prompts in accordance with the edit. The locality test checks whether unrelated knowledge is affected. An ideal method effectively applies edits, generalizes correctly and does not harm unrelated knowledge.
Following Wang et al. (0224a), evaluate MEMLLM on ZsRE, closed-book question answering dataset (Levy et al., 2017) with locality prompts selected from Natural Questions (NQ) (Kwiatkowski et al. 2019 We apply 1000 edits from the evaluation set by appending them to the end of the questions (the prompts) using the following text: "It is or they are" and bracketing them with tags, Example: " "((USER_ ST}) W What city was Luca Verdecchia born? It is or they are Naples ( ((USER END})." The memory-write model should then extract and store Verdecchia's 's place of birth, i.e.; Naples. We evaluate MEMLLM using a 5-shot QA prompt. The first four examples are typical question- answer pairs. The fifth in addition includes a full memory-read call. A prompt generalization or locality test prompt is appended to the 5 shots and a memory-read API call executed after the question mark.
We expect the model to answer the questions based on the memory filled with the edits. Some edits in the dataset overlap or are intended to replace previous edits. Therefore, if a newly extracted triple has an exact matching entity and relation with an old triple, we replace the old one with the new one.
Table 3 compares knowledge editing results for MEMLLM with three basclines. MEMLLM outperforms the baselines (AVG of .88). High reliability (.78) and generalization (.76) scores suggest that MEMLLM (i) manages to extract and store the relation triple based on the edit and (ii) utilizes the edit in the memory-read process to answer the original and the generalization test questions correctly. Moreover, since MEMLLM uses an explicit memory the applied edits only mildly affect the answers to unrelated questions: MEMLLM has a score of .97 on locality. This indicates that there is little cumulative deterioration of the explicit memory.
Qualitative Analysis. Leveraging MEMLLM's interpretable design, we identified the causes behind the 22% performance gap in reliability from 1.00 to 0.78 (REL, MEMLLM vs GRACE). Out of 216 errors, 45 were due to memory writes resulting in no triples or triples without the desired edit. In 95 cases, the edit was captured in the memory write but not retrieved by the memory read, either due to a bad query or incorrect relation extraction during the memory write. Another 63 errors occurred when the model did not effectively use the edits even though they were correctly retrieved. Many of these errors are due the limitation to 96 relations (see $ For example, the question "How endangered does the IUCN consider Hyloxalus parcus?" involves relation that is not covered: "IUCN conservation status". In another case, the question "What family lineage was Xiao Jia part of?" retrieves the correct edit ("Southern Ming Dynasty") but for an incorrect relation: (<MEM_RR) REE__EAAD (Xiao Jiaæcountry of citizenships))) as the relation "family" is not one of the covered 96. The model may then not recognize the query result as relevant to the question and ignore it. Addressing this limitation by supporting more relations would resolve many of these errors. Even with this limitation and not being specifically designed for knowledge editing, MEMLLM outperforms other model editing methods. We believe that expanding its capability to handle a broader range of relations would greatly enhance its performance.
11

---

Published in Transactions on Machine Learning Research (04/2025)
5 Conclusion
We present MEMLLM, a novel approach to endowing an LLM with an explicit structured memory. We publish training dataset that can be used to extend any standard LLM with such memory. We show that MEML.LM improves language modeling (as measured by entropy) and outperforms state-of-hhe-art knowledge editing methods on ZsRE.
Limitations
While the structured relation-based memory improves factuality and interpretability, it has its own lim- itations. The current version of MEMLLM handles only 96 relation types commonly used in Wikidata. However, to handle all types of knowledge extraction and storage, the model should be capable of extracting other types of relations. Composite relations that could be inferred from multiple already extracted rela- tions are not detected utilized in the current version of MEMLLM. For instance, if we extract (California, country, United States) and (Apple Inc., located in, California), we expect the relation (Apple Inc., located in (or Country), United States) to be inferred. MEMLLM is not a memory-aware solution. This means if a fact is not stored in the memory, but the decoding process generates a partial prompt that requires that lact, the model would either continue generation based on its parametric knowledge or hallucinate.
We acknowledge that Retrieval- Augmeeened Generation (RAG) methods are widely used for more grounded text generation. However, like other basclines in knowledge oditing, do not include RAG methods in comparisons due to the difficulty of modifying facts in their unstructured format. Updating single fact would require locating and altering every related text snippet and embedding within the RAG's knowledge base, which is highly impractical. MemLLM's memory is not pre-populated as a KB; it stores edited facts through a memory- write process, similar to non- and semi- camimparamctric editing methods. While methods like ChatDB (Hu et al.] 2023) also employ non-parametric structured memory, they rely on database tables that are more suitable for analytical tasks. This setup requires the user define task-specific database schema in advance and explicitly prompt the model with it. In contrast, McmLLM uses much more generalized memory structure that supports both language modeling and QA tasks (as discussed in the knowledge editing section) without requiring any changes to its approach or memory format. Furthermore, the nature of prompt-based approaches compared to fine-tuning makes them inherently less faithful and more dependent on the specific content stored in memory.
We refer all these limitations to future work, as this paper we have laid the initial groundwork for building more complex and comprehensive method.
References
Jinheon Back, Alham Fikri Aji, and Amir Saffari. Knowledge-augmente, language model prompting for zero-shot knowledge graph question answering. Proceedings of the Pirst Workshop on Matching From Unstructured and Structured Dato (MATCHING 2023), 2023. URL http:://pp..semaaticcchmi sewantisccool.rl.r
htgg/oorpuID::20063222
Aydar Bulatov, Yuri Kuratov, and Mikhail Burtsev. Recurrent memory transformer. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htttts://opeerevieeee
Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. anXiv preprint arXiv:2006. 1 11527, 2020.
Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading, arXiv preprint or arXiv:2310.0500 2023.
Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense X retrieval: What retrieval gramularity should we use? In Yaser Al-Onaizan, Mohit Bansal, and
12

---

Published in Transactions on Machine Learning Research (04/2025)
Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Longuage Pro- cessing, pp. 15159-15177, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.18653. 0000553/v//// /1///////2222/ http//22.emmmpmmain.n. m4in.845. URL https://acclaattoooog.oooo
emmlppp2emmmmppmmmmmmmmmmmmmppp wwt.maainn..... 845
Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. Lift yourself up: Retrieval- auginented text generation with self-memory. In shirth- Conference Neural Information Pro- cessing Systems, 2023. URL https://openreviies wwwp/eorummidd-loeaa.S.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcchre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder deeoderr for statis- tical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724 1734, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-111 URL
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvasshchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Tsard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeonlaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyzz, Erica Morcira, Rewon Child, Oleksandr Polozov, Kathorine Lee, Zongwei Zhou, Xuezhi Wang, Bremnan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Fec, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. Journal of Machine Learning Reseorch, 24(240):1 113, 2023. URL
1144.htm1,
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493 8502, Dublin, Treland, May 2022. Association for Computational Linguisties. doi: 10.18653/v1/ 2022. ac22.acl-oogg.58 581. URL htttp:://aclanthologg hwwphhhhoolgy..g/2222222222 202/ wwl0022.acl11oo. acl lon.-lloog... 581
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 6491-6506, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistios. doi: 10. 10.18653/v1/2000 emnlp-main.522 URL https://aclantholll
org/2021.......... emnlp cammmlpmmain-6550 m2in.522
Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing can hurt general abilities of large language models. arXie preprint arXiv:2401.04 2024.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mling-Wet Chang. Realm: retrieval-augmmeted language model pro-training. In Proceedings of the 37th International Conference on Machine Learning, ICML/20. JMLR.org, 2020.
Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzych Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems, 2023.
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Newral Information Processing Systems, 36, 2024.
Zexue Hc, Leonid Karlinsky, Donghyun Kim, Julian McAulcy, Dmitry Krotov, and Rogerio Feris. Camelot: Towards large language models with training-fre free consolidated associative memory. arXiv preprint arXiv:2402.1344 2024.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 98)):1735- nov 1997. ISSN 0899-7667. doi: 10.1162/neco.19000 w9t.//.......9..... 999..9.8.1735. URL /ttpp::://dii.r//11/../ 1162/neco.
9.7.9...1735
Chenxu Hu, Jie Fu; Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. Chatdb: Augmenting 1lms with databases their symbolic memory. arXiv preprint arXiv:2306.0390 2023.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:// http://mwwweeeeeeeooorrorro/aaaa not./forrmm?dddn/eeeeeee.....
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong. and Zhang Xiong. Transformer patcher: One mistake worth one neuron. In The Eleventh International Conference on Learning Representations, 2023. URL
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retricval with contrastive learning. Transactions on Ma- chine Learning Research, 2022. ISSN 2835-8856. URL httpp::/openreevier. http:////ppppeeeeeeeeeeee/ar/aaa oetf
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. TemporalWiki: lifelong benchmark for training and evaluating ever-evolving language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6237-6250, Abu Dhabi, United Arab Emirates, December 2022. Association for Compulational Linguistics. doi: 10. 20.18633/v1120220 20t./2202..mmmmmii.... 418. URL main. 418
attps://sclarthooogy..rgg. wwg.022...02202220..
Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. arXiv preprint 0rXiv:2402.01032, 2024.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(1))1133 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprinl arXio:2310.06822 2023.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, 1CML'22 '23. JMLR.org, 2023.
Jungo Kasai, Keisuke Sakaguchi, yoichi takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yn, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. Realtime QA: What's the answer right now? In Thirty-seventh Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openrevvier met./Eorum..d--HKIIPCC
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfild, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstoin, Illia Polosukhin, Jacob Devlin, Kenton Loe, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural ques- tions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019. doi: 10.1162/tacl a 00276. URL http::/s///ccaaaaoooooooor//////
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading com- prehension. In Roger Levy and Lucia Specia (cds.), Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 333 383-342, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 10. 18118863/v 118633/11/K171110 K17411044.
URL http::://aaaaaatoooooooor//////t
14

---

Published in Transactions on Machine Learning Research (04/2025)
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- augmented generation for knowledge-intensive ulp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9459- 9474. Curran Associales, Inc., 2020. URL
wtwp:ig.....rrri..cc...co.oooc pdf
fille/paper/
Xinnian Liang, Bing Wang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. Enhancing large language model with self-controlled memory framework. arXiv preprint or OrXxv:2304,133 2023.
Qi Liu, Dani Yogatama, and Phil Blunsom. Relational memory-augmente language models. Transactions of the Association for Computational Linguistics, 10:555 572, 2022. doi: 10.1162/taci a 00476. URL
https://aclantholl.. hwwpr/hhooogooogg/22220222 org/2022. tacl-1.32
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non- con-paraettric memories. Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802 9822, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: https://aclanthhlloe
acl-222...--oo..555 546. URL
2xgg/2023 acl-1ong.111on..... 546.
Pedro Henrique Martins, Zita Marinho, and Andre Martins. oc-former: Infinite memory transformer. In
Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting the Association for Computational Lònguistics (Volume 1: Long Papers), pp. 5468 5485, Dublin, Ireland, may 2022. Association for Computational Linguisties. doi: 10. (66653/v/20222acllloo //aclanthology.oo wwwp://...rr.002....oom. wwtgg2222.acl-oorr. acwallng.......... 375
URL https:
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906-1919, Online, July 2020. Association for Computational Linguistics. doi: 10. /..16633/1122020acclmmmm httpp://aclanthoooggg.ooo org. acl acl-main.1... main. 173.
URL
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359- 17372, 2022.
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. Distant supervision for relation extraction without labeled data. In Keh- Yih Su, Jian Su, Janyce Wiebe, and Haizhou Li (eds.), Proceedings of the Joint Conference of the 4 7th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 1003-1011, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15817-15831. PMLR, 17-23 Jul 2022. URL html
https://proceeding mlr. pttpss//122/mttchhl11111111
Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schütze. Ret-llm: Towards a general read-write memory for large language models. arXiv preprint arxiv:2305.1 142322 2023.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Ali Modarressi, Abdullatif Köksal, and Hinrich Schuetze. Consistent document-level relation extraction via counterfactuals. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Asso- ciation for Computational Linguistics: EMNLP 2024, pp. ber 2024. Association for Computational Linguistics. doi: /111863//112024..20 findings-cmmlp.672 URL https://acllanthol.gg www.olloooyooog/2222/ org/2024. findings--ennlp.cc 672
Miami, Florida, USA, Novem-
Charles Packer, Vivian Pang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. Memgpt: Towards Ilms as operating systems. arXiv preprint arXiv:2810.08500 08560, 2023.
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. Tn In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23), UIST '23, New York, NY, USA, 2023. Association for Computing Machinery.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418 5426, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2000 10/55/3//12222mmmm///. wwnlp-main URL https://aclantthooogg org/ 2020. emnlp main. 437
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023. URL
https://
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Fd H Chi, Nathanael Schärli, and Denny Zhou, Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 31210-31227. PMLR, 2023.
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In International Conference on Learning Representations, 2020. URL https://openreevie
Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng. and Sharifah Mahani Aljunied. Revisiting DocRED ad- dressing the false negative problem in relation extraction. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1472-8487, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10.18653/vv /v1//vv1/// 11/2022. 212/22222.mmmppppp www.mmmmin.n..... m80n URL
or2/ wwweeennaaaaeeeeeeeee maa-mmin.... 580
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint Xiv: 14768, 2024a.
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. In Teirth-seventh Conference on Neural Information Processing Systems, 2023a. URL /tttps://openreiier wwwp:nttfooommidderytttte
Yu Wang, Xiusi Chen, Jingbo Shang, and Julian McAuley. Memoryllm: Towards self-updatable large language models. arXiv preprint arXiv:2402.0400
2024b.
Zekun Wang, Ge Zhang, Kexin Yang, Ning Shi, Wangchunshu Zhou, Shaochun Hao, Guangzheng Xiong, Yizhi Li, Mong Yuan Sim, Xinying Chen, et al. Interactive natural language processing. arXiv preprint arXiv:2305.1 13246, 2023b.
Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: A cemmor-aaugmnteed transformer for sequence modeling. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (ods.), Findings of the Association for Computational Lönguistics: AACI-IJCNIP 2022, pp. 308 318, Online only, November 2022a. Association for Computational Linguistics. URL https: wwwpanthwwwooooooo/o/eee findings--aaaee aacl.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022b. URL htttps://openreviev. http://ppenrrevie..eeeffrtrrrr net/forum? id=IrjbxzRcnff
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting in language models. In The Eleventh International Conference Learning Represendations, 2023a. URL httpp:://opeerreiee..etttt
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. DocRED: A large-scale document-level relation extraction dataset. In Anna Korho- nen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Lingwisties, pp. 764-777, Florence, Italy, July 2019. Association for Computational URL
Linguistics. doi: (0.11853/v1/119-1 1074.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportumities. CoRR, abs/ 22002 2023b. doi: 10. 485500/arXiv htt.0/aaa........111111 1317.. URL https:://do...oo
1w..:0.4555//arr... 2305. 13172
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- gunge Processing, pp. tics. doi: 10.18653/v1 1118653331/22233 http///..222.////p1pp... main. 632. URL
Singapore, December 2023c. Association for Computational Linguis-
wwwee-mnaaaaeeeeeee 632
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. In The Eleventh International Conference on Learning Regresentations, 2023. URL https::/openrevveer
net met/forummid=aee Bi-
Bowen Zhang and Harold Soh. Extract, define, canonicalize: An Ilm-based framework for knowledge graph construction. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 9820-988 9836. Association for Computational Linguistics, 2024. URL https: //aclanthology.o or2/2024. emnlp-main.50 5488
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint (rXix:2401.01280 2024.
Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation of (arbitrarily) long text. anXiv preprint ar Xiv:2300.. 13304, 2023.
Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, and Ningyu Zhang. Lims for knowledge graph construction and reasoning: recent capabilities and future URL
opportunitics. World Wide Web (WWW), 27(5)558, 2024. doi:
A Memory-write Decoding Method
While one might use MEMLLM with greedy decoding for memory writes, we suggest that the finetuned model may end the memory-write too carly, before completely extracting all relations. Therefore, to ensure the model captures all relevant relations, we implement a late stopping strategy. In this approach, similar to greedy decoding, we consistently select the top-scoring token as the next token, unless it's the closing token ')3'. If the closing token scores highest, we note its position, calculate the average log probability score of the sequence up to that point, and proceed with the second highest scoring token aiden--ypiccllyy the ":" separator- resuming greedy decoding. By tracking the positions where the closing token was predicted, along with their
17

---

Published in Transactions on Machine Learning Research (04/2025)
Query (q)
Relation type (19)
country of citizenship, country, country of origin, religion, place of birth, place of death, work location, location, basin country, residence, location of formation, publication date, production company, platform, original language work, applics to jurisdiction, located the administrative territorial entily, headquarters location, inception, employer, date of birth, date of death, educated
(x,t9,c))
(eg, 19, ()
contains administrative territorial entity
Table 4: List of ambiguous queries subject to the filtering process.
corresponding logprob scores, we maintain the generation process until there enhancements in the scores for K=5 consecutive times. Subsequently, we halt the generation and select the position with the highest score as the cutoff point.
B Filtering Ambiguous Queries
As we aim to assist the model with the stored memory content, having concise query results would facilitate reaching this objective. Getting precise outputs from the memory would require queries that are tailored in way which lead to an exact match or related entities to the target entity. To reduce the chances of getting vast and wide-range amount of outputs from the memory, exclude queries that potentially leads such results. In Table 4 we demonstrate query patterns that intuitively assume based on the queried entity and the relation type that would lead to an ambignous result. Therefore, we drop any query that would match with one of the mentioned patterns.
C Memory-read Data Generation
Algorithm [ presents the pseudocode for the process of generating MEMLLM 's memory-read training data. See Section 3.3 for a detailed description of the same process.
Hyperparameters Details
We finetune MEMLLMM, with a Mistral-7B-v0.1 model (Jiang et al. 2023) using an Adam optimizer (Kingma & Bal 2015), with the learning rate set to 2 10-5, 2 epochs, and a batch size of 96. For LoRA specific parameters, apply dropout rate of 0.1, with rank of 16 and alpha weight of 8.
We opted to set Qthr to 30 based on the distribution of triples observed in the Re-DocRED dataset. During the construction of memory-write data, we found that the 95th percentile of sentences contained maximum of approximately 30 triples. This value serves as an upper threshold: the number of entities exceeds 30, it is likely to surpass the typical number of triples within a sentence. Consequently, higher count. of entities could indicate that many of them are unrelated the sentence's factual content, reducing their overall informativeness.
To select the memory retrieval hyperparameters ($3.1) must balance explicitness with the need to accommodate variations in entity mentions and relation types. This balance is influenced by the data and the entities involved, but generally, a larger Te increases explicitness. However, il also limits the number of similarly mentioned entities that can be retrieved, which depends on the use case. smaller Te could retrieve more entities, but would also increase query exccution time. The selection of r, depends on the supported relation types and the required flexibility in retrieving closely related relation types. For instance, in model editing, where handling loosely similar relation types is necessary, more relaxed 7, value is appropriate.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm Memory-read Data Generation
Input:
D): Re-DocRED documents (d E D)
Output:
DMR: Memory-read training data (d' E (D)
Auxiliary functions:
TRIPLES(erarget d) Returns all triples in document d containing the entity Starget as the subject or object:
TRiPLES( d) = ((target, f, eq) df U {(eq. i, fearget) E d} QUERYRESULTCE M): Returns the set of entities retrieved by query over memory M (e.g. Wikipedia facts extracted using memory writes).
Definitions:
positionldx: The position of an entity (or word) within the document text.
1: Initialize Trip'rr + (). SeenEntities + (}. DMR +
2: for d in D do
3: prevReadPoss
Da []]
4: for Otarget. in d, in order of appearance in text do
5: q (}. R
(R is the query-results dictionary for the queries in
6:
for triple TRirnes(ettrret d) do
7: if triple & SeenTriples then
Let ey the other entity triple
9:
if cq C SecnEntitics then
ạ + triple - étarget
11:
9 $ q then
12:
Rg t QUERYRESOUTTQ, M)
13:
if [Ryl S Qur then
14:
Add to q, R/g] - Rq
15:
end if
16:
end if end if
17:
18:
Add triple in Seen'Triples end if end for if then
19:
currentReadPos Ctarget positionlls
d' prreteext +- d.text (currentdurre d' ...uuriie () d' .results <
q ascending order by [R[g]]
for q in the first 3 elements of q do
Add q to quueries
29:
d' ressuuts + d' results R(g]
30:
end for if results = then d' d'-results + {etarget
31:
32:
33:
end If if Daal > then
34:
35:
Da posteett ddeetppeeteeelPPccuureeeedd
36:
PrevReadPos + currentReadPos
37:
end if end if
Add etarget ScenEntities end for
Da[-1] plsttee- (creetcc
DVIR + DMR Da
(ll denotes concatenation)
end
44: return DMR
Finally, T, determines the final number of outputs retrieved during the memory-read. A larger T, makes the memory more explicit in both entity and relation type. We set Ta and 7, to 0.7 and r, to 0.85. We set these values to Te 0.85, F 0.2 and T 0.6 respectively for model editing experiments.
19

---

Published in Transactions on Machine Learning Research (04/2025)
E
Filtering Prompt
In Figure 5 we demonstrate the best-performing prompt in our filtering process over the distant supervised subset of DocRED.
20

---

Published in Transactions on Machine Learning Research (04/2025)
Reasoning Prompt- Distant Supervision Filtering
To determing whether the main senfence contains information about the given relation. both the main sentence and the context will be provided. The goal is to identify whether there evidence of relation the main sentence, supported by the contest. If there relation or the evidence exisis solely the context without requiring the main scutence, respond with No. Otherwise respond with Yes. Provide reasoning to support your response.
Context:
Main Sentence: James Michael Osting bom April 7, 1977 ) is former Major League Bascball pitcher.
Relation: (' l'Osting", : "date of birth". "April 7, 1977*)
Evidence: The relation indicatos that Osting was born on April 7, 1977. The main sentenes explicitly mentions that Osting was born on April 7, 1977. The answer is Yes.
Context: Splashdown Hot Tuna album released in containing the tracks
previously unreleased live acoustic performance had been played on
the short - lived mdiio station WQIV in the mid-1970s During the recording, news of the Apollo Soyuz mission returiing to Farth after the first USA - USSR the astronauts 'adio transnissions were played at the same time Jorma and Jack continued with Police Dog Blues "The mansmissions mixed with the song were preserved for this release as the last track of side I
rendezvous space reached station.
Main Sentence: The album was Hot Tura's first release on Relix Records, and one the first Relix releases.
Relation: (" H'oot Tuna", "country of origin', "USA")
Evidence: The relation indicates that the origin of Hot Tuna the country of the United States, There is no evidence in the main sentence regarding the country of origin of Hot Tuna. The answer is No.
Context:
Main Sentence: The Chernung Canal Bank Building is located at 415 East Water Street Elmira, Chemung County, New York, United States Relation: (' ('Chemung County", "capital". "Elmira")
The relation indicates that Elmira is the capital of Chemung County. The main sentence only specifies the location of Elmira within Chemung County but does mention Elmira the capical of Cheaung County. The answer No.
Evidence:
Context: Carrie Lam Cheng Yuet ngor, GBM, GBS horn May 1957 the 4th and current Chief Executive of Hong Kong. Before that she was the Chief
Secretury for Administration, the most senior rank of principal officials of Hong Kong , from 2012 to 2017.
Main Sentenee: After graduatig from the University of Hong Kong, , Lam joined the civil service 1980 and served in various bureaux and departments. Relation: (" ("Lam,, "educated at". "University of Hong Kong")
Evidence: The relation indicates that Lam received education the University of Hong Kong. The main sentence mentions that Carrie Lam Cheng Yaer-ngor graduated from the University of Hong Kong. The answer is Yes.
Context: Pacific Fair is major shopping centre in Broadbeach Waters on the Gold Coast, Queensland, , Australia It was Queersland largest regional shopping centre until 2006 Pacific Fair was developed by Hooker Retail Developments tenants, Since then, , Pacific Fair has undergone numerous expansions and has grown to have more than 300 specially stores and four anchor tenants In January 2014, work began on major redevelopment project moet the predicted regional growth
opened in 1977 on what was swampland with specially stores and anchor
Gold Coast. Prior to redevclopment, four main major stores including four - level Myer, Kmart, , larget, , Coles and Toys 'R" Us... Daimaru operated in the centre before its Australian withdrawal, albcit briefly.
shopping centre had
Main Sentence: also had 12-screen Birch Carroll and Coyle Cinema re opened as Event Cinemas in late 2015), )- Relation: (" l'Evatl Cinemas", "country.. "Australia") The relation indicates that Event Cinemas is located in the country of Australia. The main sentence mentions that Event Cinemas is part of Pacific Fuir which is located Australia, The answer Yes.
Evidence:
Context: Benjamin Winslow Harris November 1823 February 1907) ) was nineteenth. century politician. lawyer and judge from Masstacussetts He was the father of Robert Orr Harris. Bom in East Bridgewater, Massachusetts- , Harris pursued an academic course at Phillips Academy Andover, graduating in 1847 graduated in East Bridgewater He served in the Massachusetts Sensle in 1857, was member of the Massachusetts House of Representaties in 1858, was district attorney for the sontheustern district of Massachusetts from 1858 1866 and was collector of internal revenue for the second disrrict Massachusetts Harris was elocted a Republican to the United States House of Representatives in 1872, serving from 1873 to 1883, not being candidate for renomination in 1882 to 1883, Afterwards, he resumed practicing law East Bridgewater, , Masstachssetts
Dane Law School Harvard University in 1849. was adminted to
Boston, Massachusetts in 1850, commencing practice
1873.
There, he served chairman of the Committee on Naval Affains from and was judge of probate for Plymouth County, Massachusetts from 1887 to 1906.
Main Sentence: Harri died East Bridgewater on February 7, 1907 and was interred in Central Cometery in East Bridgewater. Retation: (" H'Henjamii Winslow Harraa, place of birth", "East Bridgewater")
Evidence: The relation indicates that Benjamin Winslow Harris was bom in East Bridgewatier The main sentence lacks information about Benjamin Winslow Harri''s place birth. The evidence for East Bridgewater his birthplace is exclusively found the context, the main sensence. The answer No.
Context: Greatest Hats the first compilation album by the Canadian wave / synthpop group Men Without Hats, released in 1996.
Main Sentence: slightly modified version of the album was released in the US in 1996, , caiitled Colleetion. .
Relation: (" ('Collection", "performer'. "Men Without Hats")
Evidence: The relation indinates that Men Without Hats the performer of the Collection album, The main sentence says that Men Without Hars releated slightly modified version of the Greatest Hats album which the album Collection. The answer Yes.
Context: Anron Hobart June 1787- September 19 1858 was U.S. Representative from Massachusetts.. Hom Abington , Massachusetts Hobart pursued classical studies and gradusted from Brown University in 1805 He studied law, was admitted to the har and commenced practice in Abington He served member the Mastachusetts House Representatives and served the Massachusets State Senate. Hobart was elected Democratic Republicnn the Sixteenth Congress to fill the vacancy cansed by the resignation of Zabdiel Sampson. He was reelected as a Democratic- Republican to the Seventeenth Congress, Adams candidate the Nineteenth Congress, , and served from November 24, 1820, to March 3. 1827 He doelined to be a candidate for renomination in 1826.
elected Adams Clay Republican to the Fighteenth Congress, and reelected
Main Sentence: He then served
Executive councilor
1831 and served as probure judge 1843- 1858.
Relation: (" Aaroo Hobart", "dalc of death"" "1858")
Evidence: relation indicates Aaron Hobart passed away the year 1858. The sentence does not contain information about the given relation. ov iddence of Aaron Hebart's date of death in 1858 is solely preseal in the context and is not mentioned in the provided main seatinne. The answer is No.
Context: [[contex]]]
Main Sentence: [encess]ntenc]]]
Relation:
(' [[enttyl]]"" "[relltioo]]]' "[entity]]]")
Evidence:
The prompt for the distant supervision dataset filtering. This prompt includes the natural repre- sentation of the relation, the reasoning, and the final answer
Figure 5:
21


Reviews:
[{"review_id": "FNO8YcniMB", "reviewer": "Reviewer_KCAv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The explicit and structured memory module (and its read and write method) is an innovation, providing a clear and interpretable way to store and retrieve knowledge.\n2. The experiments are sound and the results are promising -- MemLLM demonstrates substantial improvements in language modeling tasks and knowledge editing tasks.\n3. The memory's scalability and interoperability make it a versatile solution that can be adapted to various tasks and LLMs.\n\n## Weaknesses\n1. Not enough baseline methods for fair comparison (especially for knowledge editing tasks).\nThe current implementation of MemLLM handles only 96 relation types, which may limit its applicability to more complex knowledge extraction tasks.\n2. As mentioned in the Limitation section, the model does not support composite relations inferred from multiple extracted relations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces  a novel method for enhancing LLM by integrating an explicit read-write memory module, called MemLLM. The key contributions of the paper include: 1)  interpretable, editable, and scalable memory module for LLMs; 2) improves language modeling performance, particularly in knowledge-intensive tasks (lower perplexity); 3) better knowledge editing performance. MemLLM addresses several limitations of current LLMs, including difficulties in memorizing rare events, updating memory over time, and preventing hallucination.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I agree with authors that \"In a real-world scenario, the size of the memory and, consequently, the size of query results will get large.\" and Table 2 shows a relatively small negative effect of memory size. If an figure specifically for controlling variable analysis on the impact of scaling memory size could be added, it would more clearly illustrate the trend brought about by further scaling.\n\nAuthors make a thorough related work discussion. But the baselines used in the experiments are simple and not enough, could you provide more baseline method results or explain why some previous methods can not be adopted for comparison?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NhskOYM2hi", "reviewer": "Reviewer_pBKB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n* The idea to finetune the language model to perform read and write operations over the external memory through the natural language expression is sound.\n* The illustrations in Figure 2 clearly show how to perform memory read and write operations.\n* The ablation of the proposed approach in Table 2 is extensive, which clearly shows what is the advantage of the proposed approach and its room for improvements.\n\n---\n\n### Weaknesses\n* The approach to extracting the vectors for triplet elements is not clear. More specifically, how to store the vectors for the extracted entities? If the entities appear over multiple sentences and documents, their vector representations differ across them. And, how to handle this case?\n* It is not clearly discussed whether there is an advantage to building and utilizing the external memory (consisting of the triplet-form facts) over the approach that directly retrieves and uses the raw documents (without converting them into the external memory). Experimental validations on it may be needed. \n* The discussion about the related work [1] is not so clear. Why this related work is task-specific (unlike this work), and what are the major contributions that this work makes over this related work?\n* The proposed approach to storing the facts in the external memory is similar to the approach that extracts triplets over documents for knowledge base construction [2, 3]. Also, similarity, the proposed approach to using the facts stored in the external memory can be viewed as augmenting the language model with the knowledge retrieved from the external knowledge base (or graph) [4]. However, this work lacks discussions about them and, furthermore, the proposed approach may be potentially compared against them. \n* The experimental validation is weak. Specifically, the baseline chosen is one (except for the naive language model prompting baseline without external knowledge), while there are many studies on (parametric or non-parametric) memory-augmented language model approaches, which are also discussed in the Related Work section. Also, for the knowledge editing task, while the authors claim the weaknesses of the existing approaches for it, the authors do not provide experimental evidence on it (i.e., the proposed approach can then handle the limitation of existing approaches in handling multiple facts updated). Lastly, it may be beneficial to evaluate the proposed approach with other language models (other than only one used in this work, namely Mistral-7B)\n* The process to generate the memory-read and memory-write data discussed in Section 3.3 is a bit complex. Explaining them with Algorithms may be beneficial. \n* When storing the element in the memory, how to handle the entity with the same name but different meanings? For example, the word Apple can indicate the fruit or the company.\n\n---\n\n[1] ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory. \n\n[2] Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction.\n\n[3] LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities.\n\n[4] Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims to tackle the limitation of existing language models (whose parametric knowledge can be incomplete and outdated) by using the concept of structured external memory, which can store facts in their triplet formats, e.g., (head entity, relation, tail entity). The major contribution that this work makes over existing works is to train the language model to perform read and write operations over the external memory, and, more specifically, during the write operation, the trained language models extract the facts over the sentences, meanwhile, for the read operation, the stored facts within the external memory are retrieved based on their embedding-level similarities with the query and augmented to the language models. The authors validate the proposed method, called MemLLM, on the perplexity-based evaluation task and the knowledge editing task, showing that it outperforms its variants (or its ablated versions).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the Weaknesses above.", "broader_impact_concerns": "While the authors do not present the Broader Impact Statement, there is no specific concern associated with the proposed approach.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TQy39RifsX", "reviewer": "Reviewer_EYDD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. MemLLM has both read and write access to the memory component, i.e., it can store information in the\nmemory as it processes text (or interacts with a user) and retrieve it when it needs it.\n\n2. It specifies an API for read and write access. The LLM issues read commands to retrieve from the\nmemory and write commands to write to the memory.\n\n3. Its memory has an explicit structured schema, similar to a database schema: interpretable， editable，scalable，and interoperable\n\nWeakness: \n\n1. The method part is very hard to read. For example, the notations are undefined in the paragraph of Memory writes and read. The figures are also hard too read. I suugest the authors to reorganize this part to improve the clearance. \n\n2. Inefficient experiments: (1) For LLMs, the paper only considers Mistral-7B. Larger LLMs are needed. (2) Compare to other KE methods, there is a  performance gap in reliability. While the authors provided some analysis. I think some. discussion on improving it is needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces MemLLM, a novel method of enhancing LLMs by integrating a structured and explicit readand-write memory module. MemLLM tackles the aforementioned challenges by enabling\ndynamic interaction with the memory and improving the LLM’s capabilities in using stored\nknowledge. Experiments indicate that MemLLM enhances the LLM’s performance and\ninterpretability, in language modeling in general and knowledge-intensive tasks in particular.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness part.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a read-write memory to LLMs. A model can perform write operations after seeing relations expressed in text, then subsequently read from that memory while processing later text, enabling it to surface relevant entities for use in next-token prediction.

Reviewers generally liked the proposed method of the paper. I think the paper presents a clear idea and is well-written. The main issue with the paper is raised by reviewers KCAv and pKBK, who point out that there are other baselines or competing methods that could be compared to.  Reviewer pBKB points out some closely related prior work. In particular, many practitioners may choose to use an approach like RAG over text, or a more explicitly pipelined text -> KB -> RAG over the KB system. The restriction to the 96-relation inventory in DocRED/Re-DocRED seems like a major limitation that would cause practitioners not to adopt this approach, as other labeled datasets for other relation inventories may not be readily available. Some of these weaknesses are discussed in the paper, but not in a centralized way, which seems to me that it doesn't give the whole picture here.

**Requested change:  Please add discussion of how the approach conceptually overlaps with ChatDB and the other papers mentioned by pBKB in the Limitations section. I would also suggest moving content from footnote 4 into the Limitations section and being more up front about the strengths and weaknesses of RAG.**

(Comparison to these baselines would also be great, but I don't think this is strictly required given the current framing of the paper.)

Please also upload a clean copy without any blue formatting; thanks!
Decision: Accept (poster)