Pre-training Limited Memory Language Models with Internal and External Knowledge
Authors: Linxi Zhao
Sofian Zalouk
Christian Belardi
Justin Lovelace
Jin Peng Zhou
Ryan Thomas Noonan
Dongyoung Go
Kilian Q Weinberger
Yoav Artzi
Jennifer J. Sun
Venue: ICLR 2026 Poster (2026)
Keywords: Pretrained Large Language Models
Knowledge Offloading
Primary area: foundation or frontier models, including LLMs
TL;DR: A new class of language models that offloads factual knowledge to an external database rather than encoding it in their parameters

Abstract: Neural language models are black-boxes--both linguistic patterns and factual knowledge are distributed across billions of opaque parameters. This entangled encoding makes it difficult to reliably inspect, verify, or update specific facts. We introduce Limited Memory Language Models (LMLM), a new class of language models that externalizes factual knowledge to external database during pre-training rather than memorizing them. Our pre-training approach strategically masks externally retrieved factual values from the training loss, thereby teaching the model to perform targeted lookups rather than relying on memorization in model weights. Our experiments demonstrate that LMLMs achieve competitive performance compared to significantly larger LLMs on standard benchmarks, while offering the advantages of explicit, editable, and verifiable knowledge bases.

Published as a conference paper at ICLR 2026
PRE-TRAINING LIMITED MEMORY LANGUAGE MOD- ELS WITH INTERNAL AND EXTERNAL KNOWLEDGE
Linxi Zhao, Sofian Zalouk, Christian K. Belardi, Justin Lovelace, Jín Peng Zhou Ryan Thomas Nooman, Dongyoung Go Kilian Q. Weinberger, Yoav Artzi, Jennifer J. Sun Department of Computer Science Cornell University
{1z586, saz43, ckb73, j13353, jz563, stn27, 1g793)6cornell) edu corfess edu
(kilian, jjs533]6cornell. edu
ABSTRACT
Neural language models are black-boxes both linguistic patterns and factual knowledge distributed across billions of opaque parameters. This entangled encoding makes it difficult to reliably inspect, verify, or update specific facts. We introduce LIMITED MEMORY LANGUAGE MODELS (LMLM)) new class of language models that externalizes factual knowledge to external database during pre-training rather than memorizing them. Our pre-training approach strategically masks externally retrieved factual values from the training loss, thereby teaching the model perform targeted lookups rather than relying on memorization model weights. Our experiments demonstrate that LMLMs achieve competitive performance compared to significantly larger LLMs on standard benchmarks, while offering the advantages of explicit, editable, and verifiable knowledge bases.
INTRODUCTION
Many challenges with deploying LLMs in real-world applications originate from the fact that training on vast text corpora exposes L.MSs to vast amounts of knowledge (Kaddour et al.) [023]] Allen-Zhu Li 2024a). Ideally, the knowledge in language models should be fully controllable a customer service agent for a restaurant chain shouldn' 't have the ability to answer questions about historical facts, prescription medicine, real estate law, etc. Unfortunately, current LLM pre-training procedures entangle the learning of knowledge with language competency in their neural weights. This has significant implications for both training and inference. During pre- pre-training, facts must observed many times for reliable memorization (Allen-Zhu & Li. 2024c Kandpal et al. (023). During inference, it is difficult to unlearn knowledge that may be outdated or inappropriate for particular application (Longpre et al., [2021] Maini et al.) [2024] Ovadia et al. 2024 Deeb & Roger 2225).. This tight coupling of knowledge and linguistic ability makes updating one without affecting the other extremely challenging.
LLM + RAG
LMLM
Language
Language
Competency
Competency
External Knowledge
External Knowledge
Limited
Internal Knowledge
Internal
Knowledge
Database
Database
Model
Model
Figure 1: A schematic illustration of LMLM. Unlike RAG (left), which exclusively adds knowledge from external sources, LMLM offloads knowledge from the LLMs the external database during pre-training.
'LMLM M stands for Limited Memory Language Model and is pronounced "Lamb lam". We open-source code and models at wttptttps////mm htttpps://ttthe / an group/

---

Published as a conference paper at ICLR 2026
Our vision is a pre-trained language model where knowledge is fully modular and controllable, and can be added or deleted easily for different use cases. This leads us to conclude that storing knowledge inside the model weights during pre-training should be limited as much possible. This gives rise to the following research question:
Can factual memorization be disentangled from language understanding in language models?
To realize this vision, introduce LIMITED MEMORY LANGUAGE MODELS (LMLM), new class of language models with a pre-training recipe that teaches the model to query an external database for entity-level facts rather than memorizing them. We provide an integrated solution for LMLMS spanning data preparation, pre- trainiiinnn, and inference. To prepare the training data, we annotate the pre-training corpus with database lookups to offload factual content, using a small, cost-effective LM fine-tuned for this task. During pre-trainnn pre-training, the returned facts are masked from the loss, systematically separating factual knowledge from the neural weights. During inference, instead of recalling memorized facts, the model queries the database.
Unlike the dominant paradigm of post-training and inference-time approaches such as retrieval- augmented generation (RAG) (Lewis et al. 2021) that maximize access to external information, our approach takes the perspective limiting knowledge stored inside the model parameters during pre- training (Figure D. This frees up model capacity, disentangles knowledge from language competency, and enables precise control over factual knowledge. During inference, LMLM is naturally compatible with RAG systems, tool calling (Schick et al. (023), or task-specific fine-tuning (Wei et al. (022). LMLMs always rely on lookups for factual information, so attempts to access out-of-scope know ledge trigger detectable lookup failures. In contrast, traditional RAG models without LMLM pre-training typically fall back to their internal knowledge. which is beyond the control of the user and may lead hallucinations or misinformation (Longpre al. 202T Mallen al. 2023 Sun al. 2025).
We compare our LMLMS against LLM counterparts of the same size, pre-trained on the same corpus without external memory. Our experiments demonstrate that LMLMS achieve lower perplexity than comparable LLMs ($ 4.2) Crucially, by externalizing knowledge from model weights, LMLMS achieve significant parameter efficiency-our 382M- -prrameer model matches the factual precision of a LLAAMA--7 ($ 4.44. Knowledge externalization also enables instant, verifiable updates and unlearning through simple database operations ($/..). Additionally, we present findings regarding the knowledge decoupling achieved by LMLM and examine the trade-off between internal memorization and external offloading 5). LMLMS offer a pathway toward language models with substantially reduced dependency on parameter count for factual accuracy. LMLMs have potential for integration with knowledge representation (Pan et al.] (2024). symbolic reasoning (Chaudhuri et al.] [022]. and mechanistic interpretability (Bereska & Gavves 2024), potentially transforming how language models store, access, and maintain knowledge.
2 RELATED WORK
Parametric vs -on-Paramttric Knowledge. Language models represent knowledge in two main ways: parametrie, encoded in model weights during training. and non-parametric, retrieved from external sources during pretraining or inference (Mallen et al. 2023). Pretrained models have been shown function implicit knowledge bases, storing millions of facts in parameters (Devlin et al. 2019 Petroni et af., 2019). Scaling model size and training data can further increase this
Table 1: Comparison of selected related work. V = supported, X = not supported. Full version in Table 13 in Appendix.
Knowledge Storage
Performance
Model
Internal
External
Integration
PPL.
Factuality
Unlearning
Goal
Jnference
KN
DS
RNN search e poob. interp. Retrieved does in prompi
Neighbor interpolation Improe factuality
RAG
Docs
x
x
Post training
Tooforrer
Toul calls
Extend via
Pretmining
RETRO
/f
Docs
Cross-attn. to retrieved chinks Memorization isolation
Scale with retrieval Localize memorization
MemSinks
/ (sink neurons)
x
M
/ipartial) (iportial)
Spurse KV KB
Self astention
Reduce memorization
LMLM
Explicit lookup
Decouple facts from weights
2

---

Published as a conference paper at ICLR 2026
Extracting Knowledge
Pretraining
Inference
Prompt: When was Napolcon born?
Annolated Text:
Nepoleeo was bor on (Napoloo Biith Dete)- AAugust 15, 1789) August 15, 1769.
(Nppolonn, Birth Dato)
Original Text:
Napoleon was born on August 176g.
Annotator
Next takan prediction masked Valiee
August 16, 1769
Database
(Napoloon, Birth (aata)-Auguzs 15, 1789
Output: Napolean August 15, 1789.
Special Tokens
Enily
Relafion
Return Value
Figure 2: Overview of the LMLM framework, Our framework consists of (Left) Data Preparation, where entity-level facts are automatically annotated and stored in an external database; (Middle) Pretraining, where the model is trained on the annotated text while excluding return values from the loss discourage memorization; and (Right) Inference, where the model interleaves text generation with database lookups to ground its outputs on retrieved facts.
knowledge capacity and improve factual precision (Brown et al. 2020 Izacard & Grave 2022
Allen-Zhu & Li 2024c). Yet, recent findings suggest that parametric knowledge may remain under-
trained if the knowledge is observed less than few hundred times during training (Allen-Zhu & Li
2024c). Broadly, parametric knowledge suffers from well-documented issues, including hallucination.
staleness, weak attribution, and limited adaptability (Peng et al., [2023a] Huang et al. [0024] Kandpal
cct al. 2023 Bi al. 2025) Allen-Zhu & Li, 2024b), motivating the shift toward non- non-parametric approaches that explicitly externalize factual knowledge.
Retrieval-Augmeented LLMs. A large body of work improves factuality by incorporating external knowledge inference or training time. Retrieval augmeette generation (RAG) retrieves relevant passages (Lewis et al., | Izacard & Grave 2021 Shi et al. 2023), while tool-augmented models such as Toolformer extend model capabilities with API or tool calls (Schick et al.) 2023 Yao
et al. 2023). Retrieval- -asse pretraining approaches, including REALM (Guu et al. 2020) and
RETRO (Borgeaud et al. 2022), demonstrate that integrating retrieval during training can reduce memorization and improve generalization. Semi-parametric models further externalize knowledge by attaching non- aon-arametric key-value memories (Khandelwaz wall et al.] 22200] Peng et al. 2023b Shi et al.
2022: Pan al. 2023 Wu et al. 2022 Verga et | |021 Wang et al. 2024 Wei et al. 2025 Yang ct al. 2024). In contrast, LMLM directly restricts memorization during pre- pre-training. This design enables scalable and dynamic factual memory, and naturally supports properties such instant unlearning- catrriinggcapablittteee that are difficult to achieve with prior retrieval-augmented methods (see
Table D
and Table 13)
Machine Unlearning. Machine unlearning aims remove specific knowledge from trained language model without full retraining (Jang et al. 2022 Cao & Yang 2015). Existing approaches mainly rely on gradient-based updates (c.g.. GA, GD Liu al. 2022 preference-based optimiza- tion (c.g., IdkDPO, NPO, SimNPO (Maini et al. 2024 Zhang et al. 2024b Fan et al. [0025] Rafailov
al. ). While effective on unlearning targeted knowledge on TOFU benchmark (Maini et al.
2024 these methods remain difficult to scale to larger forget sets and often degrade utility or erase unintended related knowledge, reflecting the entanglement of factual content and general ability in model parameters. MemSinks (Ghosal et al. 2025) instead introduces mechanisms to isolate memorized content during training, making it easier to unlearn. Our approach departs from both lines by structurally decoupling factual knowledge from the model itself. Facts externalized to a database, so forgetting reduces to deleting entries. This design enables scalable, precise, and verifiable unlearning. while naturally extending to knowledge editing tasks (Mitchell et al. 2022
Sinitsin et al.) 2020 Fang et al. 2025).
3 LIMITED MEMORY LANGUAGE MODELS
LMLM is class of language models designed to offload factual knowledge to external database rather than store it implicitly in model parameters from the pre-training stage (Figure 2 We outline pre-training data preparation ($ 3.1,. pretraining with lookup masking and inference 3.2
3

---

Published as a conference paper at ICLR 2026
B
1) Annotate GPT40
=
zof Finetune Corrector
Clean Data
Instruction tuning
Original Seed Data
Annotated Seed Data
Corrector
Cleaned Seed Data
Annotator
1. Seed Annotation
2. Filtering
3. Annotation
Figure 3: Training the ANNOTATOR model. We distill high-quality annotations from GPT-4o into a lightweight model that learns to identify and externalize factual knowledge from pre-training text, enabling scalable annotation of the full corpus.
3.1 PRETRAIN DATA PREPARATION: SEPARATING KNOWLEDGE
We begin by extracting atomic entity-level factual knowledge from pretrain corpus and constructing compact external database. These extracted facts are then interleaved with the original pretraining corpus through explicit lookup calls.
Knowledge Specification. We focus primarily on entity-level atomic factual knowledge, a natural starting point within the broader LMLM framework. We define facts as triplets of the form: (ent ity, relation) + value. This level of granularity aligns with previous definitions (Mallen et al. 2023) and represents the most compact and tractable form of factual knowledge to disentangle from the intertwined linguistic patterns in raw text. It also naturally maps to a knowledge graph structure, where triplets define nodes and edges (Choudhary & Reddy 2023). These atomic facts are ideal for externalization: they are straightforward to extract and verify, yet hard to encode in the model parameters, making them well-suited for storage in an external database.
Automating Knowledge Extraction. Manually extracting factual triples and constructing knowledge graphs at the scale of pre-training data is a major challenge (Hu et al., 2024). We address this by first obtaining high-quality seed annotations with GPT--4,, then refining them through filtering stage, and finally distilling the result into lightweight ANNOTATOR that scalably externalizes factual knowledge from raw text (Figure 3 Additional details and prompts are provided in Appendix A.
1. Seed annotation: We use GPT-4o to annotate a small seed dataset of M knowledge-intensive documents with lookup calls and return values. We use M 1000 our setting.
2. Filtering: We fine-tune a CORRECTOR model (LLAMA-3. 3. S-------IISTCC on the seed annota- tions with intentional underfitting. The CORRECTOR assigns high loss to improperly formatted, contextually unsupported, or overly specific lookup calls. We discard the top 10% by loss.
3. Annotation: We apply instruction-tuning to an ANNOTATOR model (also LLAMA-3.1--8 INSTRUCT) on the cleaned data and use annotate the full pre-training corpus at scale.
This annotation process serves purposes: (1) Database Construction: The extracted triplets form a token- eefennefficien external database that scales with the size of the pre-training corpus. (2) Pre-training Corpus Generation: Lookup calls are interleaved with the original text, enabling the model to learn when to rely on internal knowledge and when to issue a lookup.
3.2 PRE-TRAINING AND INFERENCE
Pre-training We adopt standard next-token prediction setup with one critical modification: During pre-training, tokens corresponding to the retrieved factual values and the ending lookup token excluded from the loss computation (Appendix A.3).
Dt E (retrieved values, <I dil.ddndd end >
((0)
Σ mi log ( x<)),
m
1, otherwise
This design discourages LMLM from memorizing facts that offloaded the external database. By interleaving lookup calls in the pre-training data together with the modified next-token prediction loss, it simplifies modeling factual content by decoupling knowledge learning into steps: (1) the model learns to generate lookup calls, with arguments inferred from context; and (2) factual
'Specifically, we represent each fact db_start entity sep relation < <ldb_ret retrievee
value < db_end > in the pretraining data.
"e skip the loss on < db_end > since, after retrieval, the token can be inserted directly during generation and does not need to he learned.
4

---

Published as a conference paper at ICLR 2026
FLAMAR JRMM
Normolloon Rool PPT..
Forpect Quality (p-value)t
Ideal Unleerinng
IALEPO
20
42
PhreSeoos
LLAMAA--30MM
Utility Trade-iff
1705
GETER COOAH
ALAM 19177
GD Statiing
Pretzain Tokens
Average Performatoo
Modeel Utility
3MEM
STANDARD
LMEM
STANDARD
LMLLM
LNLM thaom
Figure 4: Results overview. (Left) LMLM achieves consistently lower perplexity during pre-training, indicating that offloading factual knowledge improves pre-training efficiency. (Middle) LMLM significantly improves factual precision over its STANDARD counterparts while maintaining NLU performance. (Right) On the TOFU machine unlearning benchmark, LMLM forgets targeted facts while preserving general model utility, Results shown for LMLM with LLaMA backbone; denotes off-the-shelf models.
knowledge is stored and retrieved explicitly from external database rather than encoded in model parameters. This decoupling forms the core advantage of LMLM over the knowledge-deng counterparts. Intuitively, when the model can rely on accurate externally provided facts, no longer needs to expend capacity learning complex, long-tail factual distributions. The resulting architecture can achieve better parameter efficiency by dedicating model capacity reasoning and linguistic competencies. Empirically, this effect is reflected in faster convergence in pre-training and lower validation perplexity (Section
Inference Similar to tool- col-auumented models (e.g., Toolformer (Schick al. 2023 ). LMLM gener-
ates text autoregressively until special token triggers database lookup. retrieves the corresponding value, and continues generation.
EXPERIMENTS
We evaluate LMLMS via validation perplexity ($ (.2..), machine unlearning ($ as well as factual precision ($ 4,44) with summary of these results in Figure We further discuss additional implications on LMLM in $
4.1 EXPERIMENTAL SETUP
Pretraining and Model Setup. We pretrain on high-quality Wikipedia corpus (~ 3B tokens) from the OLMo2 project Grreeneveld et al. 20244, and evaluate perplexity on held-out set of 1 1,000 samples 2455 tokens). We prait LMLM from scratch using GPT-2 and LLaMA2-style architectures with their standard tokenizers and vocabularies, extended by four special tokens for lookup calls. All models are trained for 8 epochs with a context length of 1,024 tokens, using mixed precision. Each model completes training within H100-days. Details are in Appendix A.2
Database and Retrieval Setting. We construct the database by annotating the entire pretraining corpus, resulting in 54. 6M knowledge triplets. Retrieval uses fuzzy matching with cosine similarity over -LL-M12 embeddings (with rejection threshold of 0.6).
Baseline Comparisons. We consider the following pre-training settings:
LMLM (Ours): Pre-trained on our annotated data with lookup calls, using the loss in Equation
STANDARD: Pre-trained on data without lookup calls. All other settings identical.
OPE-THE-SHELF MODELS: Models with publicly available pre-training weights, including: OPENAI/GPT2-I24M, OPENAI/GPT2-355M, OPENAI/GPT2-774N (Radford et al. 2019).
http:s//wogginniro...oo//atssssa / no-mix-1122
5

---

Published as a conference paper at ICLR 2026
LLAMA2-7B (et al. 2023), and LLAMA3.1-8B (et al. 2024). OPENAI/GPT2-122 124M and OPENAI/GPT2-355M are comparable in size to our LMLM models. These models marked with an asterisk (*) in the results tables.
PYTHIA- 1B (Biderman al., 2023),
4.2 LEARNING TO LOOKUP FACTS IS EASIER THAN MEMORIZATION
Evaluation Setup. We first evaluate our models using language modeling perplexity on a held-out Wikipedia validation set. As LMLM introduces lookup calls, its perplexity is not directly comparable to that of STANDARD. To ensure a fair comparison, we report three variants of perplexity:
Static (Oracle): Assumes perfect lookup behavior, where LMLM always generates correct lookup calls and retrieves the correct values. This provides an optimistic lower bound. Perplexity is computed over all tokens excluding the lookup calls.
Dynamic: Reflects actual model behavior during inference. Lookup calls generated and executed in real time, capturing failures due to incorrect queries or failed retrievals. Perplexity is again calculated over all tokens except the lookup calls.
Normalized: Measures the combined likelihood of generating the correct queries and subsequent text. Perplexity is computed over all tokens except for the retrieved values, and normalized by the
number of tokens in the original unannotated text. See Appendix A.3 for formal definitions.
Perplexity Results. Figure re- ports validation perplexities for both LMLM and STANDARD models.
14
Standard
LML M-Dynamic
12
11.8
LML MMLM-Static (Oracle)
Normalized
LMLM were simply benefiting from retrievals without learning query properly, we would expect both Dy- namic and Normalized perplexity to remain high. Improvements across all variants indicate that LMLM learn- ing both to query and to generate more effectively.
Perplexity
9.9
9.1
9.2
8.5
7.7
6.7
6.7
6.3
5.8
5.2
GPT2-124M
LLaMA2-176M
GPT2-355M
LLaMA2-382M
Model Architecture
We observe that LMLM consistently achieves lower perplexity than STAN- DARD across all model sizes and per- plexity variants. In particular, LMLM achieves average perplexity reduction of 1.98 points under the Dynamic setting, demonstrating its effectiveness even with imperfect lookup calls. The Normalized variant highlights that LMLM assigns higher likelihoods to both lookup queries and grounded text, indicating improved training efficiency. These results support a key insight: Learning to lookup specific facts is easier than memorizing them.
Figure 5: Validation perplexity comparison between STAN-
DARD and LMLM on three variants of perplexity. Lower perplexity indicates better performance.
Downstream NLU performance. We compare LMLM and STANDARD models on five standard natural language understanding (NLU) tasks (Table I4]. This evaluation serves as sanity check to ensure that separating factual knowledge during pretraining does not at the expense of general language understanding. We find that LMLM performs on par with STANDARD across all tasks, confirming that factual offloading preserves the model's general-purposs capabilities.
4.3 MACHINE UNLEARNING: LMLM SUPPORTS INSTANT FORGETTING
One natural benefit of decoupling knowledge from model parameters is that editing and unlearning are achievable through simple operations the external memory, without compromising the model's general capabilities. To verify this, we extend our evaluation to a standard machine unlearning benchmark, TOFU (Maini et al. 2024).
Evaluation Sctup. TOFU evaluates unlearning efficacy in a privacy-sensitive setting, where the goal is to selectively forget targeted subset of information (the Forget Ser) while preserving performance on the Retain Set and maintaining general model capabilities. The objective is to produce an unlearned model that is statistically indistinguishable from model trained solely on the Retain Set (referred to as the retain model). The benchmark consists of 200 synthetic author profiles, each containing 20 QA pairs. It evaluates two key aspects:

---

Published as a conference paper at ICLR 2026
Porget Quality (povalue
0.60
Faget Answer ROUGE
0.8
ROUGE
0.8
0.3
0.55
Model Utiltyy
0.50
0.6
0.6
0.2
0.45
0.4
0.4
0.40
0.0
0.2
60
60
20
40
Unlearning Steps
Unlearning Steps
Unlearming Steps
Untearning Steps
Standard Unleam (NPO)
Standard
Retain (Ideal)
LMLM
Figure 6: Evaluation of Machine Unlearning. We compare LMLM with NPO on the TOFU benchmark. Unlike prior methods, LMLM performs unlearning without any additional training. (a-b) Forget quality vs. utility trade-off. LMLM achieves ideal forgetting (p-value > 0.05) without sacrificing general utility. (c-d) LMLM retains knowledge outside the forget set, unlike other methods that degrade retain-set performance due to parameter entanglement. Y-axis labels use or t to indicate retention or improvement. For metrics like Forget ROUGE, where lower isn 't always better, mark the retain model's performance reference.
Model utility: The average of three metrics- ROUGE (answer quality), Answer Probability, and Truth Ratio (likelihood assigned the correct answer over distractors)- measurred on the Retain Set, Real Author Set, and World Facts Set.
Forget quality: The p- value of statistical test comparing the unlearned model with the correspond- ing retain model to assess whether the targeted knowledge has been effectively removed.
We use LLAMA-3.. 1.MM22..2.2 2- 10-IB-INNg IB-IIBTRIRE I-SNSTTTCT the base model and compare against NPO (Zang et al.
2024b), a state-of-te ofate-of-thhearr unlearning method. For LMLM, we perform unlearning by simply removing entries in the database corresponding the Forget Set. See Appendix [B22 implementation details.
Unlearning Results. Figure 6 presents the results on the TOFU benchmark, where the Forget Set is 5% of the data. In Figure 6 (a-b), we show the forget quality and model utility throughout the unlearning process, where ideal performance is defined by effective forgetting without degrading model utility. LMLM achieves precisely efis-tffective forgetting with no loss in model utility-a direct benefit of decoupling factual knowledge from model parameters. Figure 6 (c-d) shows that LMLM preserves knowledge outside the Forget Set, whereas previous training-basee based methods tend to forget related information due to parameter entanglement. Importantly, forgetting is performed through simple database operations, without model updates or access to the Retain Set. contrast, RL-based methods such NPO, sfate-of-the-art baseline on TOFU, incur a utility degradation, while other unlearning methods either fail to forget or exhibit catastrophic drops model utility (See Figure 4 right). While LMLM incurs upfront costs for data annotation and model training. it provides substantial payoff in use cases where knowledge editing, removal, or compliance with data deletion requests are necessary.
4.4 EXTERNALIZING KNOWLEDGE IMPROVES FACTUAL PRECISION
We evaluate factual precision on long-form biography generation (FactScore) (Min et al.) (023) and short-form factual comple- tion using T-REx (Petroni et al.,
Table 2: Evaluations on factual precision. Subscripts show the
absolute difference from their respective STANDARD basclines.
Midel
Model Type
FactScore
T.Ex EM
PopQA Acc
OPENAI/GPT2 124M"
14.6
20.1
18.51
GPT2-124M
STANDARD
202T) and the long-tail subset of POPQA (Asai et al. 2023). FactScore measures the propor- tion of atomic facts supported by a trusted source, while T-REx and PopQA follow EM and Acc metrics from prior work (Schick al. 2023 sai al. 2023).
LMLM
20.6
54.6,
49,89,
LLAMA2- 176M
STANDARD
10.1
46.3
24.59
LMLM
30.6,mm
54.1
49.61
OPENAJ/GPT2-S55M
15.2
28.4
19.1
GPT2-355M
STANDARD
I.
23.9
58.7
52.0
LLAMA2-382M
STANDARD
14.0
$2.0
22.7
LMLM
31.9,174
58.1
50.8.
OPENAL/GP12-774M РYTHA-IBB*
17.4
35.6
18.5
21.1
As shown in Table LMLM con- sistently improves factual preci-
LLAMA2-7B
34.0
60.5
29.2
LLAMA3.1
7

---

Published as a conference paper at ICLR 2026
sion over STANDARD, its knowledge-densse counterpart. On LLAMA2-382MM it gains +17.9% FactScore, +6.1% T-REx EM, and +28.1% PopQA Ace. Notably, LMLM approaches the per- formance of much larger models such as PYTHIA-IB IB and LLAMA2-7B, despite using far fewer parameters.
How Does LMLM Compare to RAG? Unlike Retrieval-Augmented Generation (RAG), which expands knowledge access at inference time, LMLM constrains factual memorization during pre- training by externalizing entity-level knowledge into a database. Since facts are not internalized in model weights, they can be instantly modified or removed through database operations alone ($ 4.3) capabilities that are difficult to achieve with post-hoe retrieval. Current RAG systems still encode facts in parameters, so modifying specific knowledge requires model retraining or careful prompting to override internalized information (Xie et al. 2023 Hsia et al., 2024 Tan et al. 2024
To contextualize their respective strengths, we compare against con- trolled RAG baseline that retrieves the top-4 Wikipedia articles and prepends them to the standard LMs during gen- eration. As shown in Table LMLMM
Table 3: Comparison of RAG vs. LMLM on factual precision. Results are shown for GPT2-355M.
Model
FactScore 1
T-REx EM f PopQA Ace f
OPENAI/GPT2-355M
15.2
28.4
19.1
RAGG
20.1
75.8
37.5
achieves higher FactScore and PopQA
GPT2-355M-LMLM
23.9
58.7
52.0.
accuracy at the current scale, indicat-
ing that it effectively queries its database for precise factual information. In contrast, RAG achieves stronger performance on T-REx, likely because the benchmark's 's queries are drawn verbatim from Wikipedia, making retrieval from the same corpus nearly perfect.
These results point complementary strengths: RAG enriches pretrained models (including LMLM) with broader contextual information at inference, whereas LML. restructures pretraining itself to provide fine-grained factual grounding that is directly editable and verifiable. hybrid system could retrieve documents with RAG for contextual understanding while using LMLM for precise, verifiable entity lookups within those documents.
5 FURTHER ANALYSIS
Does LMLM Still Memorize Facts in Its Parameters? We provide preliminary evi- dence that LMLM reduces factual memorization through its training design. Using the TOFU syn- thetic trainset, we compare training objectives by tracking the loss on return value tokens-an the factual answers intended to be retrieved rather than memorized.
Table 4: Disabling retrieval significantly reduces performance on both FACTSCORE and T-REx; see Table 10 for the full comparison.
Model Type
FactScore f
T-REx EM f
S STANDARD
14.0
52.0
LMLM (w/o database)
12.8
19.1
38.5
19.6
LMLM
31.9
58.1
As shown in Figure 7 models trained with standard SFT objective exhibit a rapid decrease in the loss on these tokens, suggesting memorization in the model's parameters. In contrast, LMLM, trained with the masked loss maintains high loss throughout training, indicating that these facts are not stored internally. LMLM's 's successful application in the machine unlearning benchmark ($ 4.3) further supports this finding. Additionally, we observe notable performance gap in Table 4 where factual precision drops substantially when the external database disabled serabledfforcing the model rely on internal parameters. These findings suggest that editing the database offers a direct way to control what the model knows and forgets.
How Much Knowledge Should Be Externalized? For our main experiments, we aim to exhaus- tively annotate all atomic entity-level knowledge. However, this may not be optimal. To explore the trade-off' between storing knowledge in model parameters versus offloading into the database, vise a ranking criterion that quantifies the value of each lookup. Facts that require a lookup to achieve high likelihood are difficult learn ttyiinallyylll long-tail tail or highly specific knowledge and should be externalized. On the other hand, externalizing facts that already have a high -yyicclllytyiiaallyyyyll common or easily inferred proviiddepoov limited upside and may harm general capabilities.
de-
8

---

Published as a conference paper at ICLR 2026
30-
39-
FactScore (%)
25
38-
Regular SFT
36-
7.0
100
203
300
35
0.25
0.5
0gg
6.5
10-
0.25
0.5
0.75
0g0
Global Step
Knowledge Offloading Ratio
Figure 7: Training loss on re- Figure 8: Effect of different offloading ratios; within our scope, turn value tokens.
more offloading is beneficial.
Concretely, we first train a LMLM model and STANDARD model for one epoch the data described in $ For each fact, we then measure the average loss difference on its factual value tokens. A large difference indicates that STANDARD LMs struggle to learn the fact (See Appendix B.3 for qualitative examples). Using this criterion, we pretrain LLMs with varying annotation ratios (c.g. For 90% knowledge offloading,
keep the top 90% knowledge with highest loss difference), interpolating between the two extremes of STANDARD (fully parametric) and LMLM (fully offloaded).
As shown in Figure 8 greater offloading consistently improves language modeling (lower perplexity) and factual precision, while preserving natural language understanding. This suggests that, for knowledge representable as triplets, increasing the offloading ratio beneficial. Extending this spectrum to broader and more abstract forms of knowledge remains an open direction for future work.
6 DISCUSSION
Toward Efficient Scaling via Knowledge Offloading. We summarize our main findings in Figure In particular, even smaller instances of LMLM match or exceed the performance of much larger off-the-shelf models. These results highlight the potential of LMLM to scale efficiently by offloading knowledge to external database, thereby maintaining strong factual accuracy with fewer parameters. While our experiments are limited to modest scales due to computational constraints, the observed trends suggest that LMLMs offer promising direction toward parameter-effiiiet language models that externalize factual storage. Such models may enable real-time, verifiable knowledge updates and open up new possibilities for deploying LMLM in resource-constee cooutrc-constraiieed or fact-sensitive environments.
Beyond Entity-Level Knowledge Offloading. By leveraging knowledge triplets, LMLM focuses on separating entity-level factual knowledge from linguistic competency. Current research on probing (Petroni et al. 2019 Hu al. 2024), evaluating (Min et al., 2023 Wei et al., 2024). and editing (Meng et al. 2023a) internal knowledge in language models similarly concentrates on entity-level atomic facts. However, extending our method beyond entity-level knowledge remains challenge. This difficulty arises in determining effective formats for separating knowledge beyond entity-focused triplets. For instance, using a simple QA function call format raises concerns about the potential for hallucinated facts during annotation.
Additionally, while LMLM attempts to minimize the memorization of factual knowledge. some knowledge still remains unremoved. More comprehensive benchmarks for probing internal knowledge in language models, as well as distinct benchmarks for disentangling knowledge and reasoning, remain underexplored.
Limitations. LMLM is a promising step towards separating factual knowledge from language models with many exciting future directions. Our current limitations include: (1) It does not guarantee perfect factuality during generation. Noise in the database and errors from fuzzy matching can introduce inaccuracies. However, such issues are easily traceable and verifiable for LMLM. (2) LMLM introduces additional tokens for lookup queries, which increases training and inference costs. (3) The current implementation focuses on entity-level factual knowledge, which captures only subset of the broader knowledge spectrum. (4) Our experiments are limited to small models and datasets due to compute constraints. Although sufficient to show core benefits, scaling up may improve performance and support more complex tasks.

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
We introduce LMLMs, a new class language models for externalizing knowledge alongside an integrated solution to achieve this. Our results demonstrate promising trends towards efficient use of model capacity and offloading facts onto an external database. LMLM represents an alternative way to store facts during pre-training and has the potential to be integrated with other common approaches developed for LLMs, including retrieval- based methods, symbolic reasoning, as well as knowledge representation. Consequently, LMLM opens up new ways for future language models to leverage the benefits of external knowledge databases such as verifiable updates fundamentally, it is much easier and more memory-efficient to learn how look up facts rather than to remember them.
ACKNOWLEDGMENTS
We thank Eric Enouen for valuable discussions and feedback. This research was supported by gift to the LinkedIn-Cornell Bowers Strategic Partnership. SZ is supported by the Defense Advanced Research Projects Agency (DARPA) under Grant No. D24AP00259-00. CKB supported by the National Science Foundation (NSF) through the NSF Research Traineeship (NRT) program under Grant No. 2345579. JPZ is supported by a grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) (567916). DG is supported by Empire AI Postdoctoral Fellowship. This work is partially supported by Open Philanthropy; the National Science Foundation NSF under awards OAC-231 1,21,, IIS-2107 11S-210716,, IIS-1724282, IIS-2505098 and HDR-2118310 NASA under award No. 20-0STFL20-0053: the Cornell Center for Materials Research with funding from the NSF MRSECCrrogran program (DMR-1719875), DARPA; arXiv; Apple; AI-MI and NSF Award 2433348; and the New York Presbyterian Hospital. We gratefully acknowledge use of the research computing resources of the Empire Al Consortium, Inc, with support from the State of New York, the Simons Foundation, and the Secunda Family Foundation (Bloom et al. (025). Any opinions, findings and conclusions or recommendations expressed in this material are those the author(s) and do not necessarily reflect the views of the National Science Foundation of NASA.
ETHICS STATEMENT
This work complies with the ICLR Code of Ethics. We use only publicly available datasets and do not involve human subjects or private data. While our method is methodological nature, we acknowledge general risks of bias and misuse inherent to language models.
REPRODUCIBILITY STATEMENT
We open-source our code and models at hthttps://gg provide full details of datasets, model configurations, and evaluation protocols in the main text and appendix to ensure that all results can be independently reproduced.
www.grrrooMi.. We
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Zeyuan Alh and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction, 2024a. UR__nttps://axxxv
http://abb2230...3/3333 343.14316
Zeyuan Allen- and Yuanzhi Li. Physics languuag models: Part 3.2, knowledge manipulation, 2024b.
14402
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of modelss laws, 2024c.
models: Part 3.3, knowledge capacity scaling
05405
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. URL httpp://arxiv..oo
abs 20s/2210..15111
Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patifio, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, Xuan-Son Nguyen, Colin Raffel, Leandro Von Werra, and Thomas Wolf. SmolLM3: smol, multilingual, long-context reasoner. https
//hugging wttpcc//blllcoc..cc 1m3 2025.
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety-a review. arXiv preprint ar MXXi:2244.14000 2024.
Baolong Bi, Shenghua Liu, Yiwei Wang, Yilong Xu, Junfeng Fang. Lingrui Mei, and Xueqi Cheng. Parameters vs. context: Fine-grained control of knowledge reliance in language models, 2025.
URL https:
15888
Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O' Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: suite for analyzing large language models across training and scaling, 2023, URL
org/ abs 204 0033344000 01373
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL hattps://arxiia org abs/ 1911
11641
Stacie Bloom, Joshua Brumberg, Ian Fisk, Robert Harrison, Robert Hull, Melur Ramasubramanian, Krystyn Van Vliet, and Jeannette Wing. Empire ai: A new model for provisioning ai and hpc for academic research the public good. Practice and Experience in Advanced Research Computing 2025: The Power of Collaboration. PEARC '25, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 979840071 13989. doi: 00.1145/3700035.333600
URL https
org.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford. Katie Milli- can, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irv- ing, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Lau- rent Sifre. Improving language models by retrieving from trillions of tokens, 2022. URL https: fictps///arxiz www./o//////./ooooooo/ooo/..ee//
04426
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https
//arxiv. org/ http://///bbbb/.......ob.
/2005.220511.0 14165
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium Security and Privacy, pp. 463-480, 2015. doi: 10.1 10.1109//..015.55.
11

---

Published as a conference paper at ICLR 2026
Swarat Chaudhuri, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando Solar-Lezama, Yisong Yue, et al. Neurosymbolie programming. Foundations and Trends@ in Programming Languages, (3):158-243 2021.
Nurendra Choudhary and Chandan K Reddy. Complex logical reasoning over knowledge graphs using large language models. arXiv preprint arXii:2305.01155. 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved questionn answering? try arc, the ai2 reasoning challenge, 2018. URL htttps://aaxi..rrrrgt////
05457
Aghyad Deeb and Fabien Roger. Do unleaming methods remove information from language model 028277
weights?, 2025.
Jacob Devlin, Ming- bidirectional transformers for language understanding, 2019. URL https://arxiv.om
Siig-Weiiee Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
Zhengxiao Du, Aohan Zeng. Yuxiao Dong, and Jie Tang. Understanding emerggnn abilities of language models from the loss perspective, 2025. URL httl:/tpss//xxx.vooro/////////// 15796
Aaron Grattafiori et al. The llama 3 herd of models, 2024. URL www.hhtss//arxiv.og/////
Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https
warxiv.org/ae abs///ba//// /20../277... 00.00..300 09288
Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for Ilm unlearning, 2025. URL
huttps://arxie http:////xxvivooo/x///eee
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat seng Chua. Alphaedit: Null-space constrained knowledge editing for language models, 2025.
URL https: org/ abs /20.0.211... 02355
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. En- tities experts: Sparse memory access with entity supervision. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4937-4951, Online, November 2020. Association for Computational Linguistics. doi: 10. 8653/v 00//6//////220000 /ttp:/.e.0.0mmmiinnnmmm httpps://aclantholooy/ wttp:/yyy..rgg//2.mmmmm..
URL
400/
Gaurav R. Ghosal, Pratyush Maini, and Aditi Raghunathan. Memorization sinks: Isolating memo- rization during Ilm training, 2025.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Sol- daini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models, 2024. URL httlhttp///// httpvviiivv..////a//...///a.aal/ org/ abs 2402.442..00 00838
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval- augmented language model pre-training, 2020. URL.
08909
Jennifer Hsia, Afreen Shaikh, Zhiruo Wang, and Graham Neubig. Ragged: Towards informed design of retrieval augmented generation systems. arXiv preprint (0t::2003000440 2024.
12

---

Published as a conference paper at ICLR 2026
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhir Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https
2106 09685
Yujia Hu, Tuan-Phong Nguyen, Shrestha Ghosh, and Simon Razniewski. Gptkb: Comprehensively materializing factual Ilm knowledge, 2024. URL /
04920
Lei Huang, Xiaocheng Feng, Weitao Ma, Liang Zhao, Yuchun Fan, Weihong Zhong, Dongliang Xu, Qing Yang. Hongtao Liu, and Bing Qin. Advancing large language model attribution through self-improving, 2024. URL AtLtttps::/arxiiittt wttpa:iw......///a///aaa 13298
Gautier Izacard and Edouard Grave. Leveragnng passage retrieval with generative models for open domain question answering, 2021. Utt.atttse//aart..../////////112
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering, 2022. URL /ttLLttpee///att..tt///aa//////p
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yie Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models, 2022. URL httth:tt////ttttt http//aaxivv...aaaaa.aa 2208
03299
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearming for mitigating privacy risks in language models, 2022. URL https://arxi:: http://////2l// 01504
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models, 2023. URL. https://arxiv
hogg/bb/2307.. 10169
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge, 2023. URL
2211
08411
Nora Kassner and Hinrich Schütze. BERT-KNN: Adding a kNN search component to pretrained language models for better QA. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the
Association for Computational Linguistics: EMNLP 2020, pp. 3424-3430, Online, November
2020, Association for Computational Linguistics. doi: 10. 1861/ 1/2020 findings-emmll.3300 URL hat.ps://clantthllogg/ wttp:/yyyooooooooooooooooeeeeee 2020 200.:/00000.......gg-eem1.......
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models, 2020. URL httphhtts///arxii.orrt
abs/ 191 19911.1000 00172
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval--uggmente generation for knowledge-intensive nip tasks, 2021. |
Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. 2022. URL wttp://///////....../222
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing, pp. 7052-7063, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 10.18533/1/2021. emalp-main.565. URL
Entity-based knowledge conflicts in question answering. In
emn a n. 1
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for lmms 2024. URL htt.:/tts///axxii.orgg.. http://wwrrrbbab...aab/a/// 06121
13

---

Published as a conference paper at ICLR 2026
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models; Investigating effectivenesss of parametric and non non-parametric memories, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023a. UKL.httsss//aaxxi..orr//aa///// 25222 05262
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory transformer, 2023b.
wwwp.///.wwgabbbb....ooo 07229
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen Yih, Pang Wei Koh, Mohit Iyyer. Luke Zettlemoyer, and Hannanch Hajishirzi. Factscore: Fine-grained atomic evaluation of factual preci- sion in long form text generation, 2023. htt.tttss://aaxii.
hbtg://b/////// 2305. 14251
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning, Fast model editing at scale, 2022. URI.
arxi wwwpaawwwww...//////a/aa.a 11309
Ali Modarressi, Abdullatif Köksal, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schütze. Memllm: Finetuning Ilms to use an explicit read-write memory, 2025. URL
abs/2404. 11672]
Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning retrieval? compar- ing knowledge injection in llms, 2024. URL htt.::/..s//axvv..rttraa//////// 25933
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering,366 36(7):3580-33) 2024.
Xiaoman Pan, Wenlin Yao, Hongming Zhang, Dian Yu, Dong Yu, and Jianshu Chen. Knowledge- in-context: Towards knowledgeable semi-parametric language models, 2023. URL https
//axxiv.ogg/abs: 2210.. 16433
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https::////. /ttpss:///riivii org. abs /240 1757
Baolin Peng. Michel Galley, Pengcheng He, Hao Cheng. Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: Improving large language models with external knowledge and automated feedback, 2023a. URL https
// //arxii.org//acooa..aa.aa..ooaaa 23022
Guangyue Peng, Tao Ge, Si-Qing Chen, Furu Wei, and Houfeng Wang. Semiparametric language mod- els are scalable continual learners, 2023b. URRt::ts::///rtt wwwpxiiv.or//aaao hatpg//bs/23000 .113310112111
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases?, 2019. URL httpps://arx
org/abs 01066
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. Kilt: a benchmark for knowledge intensive language tasks, 2021. URL
https://axxiv. org/ abs 2000. 02252
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl, 2019. URL https://cdd..peee wtt.../tt//b///ee..eeeeeeeee.e.. are wtt.:uuuuurrieeeemltttttteeeeeee leaiskkeeeaarrrreee .ddf Accessed: 2024-11-15.
Language models /language models
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimizztionn Your language model is secretly reward model, 2024. URL
www.:o/gg/bbbssaa 18290
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad, 2018. URL wtt.tttaaa//aaiii.oooortt
03822
14

---

Published as a conference paper at ICLR 2026
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu. Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2022. URL http://////aaai..oorgaaaa////// 08207
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions, 2019. URL /ttp::///arrvv..o/////bbbb 1tt///994.0595
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL
abwpr//aabb//32.. 04761
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer knn- Nearest neighbor zero-shot inference, 2022.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen tau Yih. Replug: Retrieeal-augmmeted black-box language models, 2023. URL httts:://arxiv.orrt wwwpa/s/.oooooooooooooooo..oosss 22tp. 12652
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks, 2020. URL
00345
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. CoLAKE: Contextualized language and knowledge embedding. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.). Proceedings of the 28th International Conference on Com- putational Linguistics, pp. 3660-3670, Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational Linguistics, doi: 1/220/03///222..oooiing cowinngmmiin.3277777 URL 3277
Zhongxiang Sun, Xiaoxue Zang, Kai Zheng, Yang Song. Jun Xu, Xiao Zhang, Weijie Yu, Yang Song. and Han Li. Redeep: Detecting hallucination in retrievallaugmmened generation via mechanistic interpretability, 2025. URL /tteps///axxiv.or/abb//////
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational L Linguistics. doi: 10. 10.18633/1//119142211 URL httpps:////ttt. a/acllccll antho logy org/N19 1421/
Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. Blinded by generated contexts: How language models merge generated and retrieved contexts when knowledge conflicts? arXiv preprint arXiv:2401.1 11911, 2024.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William Cohen. Adaptable and interpretable neural MemoryOver symbolic knowledge. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tuu Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3678-3691, Online, June www.cl-main.288.8 www.a//222nnaaallemmmmmaam main.288. URL
2021. Assoccition for Computational Linguistics. doi: naacl wasacc-maiin 2888
httpss//accaamtccoooy..o wwwp:ogg.oor...... http:///or........mmmmmmaaaaaaaa
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-adapter: Infusing knowledge into pre-trained models with adapters, 2020.
(tltptt:s: // org wwwg///b///aae 2002 00....1181 01808
15

---

Published as a conference paper at ICLR 2026
Xi Wang, Taketomo Isazawa, Liana Mikaelyan, and James Hensman. Kblam: Knowledge base augmented language model. arXiv preprint arXiv:2440. 10450, 2024.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners, 2022. URL
wwwrxxvv..oggaaaa. abs/ /219..219... 011191000000 01652
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality large language models, 2024. URL
http//aaxvvvv/aa http://wxwi...rrrr/////aa/ /ttpsa/2111..00 24368
Rubin Wei, Jiaqi Cao, Jiarui Wang, Jushi Kai, Qipeng Guo, Bowen Zhou, and Zhouhan Lin. Mlp mem- ory: A retriever-pretrained memory for large language models. arXiv preprint arXiv:2508.01832. 2025.
Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers.
2022. URL https: http:://axxvvvv.o
/ org/ abs/////////// 0200/2203.000 08913
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference Learning Representations, 2023.
Hongkang Yang, Zehao Lin, Wenjin Wang, Hao Wu, Zhiyu Li, Bo Tang, Wenqiang Wei, Jinbo Wang, Zeyun Tang, Shichao Song, et al. Memory Language modeling with explicit memory. arXiv preprint XXix:2407.01178 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https://axxx
arg/abs/2210 03629
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. htt.lttpss//aa...oooo///aa////// wwt.:/ogg//abbo... 07830
Peiyuan Zhang. Guangtao Zeng. Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024a. URL
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative prefeperrcrcccccccccccciiiii From catastrophic collapse to effective unlearning, 2024b. URL http:tsss//vavvvoor/////aa/
05868
Shaochen Zhong, Yifan Lu, Lize Shao, Bhargav Bhushanam, Xiaocong Du, Louis Feng, Yixin Wan, Yucheng Shi, Daochen Zha, Yiwei Wang, et al. Mquake-remastered: Multi-hop knowledge editing can only be advanced with reliable evaluations. 2024a.
Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing language models via multi-hop questions, 2024b. URL
https://aaae //ap:i///aiiviii/aa 2ttp://ww.......///202.......... 14799
abs
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023. URL https:://///..... / aaaxiiii vv hbtp:/bb//22330 12066
16

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
A.1 KNOWLEDGE EXTRACTION DETAILS
Model and Data. We construct high-quality seed dataset by sampling 1,000 passages from SQuAD-v2 (Rajuurkares et al. 2018) and 1,000 passages from Wikipedia. These passages are annotated by GPT-4o with structured factual triples and used to train the ANNOTATOR model. The remaining Wikipedia passages are used as the pre- pre-training corpus for LMLM, with no overlap with the validation set.
Both the CORRECTOR and ANNOTATOR are based on LLAMA-3 -,AA-311130 1.1.-8--BBB 3.1--8--ISSTRCCE selected for its strong eestruttionfoollwinne capabilities. The CORRECTOR and ANNOTATOR maximum context length of 2048 tokens. All input sequences are truncated to 1024 tokens during LMLM pretraining for consistency.
GPT-4o and ANNOTATOR annotations follow the format [dblookup (' Ent itt'ty' 'Relation's -> Valuel. These are converted to token-based format for pretraining: < </bbstarrt Entity < <|spp Relation < / db.valuel Value < dbbbend]
The CORRECTOR fine-tuned using LoRA (r=32, alpha=16
6) (Hu al. 2021 for epochs
on 19k GPT-44-annotated SQuAD-v2 passages, with a learning rate of 2 x 10 and an effective batch size of 32. Sequence packing enabled improve training efficiency.
The ANNOTATOR is tuned with instruction on the 2k annotated passages using LoRA (r=32, alpha-155 6), with learning rate of 2 x 10-4 and effective batch size of 32. Training runs for 10 epochs with a maximum sequence length of 2048.
Annotation Pipeline. We adopt three-stage pipeline to distill GPT-4o's structured annotations into a lightweight and scalable ANNOTATOR model:
Table 5: Prompt templates used for GPT-4o and ANNOTATOR annotation.
Model
Prompt Template
GPT-4n
You knowledge base construction expert. Extract entity-based factual knowledge from passuge and annotate it using the format: (:blookupg (' Entity' Melationshis
>> Valuel. These anmotations simulate a knowledge base query for factual generation. Place
db dblookup right after the entity and relationship appear, keeping the text flow natural.
Based Factual Knowledge Principles:
Entities: Use full names for people, organizations, places, or works.
Relationships: Use specific, reusable labels that define the connection clearly.
Values: Keep them concise and factual.
Annotation Principles:
1. Extract ALL Atomic Facts: Each annotation should capture single verifiable fact.
2. Precise Annotations: Use correct and specific entity-cllaionnshi-vaa
triples.
3. Ensure Reusability: Use standardized and reusable entity and relation names.
Contextual Positioning Rule: Place annotations only after both entity relation appear.
5. Preserve Text and Muintain Flow:
alter disrupt the original text.
Example Annotation:
Input: Beyoncé Giselle Knowles-Carter (born September 4, 1981) is an American singer, songwriter, record producer, and actress.
Output: Beyonce Giselle Knowles-Carter (born [dblockup (' Seyoncé Giaelle
"Birth Date' SepLember 4, 1982] September 4. 1981) is an [dblookug (' Beyoncé Giselle Knowles-Carter', "Nstionalit
KnowlessCarreet
Americani American [áblookup (" Bayoncé Giselle Prowles-Carted 'Ocoupation' -> singer, sonewriter, record producer, actress] singer, songwriter, record producer, and actress.
ANNOTATOR
Your lask extract and annotate entity-hased factual knowledge from the provided text. Identify and annotate specific entities, relationships, and values using the dblcokup format: [db cblookupp! ( "Entity' 'Relationss.pp -> Value] Annotation Guidelines:
Inline Insertion: Insert dbleckup before factual statements without altering the
AAomic Facts: Each dbl ookup should capture one verifiable fact.
Entitiis: Use full names for people, organizations, places, or works.
Relatioonhips: Use specific, reusable labels (avoid vague terms).
Values: Keep them concise and factual.
17

---

Published as a conference paper at ICLR 2026
Seed Annotation. GPT-4o is prompted to annotate imput passages with structured factual triples (entity, relation, value). These are embedded directly into the text using the lookup call format. Prompt template is provided in Table
Stage 1:
Stage 2: Cleaning. A warm-start CORRECTOR model is trained on the annotated data for 2 epochs without instruction prompts. Although underfit, it is effective at identifying noisy or ill-formed annotations. Specifically, discard lookup calls where the token-level loss the entity relation is in the top 10% of the distribution.
Stage 3: Annotation. We instruction-tume tunc a new ANNOTATOR model on the cleaned dataset for 10 epochs. This model learns to detect when factual knowledge should be externalized and how to issue structured lookup queries. The trained model then applied to the full pre- pre-training corpus to generate large-scale factual supervision. Prompt template is provided in Table 5
Additional Notes. We observe bimodal loss distribution on entity and relation tokens in the GPT- 4o-generated annotations. This is likely due to GPT-4o accessing future context during generation, which breaks the left-to-right constraint of autoregressive models. As a result, some annotations are not recoverable from preceding context alone.
The CORRECTOR helps filter out such cases removing lookup calls that are (1) not inferable from prior context, (2) overly specific or inconsistent, or (3) syntactically malformed. This filtering improves the quality of supervision provided to the final ANNOTATOR model.
Ultimately, the ANNOTATOR learns to insert lookup calls only when they are contextually grounded and likely enhance factual accuracy. This encourages retrieval-based reasoning and helps LMLM offload factual knowledge from its parameters into a structured database.
Additional details of the evaluation and ablation study for the annotation pipeline. We evalu- ated different annotator training configurations GPT-4 annotated seed evaluation set containing 100 SQuAD passages and 100 Wikipedia articles, which were cleaned to ensure reliable supervision LoR A-fAA-ffitteed for 10 epochs mixed dataset of 1k SQuAD and Ik Wikipedia examples with the intermediate filtering step. Two ablation baselines were compared: w/o data mixing, which trained only on 19k SQuAD while keeping total compute constant, and w/o filtering, which trained on the same mixed data but without the filtering stage.
signals. The ANNOTATOR
As shown in Table 6 the mixed-data and filtering design achieves the lowest losses across both domains, confirming that both steps contribute meaningfully to annotation quality.
Table 6: Loss decomposition on the original corpus and the average loss over lookup calls for different annotator training configurations.
Model
Wiki (orig.)
Wiki (lookup avg.)
SQUAD (orig.)
SQuAD (lookup avg.)
ANNOTATOR (ours)
3.33e-3
7.85e-2
2.93e-3
1.22e-1 3.67e-3
wfo data mixing
4.21e-3
1.18e-1
1.81-1
w/o filtering
3.38e-3
8.21e-2
3.3e-3
1.30e-1
A.2 MODEL ARCHITECTURY AND TRAINING DETAILS
We pretrain LMLM from scratch using GPT-2 and LLaMA2-ttyle decoder-only architectures. Each model uses its original tokenizer and vocabulary, extended with four special tokens reserved for lookup calls. This results in vocabulary size of 50,261 for GPT-2 models and 32,004 for I.aMM22 variants. Full architecture specifications, including hidden size, depth, and parameter counts, shown in TTbll 7
All models are trained for 8 epochs with a context length of 1,024 tokens using mixed-precision training. For LLAMA2-1780 1-242---66M 176M and LLAMAA-382M, we use batch size of 256 and train for 105k steps, totaling approximately 8 H100-days. Training is performed using Hugging Face Accelerate in bf16 precision. Hyperparameters such as learning rate, scheduler, and warmup steps are detailed in Table g
18

---

Published as a conference paper at ICLR 2026
Table Model architecture, vocabulary size (including 4 special tokens), and parameter counts. We report both total and non-embedding parameter counts.
Model
Hidden Size
#Layers
ileads
Vocab Size
Params (Total / Non-Embed)
GGPT2-124M
768
12
12
50,261
124.4M / 85,5M
LLAMA2-176M
512
8
8
32,004
176.4M / 160.0M
GPT2-355M
1024
24
16
50,261
354.8M / 303.4M 381.8M/ 357.3M
LLAMA2-382M
768
12
12
32,004
Table 8: Training hyperparameters. 1LAMA2-176M, ALAMA2-382Man initialized in float32 and trained with mixed precision (b/16) using Hugging Face Accelerate.
Model
Batch Size
Total Steps
LR
Scheduler
Warmup
Precision
320
66k
5.0c-4
6 (mixed)
LLAMA2-176M. LLAMA2-382M
256
105k
5.0c-4
cosine
2000
6/16 (mixed)
A.3
FORMALIZATION OF TRAINING AND EVALUATION OBJECTIVES
We denote an autoregressive language model by po. which defines a probability distribution over sequence tokens (z,,..., TT) as:
po(
II polare xee))-
Each token 2, belongs one the following categories:
Tog:: original text tokens from the raw corpus;
T Return values and db_end] are excluded from training loss objective.
Te. Tr: tokens representing entities and relation arguments within database lookup calls;
Ty: tokens corresponding to retrieved factual values (i.e., return values);
Tas special tokens used to mark database lookup segments, including:
< <|db start >: begins a lookup call;
< sep >: separates entity and relation in the query;
db retriveee
signals the insertion point for the returned value;
<I dblend >: marks the end of the lookup block.
Here an example using background color to highlight different token categories:
Tap:
Napolcon was born on < <bd_start/>> Napolcon < </eop>> Birth.Date <lcb_retrievel> August 15, 1769 </db_end|> > August 15, 1769.
Timi::
Napoleon born </db.start|> Napoleon 
 Birth.Date </db.retrieve August 15, 1769 < <idboond]>
August 15, 1769.
Training Loss. The training objective excludes supervision over return values and the closing marker <| db_endd > to prevent memorization of factual knowledge:
(())=
Σ
log (ree): where Tiain {1 $ TuU dbbend|>>))
(2)
1E T
Evaluation Metrics. We report both perplexity and negative log-likelihood (NLL), computed over different token subsets depending the evaluation setting:
19

---

Published as a conference paper at ICLR 2026
Static & Dynamic Perplexity: Tokens corresponding lookup calls and return values excluded:
PPLatatiidyynmmic = exp
log (0 xx<1)
(Torg
Tors
Normalized Perplexity: This metric fairly compares generation likelihood by excluding retrieved factual values from the loss, but normalizing by the length of the original (fully reconstructed) text. Specifically:
PPLmom exp
log po xEe)
[Torg
LET
Negative Log-L.kkelihodd (NLL): Matches the training loss computation:
NLL(x)=-
Σ TT
log xee).
A.4 DATABASE AND RETRIEVAL. SETTING
We build our database by annotating the pre-training data, obtaining 54.6M knowledge triplets consisting of 9.5M unique entities, 8.5M relationships and 16.2M unique values. For retrieval, employ a fuzzy matching mechanism based on the cosine similarity of sentence embeddings from ALL-MINILMM-L--
As shown in Figure 9 specifically, given lookup call, compute its embedding and compare it with the embeddings of stored triplets in our database. If the highest similarity score is below a threshold of 0.6, we return unknown indicate that sufficiently similar match was found. Alternatively, we implement prefix-tree constrained generation, ensuring that lookup calls remain covered by the structured knowledge representations. See detailed discussion in Appendix D.3
Sentence Embedding Space
Prefix- Tree Decoding
rother
Mozert Birth Date
Napoleon, Birth Date
irth
Napol eon, Mother
START
Napoleon was born on (Napoleo,, Birthday)
Napol eon was born on (Napoleon, Birth Date)
Figure 9: Unconstrained vs. Constrained Query Generation.
Mini 1.6-v2
20

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL SETUP
B.1 EVALUATION BENCHMARKS
Perplexity. We evaluate language modeling perplexity on held-out Wikipedia test set consisting of 1,000 passages ("245k tokens). We use the same tokenizer the original model (either GPT2 or LLaMA2) and apply Hugging Face's 's Trainer with sequence packing enabled. For LMLM, reference completions are annotated using ANNOTATOR. lookup arguments, but we force lookup calls to occur at the same positions as in the reference. The exact formulation of perplexity provided Appendix A.3
the dynamic setting, the model generates its own
FactScore. We evaluate factual precision using FACTSCORE (Min et al., 2023), a benchmark for open-ended biography generation. Given generated text, FactScore extracts set of atomic facts and computes the proportion that is supported by a trusted knowledge source. We use the first 100 biography queries provided in the benchmark. All models generate outputs using greedy decoding (maximum length = 256 tokens; repetition penalty = 1.2). Factuality is validated using retrieval-augmmented prompting with ChatGPT, following the official evaluation protocol
For LMLM, which is not instruction-tuned, we use a fixed prompt template to elicit biography completions: "Tell me bio of <name>. <name> is" This prompt is applied consis- tently across all samples. To encourage structured queries during generation, we apply a logit bias to four special tokens in the vocabulary: dbb_sta |>, < sepi> >>, db_ret petrieeee eeve 1>, and < db_end >>, with respective bias values of 5.0, 2.0, 2.0, and 2.0. Retrieval is performed using fuzzy matching with a cosine similarity threshold of 0.6. If relevant triplet is found, the model continues generation using the plain text string unknown as fallback. This behavior is untrained and left for future work improve robustness retrieval failures.
T-REx, We adapt the T-REx subset from LAMA (Petroni et al. 2019) for autoregressive models by filtering examples in which the masked entity does not appear in the final position. This yields 11,615 left-to-right compatible examples, following [Schick al. (023). Each input consists of factual statement, such as: "Jaideep Sahni (born 1968) is The model expected to complete the statement with a single token (e.g., actor). We evaluate using two metrics: Exact Match, which checks whether the reference answer appears among the first five generated content words, and Precision 1, which checks whether the first generated content token matches the reference.
Indian [MASK]"
All models generate outputs using greedy decoding, and both metrics are computed after post- processing to remove lookup calls. For LMLM, enforce database lookup call the masked position using fuzzy matching with a similarity threshold of 0.6. If no match is found, the model continues with standard decoding without triggering structured lookup call, the target fact may belong to common knowledge not covered by the database.
We evaluate on the long-tail subset of PopQA, which contains 1,399 queries about rare entities (fewer than 100 monthly Wikipedia page views), following/Asai al. (2023). Performance is measured by Exact Match (EM), i.c., whether the gold answer appears in the model output. To enable fair comparison with small pre-trained models that are not instruction-tuned. we convert the QA format into a reduces dependence on instruction- followiin ability and allows answers to be appended directly. All results reported in the paper use this rewritten format. For example: Original: What is Ufa the capital of? Rewritten: What is Ufa the capital of? Ufa is the capital of.
PopQA.
task by prompting GPT-4 to rewrite each query. This
B.2 MACHINE UNLEARNING SETTING
TOFU. The TOFU benchmark (Maini al. 2024) evaluates unlearning efficacy in privacy-sensitive scenarios, aiming to selectively remove specific subset of information (the Forget Ser) from a model (the Full Model) while maintaining performance on retained information (the Retain Set) and general
21

---

Published as a conference paper at ICLR 2026
model capabilities. The benchmark's 's primary goal is to ensure the resulting Unlearned Model is statistically indistinguishable from a model trained exclusively on the Retain Set (the Retain Model).
The TOFU benchmark comprises 200 synthetic author profiles, each associated with 20 QA pairs. Performance is assessed using two primary metrics:
Model utility: An average of three metrics- ROUGE (answer quality), Answer Probability (the likelihood correct answers), and Truth Ratio (the likelihood assigned to correct answers over (istractors)-evalleee evaluated on the Retain Set, Real Author Set, and World Facts Set.
Forget quality: The p-value from a statistical test comparing the unlearned model to the correspond- ing retain model, quantifying the effectiveness of knowledge removal.
Our implementation directly builds upon the official TOFU repository
We use LLAMA-3 3.2-- 1..2-1B-INSSTT I--INSTRUCT as the base model and compare against NPO (Zhang et al. 2024b), state-of- of-the-art method for unlearning. We test on forget 5% setting. For LMLM, unlearning implemented simply by removing relevant entries corresponding to the Forget Set from the external database.
We adopt LLAMA-2 3..--112211 B I--INTTRCCT as the backbone for two reasons: (1) TOFU is designed for instruction models, and the official benchmark reports all baselines using instruction-tuned backbones; and (2) TOFU is synthetic, so using an instruction model that has never seen the underlying knowledge provides a fair initialization to show that LMLM prevents knowledge interalization by design. Additional results pretrained LMLM is in Appendix C.4
NPO baseline. Negative Preference Optimization (NPO) (Zhang et al. 2024b) a state-of-the-ar method for selective knowledge removal, especially effective for unlearning large portions (50%-90%) of training data. Unlike traditional gradient- unlearning methods, which often degrade a model's general performance and struggle to forget only 10% of training data, NPO explicitly guides the model away from undesired (negative) samples, maintaining stable training dynamics and preventing catastrophic performance degradation. Further details can be found in the original paper (Zhang et al., 2024b). We follow the official TOFU implementation, running unlearning fine-tuning three times with different random seeds and reporting the mean and variance in Figure 6
LMLM Implementation of TOFU Evaluation. To evaluate unlearning effectiveness, we assume the synthetic knowledge used in TOFU is fully represented in external database. We annotate the complete TOFU dataset (4k synthetic QA pairs) using GP,-4o, subsequently building our database from these annotations.
Since LMLM is applied to pre-existing models, we introduce an additional step to ensure the model can utilize the lookup mechanism effectively. Specifically, we first perform warm-up training stage on annotated Wikipedia data, followed by fine-tuning on the annotated TOFU training set, using the same hyperparameters as the baseline models (see Table(9).
For ROUGE evaluations, generated answers are peseeprcessed to remove structured lookup tokens before computing the scores. For likelihood- based metries (Answer Probability and Truth Ratio), we evaluate the model's probabilities only on the annotated training input segments (T _train,, as defined in Appendix [A.3). Thus, ROUGE scores remain directly comparable across methods, but likelihood-based metries are not directly comparable due to differences between raw and annotated reference answers.
Interpreting TOFU Results. When interpreting TOFU results, the key indicator of successful unlearning is how closely the unlearned model's performance matches that of the retain model. Ideally, the two models should be indistinguishable, indicated by forget quality p-value above 0.05. In practice, this means the unlearned model should achieve forget quality above 0.05 while maintaining consistent model utility, and its ROUGE scores on both the Forget Set and Retain Set should closely resemble those of the retain model. Importantly, lower ROUGE scores on the Forget Set do not necessarily imply better unlearning, these reductions could also result from general performance degradation.
22

---

Published as a conference paper at ICLR 2026
B.3 EXPERIMENTAL SETTING OF FURTHER ANALYSIS
Selective Knowledge Offloading. We conduct preliminary study on the trade-off between storing knowledge in model parameters and offloading it into the database. To enable selecfive extemalizattin, we revert only a subset of annotations based on model learning difficulty.
Specifically, we first train LMLM model with the original annotated data described in Sec. 4 and a STANDARD model with the unannotated data for one epoch. We measure the difference of the language modeling loss for the 5 tokens following each lookup for the LMLM and the corresponding tokens in the clean document for the STANDARD model. A larger gap suggests that STANDARD struggles to memorize the fact- teetitllllllly long-tai or highly specific knowledge -making a strong candidate for externalization. A smaller gap indicates that (i) the fact can be easily memorized by STANDARD, (ii) contextual hints make it trivial, or (iii) database lookups LMLM are noisy or less useful. We calculate this loss difference across the full Wikipedia dataset, and include the distribution (Figure 12 and random qualitative examples from each quantile bucket for illustration (Table 19).
Based on this criterion, we then pretrain LLMs with varying knowledge offloading ratios [0%, 25% 50%, 75%, 90%, 100%]. Here, 0% corresponds to STANDARD (fully parametric) and 100% to LMLM (fully externalized), Offloading is applied progressively, starting with triplets that show the largest loss differences. The corresponding loss thresholds are: 3.51 for 10% offloading, 2.24 for 25%, 1.25 for 50%, 0.64 for 75%, and 0. 13 for 90%.
B.4 EXPERIMENTAL SETTINGS BY FIGURE
We detail the experimental configurations corresponding to each figure in the main paper:
(Left) We eval on held-out wikipedia validaition set of 100 passages ("21 tokens) every 1000 steps during pretraining. (Middle) The detailed results for FactScore and NLU are in Table 2 and 14 (Right) The backbone model is LLAMA-3 2- I-ISSRRUUCTT. We implement LMLMM evaluation based on TOFU official repo It is forget 5% setting. The detailed results for baselines methods copied from TOFU repo
Figure 4
Figure 6 For NPO baselines, we follow the official TOFU implementation, running unlearning tunintunng five times with different random seeds (0, 42, 420, 69, 4497) and reporting the mean and variance throughout training. The ideal performance of the retain model is indicated with marker. For LMLM, which does not require training to unlearn, we show only the pre- and post-unlearning results. Details on how to interpret the results in Appendix [B.2]
Figure 7 We use the TOFU synthetic training set to compare training objectives by tracking the loss on return value tokens- -the factual spans intended to be retrieved via lookup. Models are evaluated every 125 steps during training.
Table 9: Training hyperparameters used in different experimental settings. (Full set: 4k QA pairs, Retain set: 3.8k QA pairs)
Setting
eaann ngurata
warmup_staps
numutrra nuepochs
batch_s!.e
dataset
Finetune (Standard)
1o-3
0.2
5
32
TOFU traiusct
Warmup (LMLM)
5o-5
0.25
Annotated Wikipedia P.8k chonk) Annolated TOFU trainset
Finetune (LMLM)
5e-5
0.2
5
32
23

---

Published as a conference paper at ICLR 2026
C DETAILED RESULTS
C.I ADDITIONAL NOTES ON RAG.
We include standard retrieval- auggeeedd generation (RAG) baseline as point of reference. following Lewis et al. (2021). The retriever uses BM25 (via the 3M25Retriever from FlashR ) to select the top-4 relevant chunks from English Wikipedia, segmented into 100-word passages. These passages are prepended to the model input using the prompt format: "Answer the question or complete the prompt based on the given document. The following are given documents: (REIRIEVED DOCUMENTS]] \n [OSER_QUERY]" (see Ta- ble 23 Retrieval is performed at inference time only, without fine-tuning. We evaluate RAG using OPENAI/GPT2-355M to match the scale of LMLMM.
As an additional note on Table 3 RAG performs reasonably well on FactScore, and we expect further improvements with larger, instructionttund models, smaller models may struggle effectively use retrieved context. RAG also achieves high scores on T-REx, though this may reflect the benchmark's overlap with Wikipedia, where many completions retrieved verbatim. We include RAG results provide a broader empirical context.
While both RAG and LMLM access external knowledge, they differ fundamentally. RAG retrieves unstructured text from Wikipedia and relies on the model to extract relevant content. In contrast, LMLM learns to issue explicit, structured lookups only when needed, interleaving retrieval with generation at the entity level. This enables more precise and easily verifiable access to factual knowledge. The two approaches are complementary RAG could be applied on top LMLM for potential additional gains.
C.2 FACTUAL DEGRADATION WHEN FORCING INTERNAL RECALL.
Table 10 shows the full results when database access is disabled, forcing models to rely solely on internal parameters. Across all variants, we observe consistent drop in factual precision, often below the STANDARD baseline. This degradation supports our main claim: LMLM does not memorize factual answers but retrieves them externally. These results highlight that what the model knows and forgets is determined by the database, enabling precise and direct control through simple edits.
Table 10: Impact of database access factual precision. Disabling access leads to performance drops in both FACTSCORE and T-REx, contirming that LMLM relies on retrieval from external database rather than memorization.
Metries
Model
Model Type
Database
FActScore (%)
T-REx Exact Match (%)
STANDARD
10.7
41.2
GPT2-124M
LMLM
14.9
32.0
LMLM
V
20.6
54.6
STANDARD
10.1
46.3
LLAMA2-176M
LMLM
11.3 19.3
34.9
LMLM
30.6
54.1
STANDARD
14.4
44,9
GPT2-355M
LMLM
10.4 18.5
36.4 22.3
23.9
58.7
STANDARD
14.0
$2.0
LLAMA2-382M
LMLM
12.8 19.1
38.5-19.6
LMLM
31.9
58.1
C.3 ADDITIONAL LOSS CURVES.
Figure 10 shows the full training loss curves different token types in the TOFU synthetic trainset. While the main paper focuses on return value tokens, we include loss on Torg. entity, and relationship
doca/
basel bs
24

---

Published as a conference paper at ICLR 2026
tokens here for completeness. We observe significant difference between training objectives these tokens.
3.0
1.75
Com
125-
1.0
2.0
0.75
0.00
200
400
600
400
600
600
200
400
Global Step
Step
Step
Global Step
LMLM
Regular
Figure 10: Training loss on Torg tokens, entity tokens, relationship tokens, and return value tokens.
C.4 ADDITIONAL TOFU RESULTS ON PRETRAINED LMLM.
Evaluating pretrained LMLM models TOFU is also valuable. We now include results using LLAMA2-382M. which was trained from scratch under the LMLM paradigm. As shown below, both LMLM variants achieve effective unlearning (p-value 0.05) without degrading model utility, consistent with the results in Section 4
Table 11: TOFU unlearning results. "Model utility" measures preservation of general model behavior; "forget quality" measured via two-sample test (higher p-value indicates better unlearning). Baseline numbers use Llama-3.2-1 B---BB-Inett from the official TOFU repository.
Method
Model Utility
Forget Quality (p-value
Finetuned
0.60
1.33c-13
Retain (Ideal)
0.60
1,00
GradAscent
0.00
1 1.94e- 119
GradDiff
0.53
1 1.94e-119
IdkDPO
0.07
1.12c-05
NPO
0.45
0.14
SimNPO
0,58
5.01e-100 100
RMU
0.58
4.87e-10
L.LM (LLAMA-333 11--Z-ITI-TT
0.55
0.31
LMLM (LLAMA2-382M)
0.43
C.5 COMPARISON WITH MEMORY-BASED RELATED WORK.
Table 12: Comparison of LMLM against prior memory- based work on EM (%). All baseline numbers reported from their original papers. "Params" refers to non-embedding model parameters. "External Memory" summarizes the format or corpus scale used at inference.
Model
Params
External Memory
T-REs EM (%)
CoLAKE
0.11B
/ 5M Wikidata Triples
28.8
K-Adapter
Adapter Parameters
29.1
BERT- BERT-KNN
0.34B
900M Instance Datastore
38.7
EaE
0.11B
V IM Entity Memory
38.6
1.7M Wikidata Triples
44.2
Toolformer
6.0B
API Access Wikipedia
53.5
LMLM
0.16B
/ 54.6M Triples
54.1
LMLM
0.36B
54.6M Triples
58.1
We also compare LMLM with prior memory-augment language models (Table 13) Several approaches externalize factual knowledge by attaching explicit memory modules. Fact-Injected Lan- guage Model (FILM) (Verga et al. 202 augments LMs with neuro-symbolic memory of entity-fact pairs. CoLAKE (Sun et al.; 2020) jointly pretrains language and knowledge representations through a modified Transformer encoder. BERT-KNN (Kassner & Schütze 2020) and EaE (Févry al. 2020) use kNN-style retrieval or entity-specific memories to handle rare entities. K-Adapter (Wang et al. 2020) injects structured knowledge into PLMS via adapter modules. KBLaM (Wang et al. 2024
25

---

Published as a conference paper at ICLR 2026
Table 13: Comprehensive comparison of related methods. v supported. X not supported, and partial support (e.g., edit without unlearning, or not explicitly evaluated in prior work).
Arrkieetare
Kowelelg-Stnng
Perfereees
Internal
Extemall
Intogratka
PLL
Pacozdlty
Unkenarg
Proege perpora
MB1100
Cros-tte chane clanks
Sedl acticeed prarrilzzzee Lecahoc nepaehoiiee
A434
(pasaggs
Depunds
[MLM
Dopoods
KB
Explick lookap
Doceaple lada Erven wetaho
K-bigae Wen.a 10.0
Ex alypes
Adlgte legess
Iejec KBl adipers
RAC 2222
Retiore prongt Thid call Welghe adlling
leperin lickelty
Dac-oedy
APL
sla Fileet Batiads
MBMIT
Dec-onyy
Dipoodd
INN ward prrs zaups Reaiove
CLLAKE
2033
Exceeds
SG Thers K.B
Join ML Moner KG woch
Join laapmag kanwdadge QA king form
BIRESS
Fax
HL3
Rnponds
Baliyslac Marn poots kniep.
Neanooymbhhe kih was fact lajaako traggaa PTtmine and anie eneway High- Hilh-denuppp mense.
Meed 1/ Mittmneee 100
Decords
Tattorted trihtes MLPPosudd
Road leni NEP mutaa 180t serienal
Deconds
augments knowledge in the form of continuous key-value vectors as knowledge tokens and integrates them into pre-trained LLMs with modified rectangular attention structure. MemLLM (Modarressi et al. 2025) proposes an explicit read-write memory, enabling LLM to extract knowledge into triplets, stores and later recalls knowledge for specific tasks. Concurrently, MLP-Memory (Wei et al.) 2025) distills a kNN retriever into a lightweight MLP without accessing documents, which improves Tactuality and inference throughput.
While these methods improve factuality by coupling models with external memories, they generally do not address controllable unlcarning, which distinguishes LMLM from this line of work.
Table 12 shows that LMLM achieves consistently higher precision (Exact Match) on T-REx compared to prior memory-augmented and tool-augmented models, while using fewer parameters and without depending on large external tool APIs. These results highlight both the modeling advantages of our approach and its empirical gains over existing memory-based baselines.
Additionally, Memory (Yang et al. 2024) pretrains models to write and read sparse attention key-values as explicit memories, reducing the burden of model parameters to memorize specific knowledge. Memory shares the similar motivation with LMLM that both pretrain models from scratch to access external memory to reduce parametric memorization. However, LMLM benefits from an external natural-language knowledge database, which is easy to update, verify and unlearn. In addition, LMLM offers a conceptually simple training recipe that can be integrated by finetuning existing LMs on wowwlede-inttnsive domains.
26

---

Published as a conference paper at ICLR 2026
D FURTHER ANALYSIS
D.1 DOES LMLM AFFECT LANGUAGE UNDERSTANDING?
Beyond the promising results, is important verify that our approach does compromise the general capabilities of pretrained language models. To assess this, we follow Penedo et al. (2024) and evaluate on a set of "high-signal" Natural Language Understanding (NLU) benchmarks (as shown in Table [4p. Given our focus on smaller models, we exclude benchmarks where both GPT-2 and similarly sized models fail rise above the noise floor (Du et al. 2025). This leaves us with the following benchmarks: Commonsense QA (Talmor et al., 2019). HellaSwag (Zellers et al. 2019). SIQA (Sap et al. 2019), and ARC Easy (Clark et al. 2018). Implementation details can be found in lighteva
PIQA (Bisk et al. 2019). SIQA
is important to mention that STANDARD and LMLM are pretrained solely on the Wikipedia dataset, which makes certain benchmarks not applicable for comparison with off-the-shell models. However, the chosen NLU benchmarks still effectively address concerns about the potential negative impact on the models' general language understanding ability introducing by removing factual knowledge.
Table 14: Evaluation of NLU benchmarks using normalized accuracy metrics, demonstrating that separating factual knowledge during pretraining does not compromise overall model performance. A caveat is that small-scale settings constrain the scope of what conclude about general NLU capabilities.
Metries
Model
Model Type
CSQA
HellaSwag
PIQA
SIQA
ARC Easy
All
Random Chance
20.0
25.0
50.0
33.3
25.0
30.7
OPENAUGPT2-I22M*
29.8
62.5
40.7
39.5
40.6
GPT2-124M
STANDARD
26.5
26.4
55.3
39.2
34.2
36.3
LMLM
27.9
26.8
55.1
39.9
35.0
37.0
LLAMA2-176M
STANDARD
26.6
27.0
55.4
40.4
33.9
36.7
LMLM
26.8
28.2
55.2
40.2
35.8
37.2
OPENADGPT22335MC
32.6
37.1
66.4
41.2
43.6
44.2
GPT2-355M
STANDARD
28.1
27.0
55.7
40.0
37.8
37.7
LMLM
27.7
56.8
37.7
STANDARD
27.8
28.8
55.2
41.0
35.8
37.7
LLAMA2-382M
LMLM
26.9
29.1
56.1
40.8
35.9
37.8
Models marked with asterisk (*) soff-theshhe shell models with additional training.
D.2 ENTITY FREQUENCY ANALYSIS IN THE KNOWLEDGE DATABASE
400
Entity Count
300
200
100
mm
10
10
102
103
104
105
106
Wikipedia Frequency
Figure 11: Frequency distribution of database entities matched to Wikipedia entries. The database spans wide range of entity frequencies, including many long-tail cases.
htttss///ggthbbbb.coo/onnnnnnne igheval
27

---

Published as a conference paper at ICLR 2026
Table 15: Comparison of Fuzzy Match and Prefix-Tres Tree Decoding. Fuzzy matching, used by default in LMLM, offers higher flexibility; prefix-tree decoding is included for ablation.
Model
Model Type
Decoding
FActScore (%)
FActScure w/o len. penally
Facts/ Response
STANDARD
10.1
11.8
23.7
LLAMA2-176M
LMLM
Pretix-tree
23,0 7.6
23,9
8.6
34,6
LMLM
Fuzzy Match
30.6
32.4
31.4
STANDARD
14.0
16.6
32.4
LLAMA2-382M
LMLM
Prefix-tree
23.5
31.3-1.4
28. 2-.1
LMLM
Fuzzy Match
31.9
32.7
33.7
30%
95%
Quantlles
Density
0.2
Per Knowledge Triplet Loss Difference
Figure 12: Distribution of per-triplet loss differences with quantile buckets used to select triplets for externalization.
We analyze the prevalence of knowledge triplets in our database by estimating how often each entity appears in Wikipedia. Using entity frequency statisties from Kandpal et al. (2023). apply fuzzy string matching (threshold 70) align database entities with Wikipedia entries. Figure shows the distribution of matched entity frequencies. The results indicate that our database spans both common and long-tal tail knowledge, with a substantial portion entities being less frequent in the overall training corpus.
D.3 ABLATION: UNCONSTRAINED VS. CONSTRAINED QUERY GENERATION
We compare fuzzy matching with prefix-tree constrained decoding for generating (entity, re relation) queries. In the unconstrained setting, model freely generates queries, which are then matched against the database using cosine similarity over sentence embeddings. This approach provides flexibility but may result in syntactically invalid or ambiguous queries.
In contrast, prefix-tree decoding restricts generation to valid entries encoded in trie structure, ensuring syntactic correctness and reducing hallucinations. is also compatible with beam search and nucleus sampling, allowing the model to explore multiple valid paths in the induced knowledge graph. See Appendix A.4 for more details.
As shown in Table 15 fuzzy matching consistently outperforms prefix-tree decoding in setting, and is therefore used all reported experiments. While prefix-tree decoding offers stronger structural guarantees, find its diversity be overly constrained by relatively small database. As both the database and model scale, we expect the benefits of structured decoding to become more pronounced.
D.4 ADDITIONAL RESULTS ON KNOWLEDGE-EDIIN
BENCHMARKS
To further demonstrate that LMLM can edit facts through database-leved updates alone, we evaluated it the multihop counterfactual subset of MOUAKE-REMASTEREE (Zhong al.) 2024a), the REMASTERED audited version of MQUAKE (Zhong et al. 2024b). We use the full CF-6334 subset, randomly split into train set (5334 examples) and a 1 edit eval set.
28

---

Published as a conference paper at ICLR 2026
We first instruction-tuna the pretrained LMLM on the train split with the original database for epochs, so that the pretrained model leams the QA format and uses a chain of lookup calls to answer multi-hop questions. Then evaluate edit accuracy the 1k held-out examples using the edited database. For this rebuttal experiment, we follow /Zhong et al. (2024) and use the golden ledge graph provided by the benchmark as the external datahase. Although this is an upper-bound setting, it is consistent with prior knowledge-cditing methods that assume access to the original and edited knowledge triplets.
As shown in Table T6 LMLM outperforms all knowledge-editing baselines, confirming that factual updates can be performed through LMLM database edits alone, without modifying any model parameters.
Table 16: Edit accuracy on the MQuAKF-Reemaserred CF-6334 benchmark. Baseline results for are copied from Zhong al. (2024a) (1000-edit setting) and compared with LMLM
using a LLAMA2-382M base model.
Method
Edit Accuracy 1
MeLLo (Zhong al. 2024b
19.27
GWalk Zhrong at 20248
61.79
LMLM (Ours)
69.60
D.5 ANNOTATION, TRAINING, AND INFERENCE COST
To study the trade-off' between LMLM's advantages and computational cost, we report data annotation, pre-training, and inference costs in Table 17 LMLM requires one-time data annotation step to build the database and insert lookup calls, corresponding to single inference pass of the annotator model over the pretraining corpus (5 10 FLOPs). During pretraining, compute increases moderately (1.72x FLOPs, 1.82x H100-days), mainly due to token overhead from inlined lookup calls. At inference time, generating and executing lookup calls introduces additional latency (2.21 x) which we have not yet optimized. This overhead is similar to that observed in current agentic LLMSs that call external tools and may be reduced through system-level optimizations. Overall, the main overhead lies in pretraining compute.
In Figure [8] we extend the trade-off analysis between storing knowledge in parameters versus offloading it to the external database. We now include the corresponding compute budgets. As shown in Table 18] increasing the offloading ratio consistently improves language modeling (lower NLL) and factual precision with only modest increase in compute.
Table 17: Compute comparison between standard pretraining and LMLM, including database construc- tion, pretraining cost, and inference latency. Database construction is one-time cost. Measurements knowledge-intensive corpus and represent worst-case estimate.
Method Standard Pretzaining LMLM-382M (Ouns)
Database Construction (FL0P))
Training Cost (H100-days)
Training Cost (FLOPS)
Inferrecee Latency (ms/token)
NIA
(1.0x)
(1.0k)
5,51 10 (onc-time)
(x-1.8x)
6.32 x 10 (x-172x)
21.08 (-2.21x)
D.6 ANALYSIS OF LOOKUP FAILURE CASES
We first conducted additional analyses to understand how lookup failures occur and how often they happen. From 50 manually inspected T-REx examples, 64% were exact matches. The remaining failures arose from ambiguous entities/relations (6%), missing database entries (10%). multiple matches (2%), or insufficient context in the T-REx completion setting (18%). Overall, the lookup performs reliably, with the remaining errors mostly coming from limited coverage of the database.
Current fallback behavior either returns the top-1 match or outputs "unknown", and the model is not yet trained to handle these cases robustly. To improve robustness, future work will explore RL training to enable the model to reason, verify, and backtrack when the queried external knowledge is unavailable.
29

---

Published as a conference paper at ICLR 2026
Table 18: Extended trade-off analysis (Figure increasing the knowledge offload ratio reduces NLL with modest additional training cost.
Knowledge Offload Ratio Training GPU Hours (H100-days) J
NLL
FactScore
0% (Standard LLM)
2--3
8,50
10.1
25%
2.48
7.71
17.2
50%
7.43
75%
4.10
7.19
27.1
90%
4.31
6.98
29.2
100% (LMLM))
~5-6
6.70
30.6
Those training runs launched slightly different GPU configurations, their training GPU hours not directly comparable.
D.7 QUALITATIVE RESULTS
To complement our quantitative results, we present qualitative examples comparing outputs from LMLM, STANDARD, and off-the-shelf models. As shown Table 20 21 and 22 LMLM produces concise, factually grounded responses by leveraging external knowledge, whereas standard and off-the-shelf models often include hallucinated content. These examples illustrate how knowledge offloading enables LMLM to maintain factual precision, further supporting our central finding (Figure that LMLMS achieve more with less by scaling efficiently through externalized knowledge.
Table 19: Qualitative examples of triplets selected for externalization at different quantile buckets of per-triplet loss difference. Higher quantiles correspond to facts that are harder for STANDARD to memorize (long-tail or specific knowledge). while lower quantiles correspond to easier or contextually obvious facts. The triplet of interest in each example is highlighted in color.
Delta Loss
Quantile
Example
3.31
75%-100%
Joseph Charles "Joe" Avellone III. M.D. (bom <l start Joseph Charles Avellone III< sep Birth Date<) abbretriews September
29, / cb.end September 29, 1948) dbbrrerieeee Charles Avellon TTT<| Nationality haballtyy/dbbeetrrrree
<|db.start Joseph
American<|| [rriccn<<dbbend]]] dbbendd>
American
<ldb.start |> Joseph Charles
Avellone IIIK| sep Occupation<] db.retriever |> medical doctor, businessman, politicians| politician from <| start Joseph Charles Avellone Iil<|aep Stace Residences rdbbrerrieee
l dbbendd]> medical doctor, businessman, and
Massachusetts<] db..useeett/////dd db_end|> Mas- sachusetts...Ho then worked as CEO of biomedical company <|db.start Joseph Charles Ave lone TTT<] sep |>
CRO Of< | ret Veritas
Medici nex dbjend > Veritas Medicine
3,23
75%-100%
Elsa van Her thesis, supervised by <ldb start|> Elsa van Blea<laep
Taesis Supervisors hbbrs//dbbbbrrrreeee rieee Donald Menzel< db.end
Donald Menzel, discussed the Stark effect in the Balmer lines of early type stars.
T.41
50%-75%
albicaudatas sepl> Species Type< db retrieve > sea snailk db end 1> species of sea snail, ma- start >> Kermia albi caudata< sep rine gastropod mollusk in the < <l </cbbstart/
Kermia albicaudata is [db_start]> Kermia
Family< [mmly<|db_retriere [db.retrieve]]
Raphitomidase /bbe<dddb family Raphitomidae.... <|db.start 1> Kermia iistributions aannidonn/ddeeeeeeeeee ave Persi ian Gul db] Persian Gulf'
Distribution.
This species occurs in
al bi caudsta<| sep|> Geographica
0.82
25%-550%
Havering Country Park... The land was purchased by the < <|db_start. start > Havering Country Parks sepi> Purchased By< db.retrieve |> Greater London Council4 ddd///dbedddd Greater London Council and opened to the public in < <|db-startis laavering Country Parks isep>> Opening 1975K] 19754/dbuendl] db_end]>> 1975, with ownership transfer- ring to <ldbbstart Havering Country Pazk<] Ownership Tzansferred Tok| rbcerriiee |> llavering Council< |nd_end|> Haver- ing Council in </db_start/> Havering Counci1<|sepp Ownership Year<Idb retrievel> 1986<1 ab end 1986.
0.003
096-25%
Kagoshima District, Kagoshima is district located in <ldb_start Kagoshima District, Kagoshima< | Located In Prefecture< dd zetrievel Kagoshima Prefecture, Japank áb end
Kagoshima Prefecture, Japan
30

---

Published as a conference paper at ICLR 2026
Table 20: Qualitative examples of generated outputs from models with STANDARD and LMLM
settings.
Model
Generated Output
LLAMA2 176M-
Positions| oollooo<dbbbbbeee bee.ttoooodd.rerieeeee. dbbrririeeee the </bbsttart 1> Ko Itakura< sep center center-back, defens midfie ders db ]nd center-back and defensive midfielder for </ start Ko itakurak /sppl> Current Teams abbretrieve] Japan
Ko
Trakura
LMLM
national teans db_end]>> Japan's senior squad in the < <ldb_start |> Ko Itakurak sep|> League participations (onn//dbrretttteeeee db_retrieve]]> 2022 FTFA World Cup<I 1> 2022 FIFA World Cup. He was horn on <|db.start |> Ko itakura<| sep sirth Datex dbbretrieee]]> January 27, 1997</db.end January 27, 1997.
On <|db_start Ko Itakura<|sepl>> Joined Club<l db retrieve Kawasaki Frontale< db end|> >> July 30. 2015. announced that Itakura joined JFL club Kawasaki Frontule from their youth system. In < </dbsstart/s Ko Itakura< sep Moved Tos |db_retrievee Groningen [rooningen<ddbeed]]] Ib August 2016 he moved to Groningen.
Career.
On <I start >> Ko Itakura<| sep Debut Date< db retrieve
June 17, 2019< db.end <|db.start > Ko Itakura<] sep Debut Opponent <
June 17. 2019, Itakura debuted against
Uruguay< dbb ab_ >> Uruguay in an international friendly match. After playing only one game for Groningen during his debut season, Itakura left the club at the end of the year.
LLAMA2-176M-
Ko Itakura is the name given to an ancient Japanese settlement in what is now part of Shimane Prefecture, Japan. The site was discovered by archaeologists from Kyoto University and has been excavated since 1978.
STANDARD
The first written reference to this place appears "Nihon Kōki". which dates to the seems that there were already settlements at this
year 604 AD (Kumano no Kan'l 'i), location before then. the early Heian period, the area around modern-day Shimonoseki City became known as meaning "the land between rivers". During the Kamakura period, the region came under control of the Ashikaga clan who built their capital Mount Hiei near present-day Naka Ward. After the fall of the Ashikaga shogunate, they lost control over much of eastem Japan, including the area surrounding modern-day Shimane prefecture. However, after the Battle of Sckigahara, Tokugawa Ieyasu established himself as lord of most of northern Japan with his domain centered on Matsue Castle. This castle later fell into disrepair during the Edo Period, and
31

---

Published as a conference paper at ICLR 2026
Table 21: Qualitative examples of generated outputs from models with STANDARD and LMLM
settings.
Model
Generated Output
LLAMA2-382M-
the <|db_start Ko. Itakuras [taku Positions db retrievel> center-back, defensive midfielder< db__nd]> center-back and defensive midfielder for <|db.start }> Ko stakura<|sepl2 | [b Japan
Ko
Itakura
LMLM
nationall [eam<|db_end/le Japan's national team. He born <| start |> itakura<| sirthplace< retrieve
Yokohama< dbbend Yokohama <|db.start Ke Itakura<| sepl Birth Date< dbb.dtrievee
27th January 1997. His father is from <|db_start |> Ke Itakurs<|
January 27, 1997</db.ems
Japanese rather's Origin< dbb retrieve Japanese< / end Mother's and his mother is from <|cb.start J> Ko Itakura<| trigin</db.ree21
> Japanese<| db_end| > Japanese. When he was young, he played haseball but after watching an exhibition match against a pro- fessional baseball club, he decided to become a footballer. In <ldb s Ko Itakuras sepl> Joined Club Year< db_ret rettreetri rieve |> Kawasaki Frontales I [db_nnd 2013, he joined JI League side Kawasaki Frontale. However, he could not play many matches behind Shusaku Nishikawa until <ldb.start
Itakura< sepp>> First Match as Starter<| db
debuted retrieve
Uruguays dblend |> September when
right back against
Uruguay. After that, he became regular player under manager <| Itakurak seepl> Manager Under Whom Became Regular
Players| prber///bbeee cb.reszieve On <|db.start Ko Itakuras sep Debut Date< rdb_retrien
Japan national teams dbbend|> Shinji Ono.
June 17, 2019< db_end]>> 17 June 2019, he debuted with Japan national team against Chile during the <|db.start] |> Ko Itakura< sepl>> Competition Debut< dbb Career statistics. "Updated to end of 2018 season". National team career. In August 2016, Itakura was elected to the </db.start Ko [takura<] Selections db retrevve > Japan U-20 national teamk | db db-end|>> Japan U-20 national team for the < </dbbstart|s 1> Japan U-20 national teank sep Tou rnament Partici pattiiiiiiiiiiii oon< /db_retrieeee
Uruguay<| dio dio.end > 2019 Copa América.
U U-23
2017
u-20 World Cup db lnd 2017 U-20 World Cup. At this toumament, he played all matches as back three back defense. May 2019, was selected for the db-start 1> Ko Itakuras | Senior Squad Select ioniloonseee db retriveels
LLAMA2-382M STANDARD
Ko Itakura the first album by Japanese singer-songwriter Koda Kumi, released November 19, 2003 under Sony Music Japan label. The album was reissued in March 2004 with two bonus tracks: "Kimi Uta" and "Tell Me A Bio".
32

---

Published as a conference paper at ICLR 2026
Table 22: Qualitative examples of generated outputs from different off-the-shelf models.
Model OPENAI/GP22-12MM
Generated Output
Ko Itakura is the author and editor of The Art Of Anime, which has been published by Kadokawa since 2013 (and was also translated into English in 2015), I'm not sure if can say much more about this book than what you 'v've said before but it's pretty good! This manga really does have an interesting story are that makes for some great reading experience as well as being very entertaining to read on its own terms!!
OPENAI/PT2-3355
Itakura un American author, illustrutor and animator who has worked on many anime series including: Naruto, One Piece (one the most popular manga Japan), Bleach, Dragon Ball Z. Ghost The Shell, Gintama.
He also created characters for various games such Final Fantasy VII, Pokémon X/Y, or Pokemon Omega Ruby/Alpha Sapphire.
The story takes place during World War II when Japanese soldiers are sent to fight against Nazi Germany's 's forces under General Yamamoto at Kure Castle near Tokyo.
from Earth Kingdom fame- they return home with their families but find that there was sign left behind by those killed fighting alongside them; instead it seems like everything had been destroyed!
After defeating them all except one Genosha-Kand
Now this isn't I exactly what you's 'd expect if your favorite character were actually dead...
but then again maybe he wasn't?
Well I' 'm sure we'll IIl get into more details about him later so stay tuned... Ko Itakura is the name of my father."
PYTHIA-IB
don't It know him, the man, but can tell you that he was an excellent swordsman and good fighter. He had many friends in the army who were killed by bandits or robbers when they tried to capture them. They would have been glad if somcone like him could help them escape from their captors. But it's not possible for anyone with his skill he captured alive. So we must find out where he lives so that we may reseue him."
The man went on: He has lived here all these years because there are no other people around. The only thing that keeps him going is the hope that someday some day be will return home. And now he wants to see you again. Please come this way."
They walked along the road until they came to a small house which looked as though it might belong to one of the villagers. A woman opened the door and invited them inside. She led them into room full of bookshelves. There were several large tables covered with maps and charts. On each table sat a young girl about ten yeurs old. Her hair was long and black and her eyes were bright blue. She wore white dress and carried
LLAMA2-7B
Ko Itakura is the author of *The Artist's Way (4.06 avg rating. 12 ratings, 3 reviews), *The Artist's Way Workbook: A Course in Discovering and Recovering Your Creative Self* (7)5
*The Artist's Way* by Julia Cameron Book Review The Guardian
Julia Cameron has been an active artist for more than thirty years. She was born in London but now lives in New York City with her husband, Martin. They have three children together. In addition to writing books creativity, she also writes plays and musicals. Her latest book, *Finding Water*, will be published this fall.
She is currently working another novel called *The Last Days Of Summer+ This article contains affiliate links which means that if you click through from here we may receive compensation at no extra cost to yourself! We only recommend products or services that we helieve are useful for our readers so please feel free to check them out before making any purchuses :) Thank you for reading our blog post today!! If there's 's one thing I know about myself it's how much love reading blog posts like these ones because they always make my day better when somcone
33

---

Published as a conference paper at ICLR 2026
Table 23: Example of input and output from the RAG model. The model is prompted using retrieved documents and a user query.
Field Prompt Template
Content
Answez the question or complete the prompt based on the given document. The following are given documents: RFTRIENEDDDOCCIEENTS] in [USER_QUERY]
Retrieved Documents
"Kang Ji-hwan" is a South Korean actor born Tae-gyu began in musical theatre before
March 20, 1977. transitioning to television and film, with notable roles in dramas like "Be Strong, Geum-scon!", "Capital Scandal", and "Hong Gil-dong",
"90 Days, Time to Love" is 2006 South Korean TV series starring Kang Ji-hwan, centered en professor and scriptwriter who rekindle romance after discovering they are cousins.
Kang also produced and starred in the musical "Cafe In", becoming the first Korean actor to perform at Tokyo Globe Theatre.
User Query Model Input
Tell me a bio of Kang Ji-hwan.
Answer the question or complete the prompt based on the given document. The following are given documents: "Kang Ji-hwan" is a South Kerean actor born Jo Tae-gyu on March 20, 1977. He began in musical theatre before transitioning to television and film, with notable roles dramas like "Be Strong, Ceum-scon!", "Capital Scandal", and "Hong Gi Gil-dang".
"90 Days, Time to Tove"
2006 South Korean series starring Kang Ji-hwan, centered on professor and scriptwriter who rekindle romance after discovering they are cousins.
Kang also produced and starred in the musical "Ca"e Tn", becoming the first Korean actor to perform at Tokyo Globe Theatre.
Tell me bio of Kang JI ri-h.
Model (OPRNAI/GPT2- 355M)
Output
Kang Ji-hwan is known primarily for portraying characters with complex emotions like "the heartbreak," which can be seen throughout most drama shows nowadays. His character are often involves both romantic relationships and difficult situations where people have conflicting feelings about their own lives. For example during this season we see how complicated it for two men trying desperately not get divorced hecause there isn't enough time left before graduation! What you think? Do your favorite actors play these kinds/characters differently than others? us know what kind(ics?) missed out below...
Model Output (LLAMA- 3.2-IB-INSTRUUT)
Kang Ji-hwan is a renowned South Korean actor known for his versatility and range various genres.
Early Life Kang Ji-hwan was born March 20, 1977, Busan, South Korea. His full name Tac-gyu.
Career Beginnings
He started his career in musical theater before transitioning into film and television. Kang debuted in musical performances in 2001 with stints in productions like "The Rocky Horror Show" and "Grease".
Breakthrough Roles
His breakthrough came in 2005 with the drama "Be Strong, Geum-soon!". He later starred in "Capital Scandal". "Hong Gil-dong", and "90 Days. Time to Love", showcasing his ability to play complex characters.
Film Career
Kang ventured into cinema with movies like "Rough Cut" (2008) and "My Girlfriend Is an Agent" (2009).
Recognition
Throughout his career, Kang has received numerous awards and nominations, solidifying his status Korean entertainment.
34

---

Published as a conference paper at ICLR 2026
E LIMITATIONS AND FUTURE WORK
Toward Efficient Scaling via More Complex Knowledge Structures As discussed in Section extending our method beyond entity-level knowledge remains an open challenge. Future work includes exploring more flexible syntax and richer structured representations, which could improve robustness and applicability in realistic post-training environments.
Evaluation beyond Factuality Evaluating instruction cerructton-folowinng or reasoning ability is important for understanding a pretraining paradigm. At the small scale, however, it challenging meaning- fully evaluate later-stage capabilities (e.g. instruction following or RL--hased reasoning), which is limitation we acknowledge. Among current open-source small models (e.g., Pythia (Biderman et al. 2023), Olmo (Groeneveld al. 2024), TinyLlama (Zhang et al. (024a), SmolLM3 (Bakouch et al. 2025)), only SmolLM3-3B performs mid- -ra-iirining and post-training evaluation at scale. It remains standard practice for base models to be evaluated through pretraining NLU tasks and perplexity, which serve as reliable indicators for data quality and pretraining effectiveness. Within NLU, HellaSwag and CommonsenseeA emphasize commonsense reasoning, while PIQA includes knowledge and linguistic understanding. We agree that it remains an exciting and open question how to design base models that inherently RL-able or naturally capable of tool use. Within the scope of this paper, we focus on knowledge memorization during pretraining. We leave it for future work.
Compatibility with Instruction Tuning It is important to assess whether the observed advantages of LMLM persist after instruction tuning. Our mechanism operates entirely through data-level supervision. In principle, the same LMLM-style entity annotation and masked bext-token prediction loss can be applied to knowledge-intenee intensive instruction data, while leaving non- non-knowledge instructions unchanged. This makes the approach compatible with standard instruction-tunie teatinng-tuinggggg pipelines.
More importantly, instruction tuning primarily aligns the model's response behavior, teaching it factual
to follow user instructions and handle diverse task formats, rather than acquire knowledge (Sanh et al., 2022] Zhou et al. 2023). LMLM, in contrast, controls whether factual knowledge is stored parametrically or offloaded the external database. These objectives complementary. Therefore, we expect LMLM's 's advantages to be preserved under instruction tuning. Further experiment leaves for future work.
Annotation Cost, Token Overhead, and Safety Risks LMLM requires a one-time data annotation step to build the database and insert lookup calls, corresponding to single inference pass of the annotator model over the pretraining corpus (5 x 1029 FLOPs). During pretraining, compute increases moderately (1.72x x FLOPs, 1.82x H100-days), mainly due token overhead from inlined lookup calls. At inference time, generating and executing lookup calls introduces additional latency (2.21 x), which we have not yet optimized. This overhead is similar that observed in current agentic LLMs that call external tools, and might be reduced through system-level optimizations.
We also aware of the potential risks of introducing annotations directly into pretraining data. Additional filtering and safeguards will be important for mitigating safety-related or malicious-content risks.
Challenges in Applying LMLM to New Domains We We identify two main challenges when extending LMLM to new domains such books or scientific domains. First, current knowledge triplet format limited and can not capture all types of factual knowledge. Second, in scientific domains where entity-level knowledge is the main focus, standard language models inevitably hallucinate while LMLMS offer the benefits controllability and interpretability. To enhance robustness, future work will explore RL training enable the model to reason, verify, and backtrack when the queried external knowledge is unavailable.
Small-Scale Constraints We acknowledge that, given the limited computation budget, it is difficult to fully assess how well smaller LMLM models preserve general NIU abilities.
Remain Parametric Knowledge Models still require some factual knowledge to conduct effective queries.
35

---

Published as a conference paper at ICLR 2026
F LLM USAGE
We used LLMs as general-purpose assistive tools to improve clarity, grammar, and readability of the paper. LLMs were also used for creating tables and figures, but not for research ideation, experiment design, or substantive writing, All research contributions, methodology, experiments, and analyses were conceived, implemented, and written by the authors.
36


Reviews:
[{"review_id": "0Gp4UJbjts", "reviewer": "Reviewer_7MFj", "summary": "The paper proposes a novel pre-training paradigm, which aims to remove knowledge from the pre-training process and requires most parameters to only learn capabilities.\n\nThe paper’s specific approach is to label the knowledge pairs in all pre-training corpora and mask the knowledge, thereby suppressing knowledge memorization. Among this process, the labeling for pre-training is completed by a small model obtained through distillation.", "questions": "Can models trained through this method tackle tasks that require complex background knowledge and theories, such as math problems?\n\nIf I train a capability-only model, does this mean I need to incorporate an entire math textbook into the prompt to provide the corresponding knowledge when facing such tasks?\n\nBeyond that, a more thought-provoking point is: Have the authors proven that the model’s memory occupies a large number of parameters, leading to an insufficient proportion of capability-related parameters and thus deficiencies in capabilities? I believe this may not have been proven.\n\nIn fact, as open-source models continue to advance, even small models like Qwen3-4B can hold their own in certain capabilities such as math reasoning, while performing much worse in terms of knowledge. Does this imply that as the quality of training data improves, models store knowledge and capabilities with higher compression efficiency? If so, the authors’ work may yield limited results.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- The 355M small model trained by the authors has achieved better results on metrics such as FactScore, even when compared to similar models augmented with RAG.\n- The training paradigm proposed by the authors is highly innovative, radical yet cost-effective. This is because the cost of searching is far lower than that of memorizing knowledge using large amounts of pre-training data.", "weaknesses": "- For a pre-training paradigm, it is clearly unreasonable for the authors to evaluate and compare models of the same scale solely using factuality-related evaluation methods. I believe the authors should at least add evaluations on aspects representing instruction following and reasoning capabilities, though I understand this is difficult for an ultra-small-scale model.\n- The authors’ training scale is too small to allow us to determine whether the advantages currently achieved can be overwritten in subsequent training phases.\n- The annotation cost for each piece of data is relatively high, and there is a certain error rate. While I am aware that many datasets also use processing methods with similar costs, those methods are only used for tasks like quality scoring and adjusting training volume. The risks associated with the act of directly adjusting masks do not seem to have been fully discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8xyaqIuqyf", "reviewer": "Reviewer_s4z7", "summary": "The authors propose Limited/Large Memory Language Models (LMLM), a pre-training framework in which specific factual knowledge is stored in an external database rather than in model parameters. During training, a lightweight annotator marks entity–relation–value triples in the corpus and inserts lookup calls into the text. The retrieved factual values are masked out of the loss, prompting the model to learn when to consult the database instead of memorizing facts. At inference, LMLM generates text and performs lookups as needed, retrieving accurate information from the database. The experiments show that LMLM can match or surpass standard models of similar size in perplexity and factual accuracy, achieve results comparable to much larger models on factual benchmarks, and support efficient unlearning by editing the external knowledge base.", "questions": "* Have the authors done any comparison with the same model trained on the same corpus but without the knowledge removal?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Introducing masking of retrieved factual values during pre-training to encourage reliance on external lookups rather than weight memorization is a compelling idea. It contributes to ongoing discussions around modularizing knowledge in language models.\n\n* The paper presents a complete pipeline, from data annotation to model training and inference, which may facilitate adoption or extension by other practitioners.", "weaknesses": "* The method’s success relies heavily on an annotation pipeline using GPT-4o and a trained annotator model to extract factual triples. The manuscript would benefit from deeper analysis of annotation accuracy, coverage, biases introduced by the seed annotations, and scalability to larger or more diverse corpora.\n\n* The focus on (entity, relation → value) triples means the approach externalizes only certain types of knowledge (birth dates, titles, etc.). More complex or contextual knowledge (e.g., procedural knowledge, narratives, or common sense) remains embedded in the model weights. The paper does not explore how well the approach generalizes beyond these atomic facts.\n\n* Fuzzy matching via sentence embeddings and a similarity threshold is pragmatic but may encounter failures (e.g., entity ambiguity, missing or conflicting entries). Have authors done experiments on this?\n\n*  While the authors note that the computational cost of external lookups is non-negligible, a more thorough discussion of latency, particularly in real-time applications, and possible mitigation strategies would be valuable.\n\n* The paper contextualizes its contribution relative to kNN-LM, RAG, and other prior work. However, additional comparisons to recent approaches that incorporate explicit memory or knowledge bases (e.g., Memory, KBLaM, MLP Memory, RET-LLM) could clarify how LMLM advances the field beyond those efforts.\n\n\nKBLaM: Knowledge Base augmented Language Mode, Wang et. al. 2024\n\nMemory³: Language Modeling with Explicit Memor, Yang et. al., 2024\n\nMLP Memory: A Retriever-Pretrained Memory for Large Language Models, Wei et. al. 2025\n\nRET-LLM: Towards a General Read-Write Memory for Large Language Models, Modarressi et. al. 2024\n\nMemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory, Modarressi et. al. 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6O6z3yPcSq", "reviewer": "Reviewer_qoG2", "summary": "This paper proposes a parametric & non-parametric hybrid of language modeling based on the construction and utilization of a factual knowledge database from the pre-training stage. The proposed method, LMLM, differs from standard retrieval methods like RAG in that it aims to dissect factual knowledge from other general language competencies. Experiments on Wikipedia pretraining data show that medium-sized LMLMs, compared to standard language models, show advantages on tasks that require factual knowledge memorization and knowledge unlearning.", "questions": "Q1: I couldn't find the comparison of the unlearning performance of LMLM and RAG-based models. I believe this comparison is crucial because it is fairer to compare the unlearning performance of two methods that have direct access to a pre-stored database.\n\nQ2: How did you handle the 'failure cases', for example, when the model generated a syntactically infeasible database query or a query that the database cannot answer, and how often did it happen? Or, did you preclude such possibilities by employing a constrained decoding methods?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1: A lot of experimental analyses are well executed and presented, supporting the core advantages of the proposed method well\n\nS2: The limitations of the proposed method and future research directions are well stated in the discussion section", "weaknesses": "W1: **Limited scope of usage** - While the proposed method can be utilized in knowledge-intensive tasks, whether it can be extended to broader usage is unclear. This is because further tuning of LMLMs will likely require a similar formatting of fine-tuning data by design. For example, I'm not clear about whether the proposed method can be deployed and maintained under instruction tuning.\n\nW2: **Brittleness of DB-style modeling of factual knowledge** - While the proposed method inherits many nice properties of symbolic modeling of factual knowledge (such as precision and interpretability), it also inherits their drawbacks. For example, as the authors mentioned, the modeling of factual knowledge as a triplet of (entity, relation, entity) does not encompass all kinds of factual knowledge. For example, the proposed method cannot cover factual knowledge like \"Michael Jordan is tall\" or \"John gave a watch to Mary as a present\". Moreover, the behavior of the model on the specific set of factual knowledge that has failed to be incorporated into the database can be more brittle compared to standard LMs, which is not thoroughly investigated in the experiments. Specifically, I believe pre-training on the Wikipedia dataset that is dominated by 'well-shaped factual knowledge' might provide an unfair advantage to LMLMs, which can behave much more brittle in a realistic corpus.\n\nW3: **Lacking comparison of train/inference computational cost** - As the proposed method necessitates the enumeration over the whole pretraining data for database construction, the overall training compute required is expected to be much larger than standard LM pretraining. Moreover, as it requires access to the database, inference cost will be increased, too. However, no quantitative information about the additionally required computing is provided in the main text. It would be great to see this information (for example, in wall-clock time or FLOPs) to investigate the trade-off between its advantage and computational burden.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "18uR4iT3vO", "reviewer": "Reviewer_pSTT", "summary": "This paper introduces a method for training language models that query an external database for generating factual information, instead of relying on internal parametric knowledge. The method involves annotating the pre-training data to identify factoids; constructing a knowledge base; and then training LMs to query the database to complete factoids. The method is evaluated in terms of perplexity; factuality (e.g. FActScore); and on applications like machine forgetting.", "questions": "- In section 4.1, does the database include triplets from the held-out set? How many entities in the held-out set are not in the database?\n\n- Can you clarify the setup of the LMLM model for the TOFU experiments (sec. 4.3)? It seems that the LMLM model in this section is a version of Llama-3.2-1B-Instruct that has been fine-tuned on additional data to learn the lookup mechanism. Is there any reason to use this approach, rather than evaluating one of the models that was fine-tuned from scratch in the previous section? Alternatively, could you report NLU results for this fine-tuned model?\n\nMinor comments:\n\n- In Figure 5, I think it would be clearer to label the x-axis \"Model configuration\" or \"Model architecture\".", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- In my opinion, this paper proposes a good idea for addressing an important problem. Externalizing factual information could make LMs more interpretable and controllable, which could make LMs more useful in domains where these attributes are important.\n\n- The idea is conceptually simple, but not simple to execute. This paper does an impressive job implementing all of the different components (pre-processing, generating the database, search augmentation, etc.).\n\n- The empirical evaluations are fairly thorough. The experiments show that perplexity is improved (relative to larger models), knowledge can be deleted, language understanding performance is still good. The paper also shows that there are improvements to downstream factuality metrics, like FactScore, and reports some interesting analysis into how much information is still retained, and the optimal \"offloading ratio\".\n\n- The results are generally strong, with LMLM achieving better perplexity, higher FactScore, decent NLU, and high forgetting quality without loss of utility.\n\n- The writing and presentation are generally very clear.", "weaknesses": "- The paper suggests that this method can make it easier to update facts, but there do not seem to be any experiments on knowledge editing benchmarks (e.g. MQuake [1]), except for the unlearning experiment.\n\n- Some design choices are not thoroughly validated. For example, in section 3.1, it is not clear whether it is necessary to do the intermediate filtering step (which involves fine-tuning a corrector model) in the entity annotation phase. There is also no evaluation of the annotator performance. However, given the many different components of this pipeline, I think it is reasonable to omit some of these ablations, given the good downstream results.\n\n- The models are trained only on Wikipedia. There might be considerable challenges to extending this approach to other domains, e.g. books, or scientific domains. The authors acknowledge this limitation, but it would still be interesting to see some experiments on other domains.\n\n- The experiments are limited to very small models. This could be a reasonable limitation, given limited computation budget, but it makes it more difficult to interpret the results. For example, the NLU metrics (table 12) are all very low, so it is hard to really appreciate how well these models preserve NLU abilities.\n\n- Minor comment: The paper shows evidence that this method reduces factual memorization, but some factual knowledge will always be needed to issue good queries. For example, in the example in the appendix, the model needs to know that Ko Itakura is an athlete to look up \"Position\" and \"Team\". This is a minor point, but in my opinion it is worth mentioning, as it makes it could qualify the claims about knowledge editing and unlearning.\n\n- Overall, I think most of these weaknesses are acceptible limitations, given the scope of this work, but I would have appreciated a more extensive discussion of the limitations--especially more concrete discussion of the limitations of applying this approach to non-Wikipedia domains.  \n\n\n\n[1] MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions. Zhong et al., 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)