Self-Updatable Large Language Models by Integrating Context into Model Parameters
Authors: Yu Wang
Xinshuang Liu
Xiusi Chen
Sean O'Brien
Junda Wu
Julian McAuley
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Knowledge Injection
Self-Updatable LLMs
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper proposes a simple method to absorb contexts of any form into model parameters.

Abstract: Despite significant advancements in large language models (LLMs), the rapid and frequent integration of small-scale experiences, such as interactions with sur- rounding objects, remains a substantial challenge. Two critical factors in assimilating these experiences are (1) **Efficacy**: the ability to accurately remember recent events; (2) **Retention**: the capacity to recall long-past experiences. Current methods either embed experiences within model parameters using continual learning, model editing, or knowledge distillation techniques, which often struggle with rapid updates and complex interactions, or rely on external storage to achieve long-term retention, thereby increasing storage requirements. In this paper, we propose **SELF-PARAM** (Self-Updatable Large Language Models with Parameter Integration). SELF-PARAM requires no extra parameters while ensuring near-optimal efficacy and long-term retention. Our method employs a training objective that minimizes the Kullback-Leibler (KL) divergence between the predictions of an original model (with access to contextual information) and a target model (without such access). By generating diverse question-answer pairs related to the knowledge and minimizing the KL divergence across this dataset, we update the target model to internalize the knowledge seamlessly within its parameters. Evaluations on question-answering and conversational recommendation tasks demonstrate that SELF-PARAM significantly outperforms existing methods, even when accounting for non-zero storage requirements. This advancement paves the way for more efficient and scalable integration of experiences in large language models by embedding knowledge directly into model parameters.

Published as a conference paper at ICLR 2025
SELF-UPDATABLE LARGE LANGUAGE MODELS BY IN- TEGRATING CONTEXT INTO MODEL PARAMETERS
Yu Wang': Xinshuang Liu'r, Xiusi Chen', Sean O'brien', Junda Wu', Julian McAuley' University of California San Diego, University of Illinois Srbana-Champann
ABSTRACT
Despite significant advancements in large language models (LLMs), the rapid and frequent integration of small-scale experiences, such interactions with surrounding objects, remains a substantial challenge. Two critical factors in as- similating these experiences are (1) Efficacy: the ability to accurately remem- ber recent events; (2) Retention: the capacity to recall long-past past experiences. Current methods either embed experiences within model parameters using con- tinual learning, model editing, or knowledge distillation techniques, which often struggle with rapid updates and complex interactions, or rely on external stor- age to achieve long-term retention, thereby increasing storage requirements. In this paper, we propose SELF-PARAM (Self-Updatable Large Language Mod- els by Integrating Context into Model Parameters). SELF-PARAM requires no extra parameters while ensuring near-optimal efficacy and long-term retention. Our method employs training objective that minimizes the Kullback-Leibler (KL. divergence between the predictions of an original model (with access to contextual information) and target model (without such access). By gener- ating diverse question-answer pairs related to the knowledge and minimizing the KL divergence across this dataset, update the target model internal- ize the knowledge seamlessly within its parameters. Evaluations on question- answering and conversational recommendation tasks demonstrate that SELF- PARAM significantly outperforms existing methods, even when accounting for non-zero storage requirements. This advancement paves the way for more ef- ficient and scalable integration of experiences in large language models by em- bedding knowledge directly into model parameters. Code is open-sourced at www.ttttww.......ooooeeee https://github... com wwt.w/.wwwwiisnnnnnoiaom SEES-EEELLEEEEEEEEE -PAAMM
INTRODUCTION
dynamic environments, whether in virtual worlds such as video games or real-world human so- cieties, developing cognitive system capable of effectively interacting with objects poses signif- icant challenges. We define the interactions between the cognitive system and its environment as experiences. To remain functional and adaptive, cognitive system must continuously evolve by integrating new experiences and reflecting upon past interactions whenever engaged by the environ- ment (Wang et al. (0244)) One of the primary challenges this evolutionary process is the system's ability to absorb experiences, adapt accordingly, and recall key events in the future. This challenge highlights critical properties that the system must possess: (1) Efficacy: The system must main- tain an impeccable memory of recent events, ensuring that each step of knowledge integration is both accurate and effective. (2) Retention: The system should demonstrate robust ability to recall experiences from the past, indicating strong term- memory capabilities.
To achieve long-term Retention, cognitive system must effectively store its experiences. Ex- isting methods for managing past experiences can be categorized based on whether they require additional modules or parameters beyond the model's inherent parameters: (1) Methods without Additional Modules or Parameters: These approaches, including continual learning, model edit- ing, and knowledge distillation techniques, embed knowledge directly into the model's parameters. For instance, model editing methods such MEND (Mitchell et al. 2022), ROME (Meng al.) 2022], MEMIT (Meng et al. 2023), and Moodel-Editing-FT Ganggdhar & Stratos
2024) update
Y. Wang and X. Liu contribute equally; correspondence {yuwl 64, x 1235 (33500 www.ccd.eeee..

---

Published as a conference paper at ICLR 2025
specific components of the language model to incorporate modify information. These methods are particularly effective for inserting factual statements like "The highest mountain above sea level is Mount Everest." However, they may be limited in handling more complex experiences that ex- tend beyond simple facts. Continual learning methods (Sun et al. 2020a Wu et al., (024) can manage more intricate information but typically require compiling corpus for pre-training fine- tuning, which may not be suitable for frequent and rapid updates. Moreover, continual learning often relies on Nerd-Word Prediction (NWP) loss, which may not guarantee strong Efficacy in im- mediate knowledge integration. Existing Knowledge Distillation methods mainly focus on distilling factual statements (Padmanabhan et al., 2024) or prompts (Choi et al. 2022) into model parame- ters, which may fall short when injecting complicated experiences. (2) Methods with Additional Modules or Parameters: These approaches employ external components such as text repositories, constructed knowledge bases, or memory modules to store past experiences, thereby facilitating long-term Retention. For example, MemoryLLM (Wang et al., 2024c) and Retrieval-Augmented Generation (Lewis et al. 2020) allow models to access external information during inference. The language model can directly attend to external modules, which can enhance Efficacy by providing accurate and relevant information. However, relying external storage necessitates maintaining large modules to store all past experiences, thereby increasing storage requirements. Smaller exter- nal modules usually offer only marginal benefits (Wang et al. 2024c Bulatov al. 2022 2023).
To address these challenges, we propose a novel method that requires zero additional parameters while yielding nearly optimal Efficacy and maintaining robust long-term Retention. We call this method SELF-PARAM (Self-Evollvblee Large language models with Parameter Integration). We formally define context injection as effective when the model can accurately answer questions about specific contexts after incorporating them into its parameters. Based on this definition, we introduce a training objective that minimizes the Kullback-Leible (KL) divergence between the predictions of an original model (which has access to the context) and target model (which does not). The intuition behind using KL divergence that the conditional probabilities of the original model capture the relationships between future queries and the context. If our training sentences sufficiently diverse (ideally encompassing a wide range of possible queries), the target model should be able to generate appropriate responses to any queries related to the context after training. Since is infeasible to use all possible sentences for training, we employ an instruct model to generate question-answer pairs about the context, along with sentences sampled from the pre- pre-training dataset, to construct a target sentence set for each context. By minimizing the KL divergence between the conditional distributions of the original model (conditioned on the context) and the predicted distributions of the target model (without the context), we effectively update the target model to interaaiiz the context within its parameters.
To demonstrate the efficacy of our method, we evaluate SELF-PARAM on tasks involving Single Context Injection and Batch Context Injection. To assess knowledge retention, we conduct exper- iments on Sequential Context Injection. Additionally, we apply our method to a Conversational Recommendation task, injecting conversations between users and the system into an instruct model. In this setting, the models' recommendation recalls improve significantly over all other baselines. These comprehensive experiments show that SELF-PARAM surpasses existing methods by large margin while maintaining zero additional parameters. This advancement effectively achieves both high Efficacy and robust long-term Retention without the need for extra storage modules.
2 RELATED WORK
Based on their extra storage needs, existing approaches for injecting context into models can be categorized into methods without extra storage and those with extra storage. We provide a detailed overview of each category below.
Methods without extra storage. This category primarily encompasses Model Editing, Continual Learning, and Knowledge Distillation techniques. For model editing methods (Yao et al., 2023). we focus on approaches that directly modify the model parameters, thereby embedding new knowl- edge within the parameters themselves. MEND (Mitchell et al. 2022) introduces rank-one model edits to update the model parameters, utilizing meta-learning to train a lightweight meta-network for the editing process. Similarly, methods such ROME (Meng al. 2022) and MEMIT (Meng jet al. 2203)) propose closed-form solutions for editing the MLP layers. In contrast, Model-Editing- FT (Gangadhar & Stratos 2024] fine-tunes the entire model the factual statements that need to
2

---

Published as a conference paper at ICLR 2025
be injected. T-Patcher (Huang et al. 2023) and CaliNet (Dong et al. 2022 store new knowledge in additional neurons. However, these approaches may face challenges related to the continuous growth of parameters when the knowledge to be injected comprises lifespan experiences. As for continual learning, it is usually used in the following situations: (1) Real-Time Assimilation of Dy- namic Data: Continual learning facilitates the integration of information from diverse sources such as news (Sun et al. 2020b), scholarly articles Cossu et al. (2022), and social media Coosu et al. (2022). (2) Injection of Extensive Knowledge into Language Models: [aang et al. (2022) presents method for continual knowledge learning that effectively updates LLMs with new data without compromising previously acquired knowledge. (3) Domain Adaptation: Continual training on new data streams enables models to adapt to specific domains in both language and vision. Coosuu et al. (2022) demonstrate this by continuously training models on evolving datasets. Additionally, Ke et al. (2023) propose soft-masking mechanism to update language models with domain- corpora, enhancing performance while preserving general knowledge. Various domain-adapted models have been developed, including those tailored for the financial domain (Xie et al., 2023), E-commerce (Ma et al.) 2023). J and academic content (Wei et al. 2023). (4) Instruction Tuning for Enhanced Reasoning and Interaction: To enable models to tackle novel tasks, dask-incremental continual instruction tuning has been proposed (Wang et al. 2024b 2023). These tasks encompass mathematical problem-solving (Azerbayev et al. 2023). utilization of calculators, search engines, and databases [Qin et al. 2023). With the rapid development of new tools such as advanced soft- ware libraries, novel APIs, and domain- domainn-pecific utilities (Liang al. 2023 Jin et al. 2023), there is an increasing necessity to continuously update language models to swiftly adapt and master these innovations. In the realm of Knowledge Distillation, Choi et al. (2022) propose method for dis- tilling prompts directly into model parameters, effectively replicating scenarios where prompts are appended prior to generation. Later, Padmanabhan et al. (2024) introduces a technique for distilling factual knowledge into model parameters such as "ChatGPT is an AI chatbot developed by Ope- nAI." Both approaches demonstrate impressive performance in knowledge injection. However, they may not be directly applicable to the injection of experiences which can be much more complicated.
Methods with extra storage. While the preceding category primarily focus on updating the model itself with new knowledge, another line of research involves storing knowledge in external modules and leveraging these modules to generate responses as needed. This approach encompasses sev- eral methodologies: (1) Retrieval-Augmennted Generation: This framework involves maintaining repository of past knowledge, which can be either in the form of knowledge graphs or organized
texts (Wang al. 2024d), from which relevant information is extracted using a retrieval mechanism
when generating responses (Gao et al., 2023] Karpukhin et al. 2020 Sarthi et al.) 2024] Gutiérrez al.] 2024, Izacard et al. [022] Wang et al. 2022). (2) Long Context Methods: These methods store all relevant knowledge within the model's is context window. When prompted, the model pro- cesses both the extensive context and the input prompt to generate an appropriate response (Wang let al., 2024a;; Tworkowski al.) [2033] Chen et al. 2023 Han et al. 2023). (3) Memory-Augmentt Methods: This approach typically involves maintaining a memory pool where knowledge is stored. These methods generally include two main operations: (i) Write Operation: Updating the memory pool, often through the compression of textual information into the memory space (Zhou et al. 2023]
Wang et al. 2024c). (ii) Read Operation: Retrieving relevant information from the memory pool to inform response generation (Wang et al. 2024cc Zhong et al., 2023). Although mngmore-augmented methods can achieve impressive performance, they may require significant additional storage space to accommodate complex knowledge externally. In contrast, SELF-PARAM circumvents the need for external storage by embedding all information directly into the model weights via backpropaga- tion. This approach eliminates the additional space requirements associated with external modules, allowing the model to retain and utilize knowledge internally without external dependencies.
3 METHODOLOGG
3.1 DEFINITION OF CONTEXT INJECTION
This paper introduces the concept of Context Injection in language models. We define Context Injection as the process of modifying language model to incorporate specific context ar enabling the model to generate accurate and context-aware responses to related queries. Let 0 represent the original language model that does not contain the information in x. When querying the original model 0 with a prompt p related to the context æ, the generated answer aj is given by:
a1 Generate(0,
3

---

Published as a conference paper at ICLR 2025
p.
Py e7
X
LM
p
* θ %
p
min KL
The Process of Context Injection. Given the original model and a context æ, our goal is to obtain new model 0' that incorporates directly into its parameters. To achieve this, require that for any prompt p. the model 0' generates the same output as the original model 0 would when provided with the combined input p. In other words, l' should emulate I's behavior with access to the context z when only prompted by For a random sentence s, let P(s x + 0) denote the token distribution generated by the original model when prompted with x + p. and let P(s | p. (') represent the distribution from the new model 0' when prompted with p. Ideally,
Figure 1:
for all possible sentences 8, we aim to minimize the KL divergence between P(s x + p, 0) and P(s p, 0*). This ensures that (' accurately integrates the context within its parameters.
For example, consider the following context and prompt:
x national survey only 388 wi willingly
p What percentage of bank branches willingly disclose fees ...??
Here we omit some words with "... " to save space. In this scenario, the answer aj produced by 0 is likely to be incorrect due to the absence of relevant information in the model. However, by incorporating the context into the prompt, the model can generate a more accurate response:
ag Generate p).
where x + p denotes the concatenation of x and into a single, extended prompt. Assuming that 0 is sufficiently robust and the necessary information is contained within æ; the resulting answer d2 should be correct. Then we define a new model /' as the obtained model after injecting the context into the original model When 0' queried with the prompt p. the generated answer ag is:
ag = Generate
03 is accurate, this indicates that have successfully incorporated the context into the original model 0. In summary, Context Injection involves updating the original language model f to 8' by embedding specific context x. therehy enabling the model to accurately respond any questions related to x. This holds true provided that the original model A can generate correct answers when supplied with both the context x and the relevant question.
This definition leads the first essential property of Context Injection: Efficacy. Efficacy is achieved when the modified model 0' can accurately answer questions pertaining to the injected context Furthermore, consider sequence of contexts 21...., In injected sequentially into 0 to obtain a final model 0n. Retention defined as the degree to which 0, preserves the information from previously injected contexts T1,... nn Retention
be measured by querying 6,, with questions related to each x; for i E (1, (1,..., n - 1}. In this paper, our objective is to maximize both efficacy and retention while maintaining storage complexity S of zero.
3.1.1 THE PROCESS OF CONTEXT INJECTION
To integrate a context into the model parameters, we aim to ensure that the updated model ø' generates responses identical to those produced by the original model 0 when the latter is provided with the combined prompt + p. Specifically, the original model utilizes the context a when responding to the prompt This approach is illustrated in Figure [ Formally, to enable 0' to respond accurately to queries about æ, we seek to approximate the following relationship:
Cenerate(B', p) Generate(f.
where p represents a prompt related to the context æ, such as question pertaining to x. Achieving an ideal scenario where Equation Eq. holds for any possible prompt p implies that any question
4

---

Published as a conference paper at ICLR 2025
answerable by 9 with the context + p also be addressed by 0' using only p. This signifies effective context injection. Consequently, we define the following optimization objective:
8' = arg min E, KL [Po(s PI p)]]
(2)
where s denotes any sentence. whether related or unrelated to x, and KL|-II-] represents the KL divergence. If the objective in Equation Eq. iis minimized to zero, then Equation Eq.[[iss inherently satisfied. This concept is depicted in Figure] 1
In practice, it is exhaustive to collect all prompts, thus we simply absorb the prompt into obtain the following objective:
0' arg min E, KL [Po a)) P
6
Moreover, aggregating the KI. divergence across all possible sentences is infeasible. Therefore, we employ sampling techniques to approximate the objective effectively. The sampling methodology for is detailed in the subsequent section. We denote the set of sentences used to train O' according to Eq. 3 as & and we call this set as Target Sentence Set in the remaining paper.
3.2 CONSTRUCTION OF TARGET SENTENCE SET
To absorb I into 0, we need to construct the target sentence set which contains two parts: (1) sentences related
(to inject the context 2) and (2) sentences unrelated to I (to maintain the model's original abilities). As for (I), we employ an instruct model (we tried two settings: using gpt-4o-mini and using the corresponding instruct model, for Mi mar-77 and -lama-3-8B-Inse ruct for Llama-3-8B) to generate set of contextually relevant question- question-answer answer pairs as the sentence set. The detailed prompt used for constructing is shown in Appendix A.3 Then for (2), we randomly sample sentences from SlimPajama dataset (Soboleva al., 2023) as the unrelated set of sentences. form Note that SlimPajama helps preserve the model's general capabilities, effectively acting of regularization during training. With the balanced sentences from the context-related and context- unrelated datasets, hope to inject the specialized knowledge encapsulated in into the model while maintaining the model's broader linguistic capabilities.
i.e. Mistral-7B-Instruett
3.3 DISCUSSION
Where does the performance gain come from? We argue that the new objective in Eq, Jencour- ages more effective knowledge injection than simply using the next-word prediction (NWP) loss that is widely used in continual learning (Zhang et al. 2023). In our experiments (shown in Table T and T).le we implement the baseline of fine-tuning the model on the context and then querying it with the question Q (denoted as "FT (C), Q"), where the results show that the model fine-tuned with NWP loss yields poor performance when responding to queries, indicating failure in effective context injection (i.e., failure of Efficacy). To understand the inherent reason, consider the following example: let x be a simple sentence "David likes apples. " a model trained with NWP loss on may overfit to the specific form of the question, failing to answer correctly when queried in a different format- - for example, "What fruit does David like?". However, our method trains various forms of question-answee answering pairs pertaining to the same underlying knowledge, which encourages the model to both absorb the knowledge and use it, instead of simply memorizing the facts in specific forms.
Analysis of the Computational Cost We acknowledge that our algorithm could introduce addi- tional computation costs. Compared to the fine-tuning baseline, our approach roughly doubles the overall computational cost in terms of both time and monetary resources. Specifically, our method involves four key steps: (a) generating QA pairs, (b) computing base logits. (c) computing updated logits, and (d) performing backpropagation and optimization. contrast, continual fine-tuning base- lines, such as FT(C) and FT(S) in Tables 1-3, only require steps (c) and (d). Assuming each step is of comparable time complexity, our total cost is approximately twice that of continual fine-tuning. However, given the effectiveness of our method, we argue that these additional costs are justified.
Hallucination Mitigation As the context, there the risk of hallucination, where the model produces an answer even when lacks the necessary knowledge. To mitigate this issue, carefully filter out prompts that require
encouraging the model to generate answers without seeing
5

---

Published as a conference paper at ICLR 2025
direct access to prior context in the original PwC dataset. For example, prompts such as "summa- rize the previous context" and "write a title for the above text" were removed, leaving only factual questions. This refined dataset contains context and questions that are based constant knowl- edge, such as Question: What language was spoken in the tape recordimg ng of Jamal 1. Khashogg!'s s murder? Answer: Arabic. These factual statements less likely to induce hallucination, which helps support the method's robustness. Regarding the Target Sentence Set, we are not directly fine-tuning the model to predict each word within this set. Instead, we employ KL divergence to transfer the behavior from the original model to the updated one, which may also reduce the risk of hallucination.
EXPERIMENTS
In our experiments, we mainly consider the following tasks: (1) Single Context Injection ($ 4.1 We adopt SELF-PARAM inject single context into the model and then ask the model with the related questions. (2) Batch Context Injection (....). We adopt 3ELF-PARAM to inject multiple contexts simultaneously and then ask the model with the questions related to the injected contexts. (3) Sequential Injection ($ 4.3) We sequentially inject multiple contexts into the model, ensuring that the model does not retain access to earlier contexts during the injection of newer ones. We then test the model's ability to recall and respond to questions related to the previously injected contexts, thereby assessing its long-term retention. (4) Conversational Recommendation (.44. We inject up to 1,000 conversations between users and the system into the model to allow the model to read the conversations. After injection, we prompt the model to generate movie recommendations and calculate the recall of these recommendations to measure the recommendation quality. In these tasks, (I), (2), (4) correspond to Efficacy evaluation, while (3) represents the Retention evaluation.
4.1 SINGLE CONTEXT INJECTION
Experimental Setup We use PwC dataset (Ge et al. 2023), consisting of triples in the form (context, quest ion, answer). To ensure concise answers, we filter out examples where the answers exceed five tokens, resulting in 1,581 unique contexts and 3,353 unique questions. From these, select the first 100 contexts paired with 225 questions for the single context injection task. For each (context, quest ion, answer) example, we inject the context into the model and then query the obtained model with the corresponding question. We evaluate performance using the QA-F1 score, as defined in LongBench (Bai et al. 2023). We conduct experiments with three backbone models: OpenLLLMA--33
(Geng Liu 2023), Mistral-7B (Jiang et al.. 2023), and Llama3-8B (Dubey et al. 2024). We compare our proposed method, SELF-PARAM, against following baselines: (1) Base: The original, unmodified model. (2) FT (C): Fine-tuning the original model solely on the context. (3) FT (S): Fine-tuning the original model the target sentence set.
Overall Performance Comparison The results are summarized in Ta- ble [ Our method, SELF-PARAM, significantly outperforms the fine- tuning baselines, demonstrating its effectiveness in knowledge injec- tion. Specifically, fine-tuning the model solely on the context (FT (C), Q) yields poor performance, albeit slightly better than the unmodified base model (Base, Q). This indi- cates that while some knowledge is injected, much of it remains inacces- sible. The poor performance of FT (C), Q highlights a major limitation of continual learning approaches that rely on NWP loss: they suffer from low Efficacy, evidenced by the model's 's inability to generalize knowledge across different query formulations. In contrast, SELF-PARAM trains the model using diverse question-answer pairs related to the same context, which encourages the model to both absorb and utilize the knowledge effectively. This approach prevents overfitting specific question formats, ensuring that the model can handle varied queries
Openllama-3B-v2
Mistral-7B
Llama3-8B
Base, C+Q Base, Q
0.5043
0.5461
0.5368
0.1122
0.1503
0.1051
FT (C). Q
0.2217
0.2433
0.1213
FT Q
0.2742
0.3246
0.3115
SELF-PARAM, Q
0.4927
0.3705
0.4213
Table 1: Experimental results for Single Context Injec- tion. "C+Q" indicates that both context and question are provided as input to the model, serving as an upper bound for injection performance. "Q" denotes that only the quest ion is provided, requiring the model to answer the questions based solely on its internal parameters.

---

Published as a conference paper at ICLR 2025
about the injected context. Moreover, FT (S), Q performs worse than SELF-PARAM, underscoring that our method does more than merely injecting generated QA pairs. Instead, SELF- SELF-PARAM facil- itates the model's ability to internalize and apply the context dynamically. This observation aligns with our training objective in Eq. [] where minimizing the KI. divergence between the original and target models allows SELF-PARAN to better approach the upper bound represented by Base, C+Q.
4.2 BATCH CONTEXT INJECTION
Experimental Setup We use the PwC dataset and the same three backbone models as in $ 4.1
The preprocessing is the same. Then we extract 100 and 500 contexts, with 225 questions and 1044
questions, respectively, from the obtained subset to perform batch injection. The injection step and the evaluation process are separated. We first inject 100/500 context into the model, and then we ask the model all questions. Note that one context may have more than quest ion, calculate the average fl-score of multiple questions for one context and then calculate the average fl-score across all contexts. We conduct experiments the same three backbone models as in $ 4..1 OpenL.LaMA-3B--
(Geng & Liu 2023). Mistral-7B (Jiang et al. 2023). and Llama3-8B (Dueyy et al. 2024). We compare our proposed method, SELF-PARAM, against the following baselines: (1) Base, the original, unmodified model; (2) FT (C), fine-tuning the original model solely on the contexts; (3) FT (8), fine-tuning the original model on question-answerinn
pairs generated by an
instruct model; model; (4) MemoryLLM
method continually trained on Llama3- 8B with an additional memory module of size 1.67B tokens (Wang et al.) 2024ch: (5) InfLLM, a long-context method that can be integrated with large language models to achieve an effectively infinite context window; (6) Dense Passage Retrieval Karpukhnn et al. 2020), a retrieval- augmented generation (RAG) method that encodes documents and questions into embeddings and uses faiss s for document retrieval; (7) BM25, a sparse retriever-based RAG method; and (8) RAP- (Sa(harthi et al. 2024), which constructs knowledge graphs using an advanced language model (we use gpt-4o-mini and leverages the knowledge graph for retrieval. For the long-context methods, we concatenate all contexts to form single, extended context input and prompt the model with: "contexti,"" .coneexty, question" For RAG methods (DPR and BM25), treat each context an individual document and perform document-level retrieval. Specifi- cally, we use top-4 retrieval for OpenLLaMA-33--v2 top-8 retrieval for Mistral-7B and Llama3-8B, and top-1 retrieval for RAPTOR.
a anemory-augmente
Here we introduce the concept of Storage Complexity S as proposed in Wang et al. (002d)), which indicates the relative storage space required store all the contexts that need be injected compared to the size of all the contexts (such as the total number of tokens in all contexts). As we do not require any additional storage space, SELF-PARAM has S as zero In contrast, RAG and congccotextt methods need to store all contexts, indicating (n)) storage complexity. As MemoryLLM has fixed-sized memory, the storage complexity (1). Here for each baseline, we mark the storage complexity to indicate the general complexity which could also be applied to similar methods, rather than the specific complexity for every single baseline, thus offering a more generalized perspective.
Overall Performance Comparison The results are summarized in Table 2 Our method, SELF- PARAM, consistently outperforms all baselines across differentt models and context sizes, demon- strating its superior ability to inject and utilize multiple contexts without requiring additional pa- rameters. From Table 2 it is evident that SELF-PARAM consistently achieves the highest QA-F1 scores across all models and context sizes. Specifically, SELF-PARAM closely approaches the upper bound performance of Base, C+Q without requiring any additional parameters. This demon- strates the method's ability effectively inject and utilize multiple contexts solely through param- eter updates. In contrast, the fine-tuning baselines (FT (C), FT (S)) exhibit significantly lower performance, especially as the number of contexts increases. Notably, FT (C), which relies on next- word prediction (NWP) loss, struggles with maintaining Efficacy in knowledge injection, similar to observations in single context injection. Meanwhile, FT (S) performs better than FT (C) but still falls short of SELF-PARAM, highlighting that our method does more than merely injecting gener- ated QA pairs-it it facilitates the model's ability to internalize and apply the context dynamically. This observation aligns with our training objective in Eq. 3 where minimizing the Kullback-Leibler (KL) divergence between the original and target models allows SELF-PARAM to approach the up- per bound represented by Base, C+Q. deemory-augmenter methods like MemoryLLM-8B and retrieval- auggeentd generation (RAG) methods such as DPR, BM25, and RAPTOR require ad- ditional parameters or modules store contexts, leading to increased storage complexity (O(n)).
7

---

Published as a conference paper at ICLR 2025
Openllama-3B-v2
Mistral-7B
Llama3-8B
S
# of Contexts
100
500
100
500
100
500
Base, C+Q Base, Q
0.5043
0.4869
0.5461
0.5725
0.5368
0.5089
0.1122
0.0814
0.1503
0.1382
0.1051
0.0967
FT (C), Q FT (S), Q
0.2085
0.1231
0.1659
0.1433
0.1065
0.0878
0
0.1925
0.1784
0.2350
0.3459
0.3179
0.2848
0
MemoryLLM-SB
0.1435
0.0841
0(1)
InfLLM
0.1437
0.1003
0.1619
0.1783
0.1301
0.1244
((n)
DPR
0.2795
0.2528
0.3175
0.3092
0.2184
0.2310
((n)
BM25
0.1475
0.1872
0.3104
0.3135
0.3083
0.2862
O(n)
RAPTOR
0.1344
0.1529
0.2133
0.1969
0.2000
0.2055
O
SELF-PARAM, Q
0.5082
0.5048
0.4521
0.4384
0.4368
0.4221
0
Table 2: Overall performance comparison on the task of batch injection. Here "C+Q" means provid- ing the model with the specific context containing the answer for each question. Thus "Base, C+Q" serves as the upper bound.
0 14
0.46
50
0.46
0.50
0.46
0:47
0.46
0.44
0.44
0.48
0.48
0.35
0.35
0.34
0:31
0.36
0.36
0.30
0.33
0.31
0.50
0:35
0.33
0.37
0.38
0.36
3
0.11
0.34
14
0.48
0.42
0.45
0.43
0.40
0.40
0.38
0.37
0.38
0.35
0.28
0.22
0.24
0.26
0.26
0.20
0.27
023
0.6
0.
0.51
0:47
0.45
0.46
0:44
0:45
0.42
0.38
0.35
0.31
Index of the context
81 71615141111111 01154413121111 211010 8 7 665
0.19
0.17
0.17
0.19
0.22
0.16
0.15
0.21
19
25
0.51
0.22
0.59
0.41
0.41
0.52
0.43
0.53
0.43
0.53
0.41
0.46
0.51
0.36
0.34
0.50
0.32
0.45
0.47
0.32
0.34
0.47
053
0.34
0.28
0:43
0.32
0.43
0.28
0.41
0.44
0.30
0.5
0.12
0.13
0.13
13
0.14
0.13
0.57
0.45
0.37
0.33
0.34
0.29
0.08 0.09 0.14 0.15
15
0.12
0.10
0.11
0:46
0.39
0.36
0.29
0.30
0.31
0.31
0.27
0.24
0:33
0.29
0.31
0:29
0.24
15
0.21
0.19
0.14
0.43
0.40
0.41
0.46
0.4
0.12 0.13
0. 0818
0.21
20
0.22
0.21
0.19
0:15
0.17
0.49
0.38
0.37
0.37
0.38
0.38
0.30
0.35
0.34
0.29
0.33
0.10
0.38
0.21 0.20
19
0.19
0.
0.23
0.20
0.16
0.18
0.20
19
0.24
0151
0.44
0.48
0.42
0.41
0 40
0.35
0.36 0.36
0.3
0.47
0.08
0.09
08
0.31 0.08
0.09
0.10
0.09
0.11
0.10
0.
0.14
0:13
0.09
0:57
0.37
0.40
O. 40
0.30
0.29 0:29
0.13
13
0.17 0.17
0.17
0.17
0.15
0.14
0.19
0.16 0.18
0.22
0.55
0.40
0.36
0.14 0.18 19 0,20
15 0.14
0.13
0.12
0.15
0.15
0.11
0.14
0:13
0111 0.09
010
0.33
0.35
0.29
0.29
0.2
0.06 0.06
0.11 0.34
14
0.18
0.15
0.17
0.14
0.15
0.13
0.16
0.12
0.16 0.15 0.15
0.16
4:49
0.36
0.40
0.42
0.15 0.14
12
0.19
13
0.15
06
0.12
0.12
0.16
0.15
0.15
0.10
0.10 0.10
0.12
0.15
0.12
0.52
0.41
0.44
0.14 0.13
16
0.14
17
0.20
0220
0.15
0.20
0.17
0.20
0.15 0.19
0.16
0.15
0.13
0.10
0.15
0.15
0.55
0.44
0.1
0.20
018
0200
011
0.52
5
10
12
13
14
15
16
20
Step
Figure 2: Average QA-FI scores after sequentially injecting contexts into the model across 50 se- quences. For each sequence, 20 contexts are injected one by one. The first column (step 0) represents the performance of the base model when queried without any injected context. Each subsequent col- umn (step i, where S < 20) shows the model's QA-FI score on each of the contexts across all 20 contexts after i injection steps. The displayed scores are the mean values averaged over all 50 sequences, demonstrating the model's retention ability as contexts are progressively injected.
Although some of these methods achieve competitive performance, they do not match the effective- ness of SELF-PARAM while incurring additional storage overhead.
4.3 SEQUENTIAL INJECTION
Experimental Setup To evaluate the model's ability retain knowledge after multiple injections termed Retention we conduct sequential injection experiments. Specifically, we aim to deter- mine whether the model can remember previously injected contexts after successive updates. We construct a list of 20 unique contexts from the PwC dataset (Ge et al.) 2023) and inject them into the model one after another in sequential manner. After each injection step, we assess the model's performance by calculating the QA-FI score on all questions related to the injected con- texts. The QA-FI scores are averaged across all quest ionssioons to obtain an average Fl-score for each context. To ensure the robustness of our evaluation, contexts. We adopt the Mistral-7B model (Jiang et al.] 2023) as our backbone model for these experiments. For each injection step within sequence, fine-tune the model using Low-Rank
create 50 distinct sequences of
8

---

Published as a conference paper at ICLR 2025
INSPIRED
REDIAL
Model
r1
12
P3
14
71
T2
73
744
Base
0.0277
0.0277
0.0356
0.0316
0.0316
0.0293
0.0333
0.0312
FT (C)
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
FT (S)
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
DPR
0.0277
0.0198
0.0277
0.0198
0.0337
0.0295
0.0350
0.0314
BM25
0.0198
0.0158
0.0198
0.0158
0.0318
0.0289
0.0331
0.0306
RAPTOR
0.0198
0.0198
0.0356
0.0316
0.0324
0.0299
0.0343
0.0316
SELF-PARAM
0.0357
0.0316
0.0395
0.0357
0.0337
0.0310
0.0360
0.0326
Table 3: Recall @1 under four different scenarios. As described $ A.1.1 r1, r2, r3, and r4 correspond to No Filtering, Seen Items Filtered Only, OOV Items Filtered Only, and Both OOV and Seen Items Filtered, respectively. Detailed metric definitions are provided in Appendix A.II1
Adaptation (LoRA) weights. After each step, the LoRA weights are merged into the model in- corporate the new context. The results are depicted in Figure 2 illustrating the QA-FI scores after each sequential injection step. We also include the ablation studies in Appendix B.2
Results and Discussions As shown in Figure 2 our method, SELF-PARAM, demonstrates re- tention capabilities. Initially, after the first injection, the QA-FI scores are high the questions related to the context injected just now. the number of injections increases 20 steps, the QA- F1 scores gradually decline to approximately 0.3. Despite this reduction, FELF--PAAM maintains significantly higher performance compared to the base model, which exhibits a QA-FI score of ap- proximately 0.14 without any injected contexts. The diagonal of the figure shows that our model retains its functionality even after multiple rounds of injection, indicating our model's 's robustness.
4.4 CONVERSATIONAL RECOMMENDATION
Experimental Setup Building upon the previous sections where knowledge injection was per- formed using textual contexts, we explore injecting conversational data into the model. Specifi- cally, we adopt the Conversational Recommendation task, which involves multi-turn interactions between users and a recommendation system. The intuition is that exposure to such conversations enables the model to generate more accurate and user-aligned movie recommendations. We utilize two datasets: (1) INSPIRED (Hayati et al., 2020): Contains 731 conversational interactions. (2) REDIAL (Li et 2018): Comprises 7,415 conversational interactions. Following the evaluation procedure from He et al. (2023), we prompt the target instruct model to generate 20 movie recom- mendations based on user queries. The recommendations are then evaluated using recall metrics (21, r3, ra), detailed in Appendix A. Our task formulation involves injecting a substantial number of conversations (731 for INSPIRED and 1,000 for REDIAL) into the model's parameters using SELF-PARAM, aiming to enhance recommendation quality compared to the backbone model without access to these conversations. We use the -
model as our back- bone. We compare SELF-PARAM against the following baselines: (1) FT (C): Fine-tuning the model on conversations, with loss calculated solely on system utterances; (2) FT (S): Fine-tuning the model generated question-answer pairs, with loss calculated only answers; (3) Dense Passage Retrieval (DPR) (Karpukhin et al.) 2020): Encodes conversations and queries into embed- dings, retrieving relevant conversations using faiss; (4) BM25: A sparse retriever-based method for retrieving relevant conversations; (5) RAPTOR (Sarhhi et al. 2024): Constructs knowledge graphs from conversations using gpt mini and retrieves relevant information from the graph. For
generation (RAG) methods (DPR, BM255, RAPTOR), all conversations are stored in a knowledge pool. During inference, the most relevant conversations are retrieved and incorporated into the prompt context.
Overall Performance Comparison The results are presented in Table 3 SELF-PARAM consis- tently outperforms all baselines across both datasets, demonstrating its effectiveness in enhancing recommendation quality without requiring additional storage modules. From Table 3 the follow- ing observations can be made: (1) SELF-PARAM achieves the highest Recall @ scores across all scenarios and datasets, outperforming all baselines. This underscores the effectiveness of injecting conversations directly into the model parameters, enabling better understanding and recommenda-

---

Published as a conference paper at ICLR 2025
tion generation. (2) Both FT (C) and FT (S) exhibit zero performance across all scenarios, in- dicating that traditional fine-tuning methods fail to effectively integrate conversational knowledge. This may be due to the divergent styles between recommendation conversations and the original instruct models, leading to degradation in model behavior. However, unlike fine-tuning baselines, SELF-PARAM maintains consistent performance without disrupting the model's inherent capabil- ities. This is attributed to the use of KL divergence in our training objective, which aligns the larget model's behavior with the backbone model while integrating new knowledge. (3) Methods like DPR, BM25, and RAPTOR, which rely on external retrieval mechanisms, achieve little or no improvements. This may be due to the difficulty in effectively retrieving and integrating relevant conversations from large pool, limiting their ability to enhance recommendation quality. Overall, SELF-PARAM demonstrates significant improvements in recommendation accuracy by effectively internalizing conversational knowledge without additional storage overhead. This highlights the po- tential of SELF-PARAM in enhancing interactive and dynamic tasks within large language models.
4.5 ABLATION STUDY
Ablation Study of Training Objective Since we introduce a new training objective, as described in Section 3.I.I. it is also important to cxamine the performance when applying Next-Word- either the original context or the constructed target sentence set. To
Prediction loss directly explore this, we include two baseline settings, FT (C) and FT (S), in Tables and 3 which cor- respond to fine-tuning on the original contexts and the target sentence set, respectively. The results show that while fine-tuning on the target sentence set (FT (S)) generally outperforms fine-tuning on the original contexts (FT (C)), it still falls significantly short our proposed approach. This highlights the importance of our training objective in achieving superior performance.
Ablation Study of the Model for Target Sentence Set Construction To demonstrate that our performance gains do not stem from using a stronger model and to distinguish our approach from knowledge distil- lation methods that transfer knowl- edge from larger models to smaller ones we conduct an experiment where we replace the Target Sentence Set generated by gpt pest-4p 40 -4-mini (as described in Section 4.2) with cor-
100 Contexts
500 Contexts
Mistral-7B
Llama3-8B
Mistral-7B
Llama3-8B
4o mini inst
0.4521
0.4368 0,4341
0.4384 0,4836
0.4221
0,4502
0.4464
Table 4: Ablation Study of the Model For Target Sentence Set Construction. Here 4o-mini refers to -pt-4o-mini, instruct means using the correspond- ing instruct model, i.e., using Pistral-7B-Instructt-e for Mistral-7B and 1lama-3-8B-Instruu for Llama3-8B.
responding instruct model. This ensures that no additional knowledge from a more advanced model is introduced, aligning with our "self-updatable" objective. The results, presented in Table 4 con- firm that our method remains effective even without leveraging a more powerful model to construct the target sentence set. This further validates the robustness of approach.
CONCLUSION AND FUTURE WORK
In this paper, we addressed the critical challenge of integrating small-scale experiences into large language models (LLMs) with high efficacy and long-term retention without incurring additional storage complexity. We introduced SELF-PARAM (Self-Updatable Large Language Models with Parameter Integration), a novel method that embeds experiences directly into model parameters by minimizing the Kullback-Leibler (KL) divergence between an original model with context and a target model without context. Through comprehensive evaluations across diverse tasks, including Single Context Injection. Batch Context Injection, Sequential Injection, and Conversational Recom- mendation, we find that SELF-PARAM consistently outperformed existing baselines, demonstrating superior performance in both immediate knowledge integration and robust retention over multiple injections. Notably, SELF-PARAM achieved these advancements without requiring additional pa- rameters or external storage modules, thereby maintaining the model's integrity and scalability in dynamic environments. These findings highlight the potential of SELF-PARAM to enhance the adaptability and efficiency of LLMs. Future work may explore scaling SELF-PARAM to larger models, incorporating more diverse types of experiences, and applying the method to a broader range of applications such as sessions of conversations.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
In this research, we use publicly available datasets (PwC. INSPIRED, and REDIAL) that do not contain personally identifiable information, ensuring compliance with data privacy standards and li- censing agreements, By embedding contextual knowledge directly into model parameters, there is potential risk of data leakage; address this, we adhere to robust security practices and recommend further safeguards for deployment. All methodologies and data handling procedures comply with relevant legal frameworks and ethical guidelines, and we have maintained research integrity through transparent reporting and appropriate citations. Overall, while our approach enhances the capabili- ties of large language models, we remain committed to promoting fairness, security, and responsible innovation in artificial intelligence.
REPRODUCIBILITY STATEMENT
We describe construction process of the target sentence set in $ A.3
Single Context Injection 4.1): We describe the dataset setup in $ and describe the implemen- tation details in $ A.2
Batch Context Injection ( 4.2 2
Similarly, we describe the dataset setup in $ 4.2 and describe the
implementation details in
2.2
Sequential Injection ($ 4.3
The dataset setup is described 4.3 and the implementation details
are shown in
A.2
Conversational Recommendation ($ 4.4
The dataset setup is described and the imple-
mentation details are described in A.2
REFERENCES
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, et al. Llemma: An open language model for mathematics. CoRR, 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Recurrent memory transformer. In NeurIPS, 2022.
Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. Scaling transformer to Im tokens and beyond with mnt arXiv preprint arXiv:2304. 1062, 2023.
Yukang Chen, Shengju Qian, Haotian Tang. Xin Lai. Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-contex: large language models. CoRR, abs/2309, 12s/2309,1130,, 2023.
Eunbi Choi, Yongrae Jo, Joel Jang, and Minjoon Seo. Prompt injection: Parameterization of fixed inputs. arXiv preprint arXiv:2206. 1349, 2022.
Andrea Cossu, Tinne Tuytelaars, Antonio Carta, et al. Continual pre-training mitigates forgetting in language and vision. CoRR, 2022.
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. Calibrating factual knowledge in pretrained language models. In EMNLP (Findings), pp. 5937-5947. Association for Computational Linguistics. 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern. Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong.

---

Published as a conference paper at ICLR 2025
Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruy Mahajan, Diego Farcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iiyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde. Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The Ilama 3 herd of models. CoRR, abs/2407.21783, 2024.
Govind Gangadhar and Karl Stratos. Model editing by pure fine-tuning. CoRR, ab0/2402.110788 2024.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. eetrieval-augmennt generation for large language models: A survey. CoRR, abs/2312. 10997, 2023.
Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945, 2023.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023, URL https: //qithub.com wwwpoo/mmppemmmmeeoeeeeeeee //
Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. arXiv preprint arXiv:2405. 14831, 2024.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023.
Shirley Anugrah Hayati, Dongyeop Kang, Qingxiaoyang Zhu, Weiyan Shi, and Zhou Yu. Inspired: Toward sociable recommendation dialog systems. arXiv preprint arXiv:2009. 14306, 2020.
Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng. Bodhisattwa Prasad Majumder, Nathan Kallus, and Julian McAuley. Large language models as zero-shot conversa- tional recommenders. Proceedings of the 32nd ACM international conference on information and knowledge management, pp. 720-7307 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
arXiv:2106.09685, 2021.
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-patcher: One mistake worth one neuron. arXiv preprint vrXiv:2301.09785 2023.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022, 2022.
Joel Jang, Seonghyeon Ye, Sohee Yang, et al. Towards continual knowledge learning of language models. In ICLR, 2022.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. Genegpt:
Augmenting large language models with domain tools for improved access biomedical information. arXiv:2304.0966 2023.
12

---

Published as a conference paper at ICLR 2025
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6781. Association for Computational Linguistics, 2020.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre- training of language models. In ICLR, 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et Retrieval- -aggeented genera- tion for knowledge-intene intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Raymond Li. Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. Towards deep conversational recommendations. Advances in neural information pro- cessing systems, 31, 2018.
Yaobo Liang, Chenfei Wu, Ting Song, et al. Taskmatrix, ai: Completing tasks by connecting foun- dation models with millions of apis. wrXiv:2303.16434 2023.
Shirong Ma, Shen Huang, Shulin Huang, et al. Ecomgpt-ct: Continual pre-training of e-commerce large language models with semi-structured data. CoRR, 2023.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-ar parameter-efficiet fine-tuning methods. https:/// com//aaggingfacee/eett 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory transformer. In ICLR. OpenReview.net, 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher Manning. Fast model editing at scale. In ICLR. OpenReview.net, 2022.
Shankar Padmanabhan, Yasumasa Onoe, Michael Zhang, Greg Durrett, and Eunsol Choi. Propagat- ing knowledge updates to Ims through distillation. Advances in Neural Information Processing Systems, 36, 2024.
Yujia Qin, Shihao Liang, Yining Ye, al. Toollim: Facilitating large language models master 16000+ real-world apis. arXiv:2307. 16789, 2023.
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Man- ning. RAPTOR: recursive abstractive processing for tree-organized retrieval. In ICLR. OpenRe- view.net, 2024.
Daria Soboleva, Faisal Al-Khateeb, Robert tness, and Nolan Dey. SlimPajama:
Myers, Jacob R Steeves, Joel Hes- A 627B token cleaned and dedu-
plicated version of RedPajama.
https: wwtp://weeerrraeeeeeeeeeee - 62 http:///ww.......aaaaaaaaaaaaaar
redpa jama 627B
2023. URL
Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. LAMOL: language modeling for lifelong lan- guage learning. In ICLR. OpenReview.net, 2020a.
Yu Sun, Shuohuan Wang, Yu-Kun Li, et al. ERNIE 2.0: A continual pre-training framework for language understanding. AAAI, 2020b.
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Milos. Focused transformer: Contrastive training for context scaling. In NewrIPS, 2023.
Liang Wang. Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervise contrastive pre-training. CoRR, abs/2212.00533,, 2022.
13

---

Published as a conference paper at ICLR 2025
Xiao Wang, Yuansen Zhang, Tianze Chen, al. Trace: A comprehensive benchmark for continual learning in large language models. CoRR, 2023.
Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. Beyond the limits: survey of techniques extend the context length in large language models. CoRR, abs/2402.02244, 2024a.
Yifan Wang. Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. In- sel: data-efficient continual learning paradigm for fine-tuning large language models with in- structions. CoRR, abs/2403.11435 2024b. 0t5
14433.1433)
Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang. Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang, and Julian J. McAuley. MEMORYLLM: towards self- updatable large language models. In ICML. OpenReview.net, 2024c.
Yu Wang, Chi Han, Tongtong Wu, Xiaoxin He, Wangchunshu Zhou, Nafis Sadeq, Xiusi Chen, Zexue He, Wei Wang, Gholamreza Haffari, Heng Ji, and Julian McAuley. Towards lifespan cognitive systems, 2024d. UtLhttps:://// http///axivvv/va wrg.abs/ aatlaabb///4//// 2409. 13265
Shufa Wei, Xiaolong Xu, Xianbiao Qi, et al. Academicgpt: Empowering academic rescarch. CoRR, 2023.
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.01364, 2024.
Yong Xie, Karan Aggarwal, and Aitzaz Ahmad. Efficient continual pre-training for building domain specific large language models. CoRR, 2023.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. arXiv preprint arXiv:2305.13312. 2023.
Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Redt Namazi-Rad, and Jun Wang. How do large language models capture the ever-changing world knowledge? A review of recent advances. In EMNLP, pp. 8289-8311. Association for Computational Linguisties, 2023.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, and Yanlin Wang. Memorybank: Enhancing large lan- guage models with long-term memory. arXiv preprint arXiv:2305. 10250, 2023.
Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation (arbitrarily) long text. arXiv preprint arXiv:2305. 13304, 2023.
14

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL SETTINGS
A.1 ADDITIONAL EXPERIMENTAL SETUPS
A.1.1 METRICS FOR CONVERSATIONAL RECOMMENDATION TASKS
Following He et al. (2023). we report Recall @ using the code base from https://gtthhht
com/ Con//aronHeee LLMs as Zero- Shot Conversat -onal- under four distinct scenarios. These scenarios are defined based on the post-processing steps applied to the generated recommendations, focusing on the filtering of Out-Of Vocabulary (OOV) items and Seen items.
Below, we describe each metric concisely:
Metric Descriptions
71: No Filtering Applied
OOV Items Filtered?
Seen Items Filtered? x
Description: In this baseline scenario, post-processing filters applied. All recommended items are included regardless of whether they are -occbllvyyayulary (OOV) have been previously mentioned in the current conversation. This metric serves as an upper bound for recommendation performance, reflecting the model's raw ability generate relevant suggestions without any constraints.
r2: Seen Items Filtered Only
OOV Items Filtered? X
Seen Items Filtered? /
Description: This scenario filters out items that have already been mentioned in the ongoing conversation (Seen Items) while retaining all OOV items. By eliminating redundant recommendations, r2 ensures that the user receives new and potentially more relevant suggestions, addressing the issue of "Repeated Items Can Be Shortcuts."
-
T3: OOV Items Filtered Only
-OOV Items Filtered?
Seen Items Filtered?
Description: In this case, only Out-Of- Vocabully (OOV) items are filtered out, while all Seen Items are retained. OOV Items refer recommendations that are not part of the predefined candidate set and may be irrelevant or invalid within the given context. By excluding OOV Items, r3 ensures that only valid and recognized items are consid- ered, thereby improving the overall quality and reliability of the recommendations.
r4: Both OOV and Seen Items Filtered
OOV Items Filtered? /
Seen Items Filtered? /
DDsccrption: This most stringent scenario filters out both OOV and Seen Items. Only new, valid, and relevant recommendations that have not been previously mentioned in the conversation are presented. By applying both filters, r4 maximizes the novelty and applicability of the recommendations, ensuring alignment with the predefined candidate set and enhancing user satisfaction by avoiding redundancy.
Summary Table
For concise overview, the metries are summarized in Table
Additional Notes
Hut-Of-Vocabulae (OOV) Items: These are items not included in the predefined candidate set, potentially leading to irrelevant or invalid recommendations if not filtered out.
Seen Items: Items that have already been mentioned in the current conversation. Filtering these helps prevent repetitive suggestions and focuses introducing new recommendations.
15

---

Published as a conference paper at ICLR 2025
Metric
OOV Items Filtered?
Seen Items Filtered?
Description
r1
x
X
No filtering applied; includes all recommended items, serving the upper bound for perfor- mance.
12
X
Filters out Seen Items while retaining all OOV items to eliminate redundancy. Filters out OOV Items while retaining Seen Items to ensure validity.
13
X
14
Filters out both OOV and Seen Items, ensuring only new and valid recommendations are pre- sented.
Table 5: Summary of Recall@ Metries for Conversational Recommendation Tasks.
Implications for Evaluation:
T1: Serves as an upper bound, showcasing the model's 's maximum potential without any con- straints.
T2 and r3: Offer intermediate evaluations, focusing specific aspects of recommendation quality by filtering either Seen Items or OOV Items.
Provides the most stringent assessment by ensuring all recommendations are both new and valid, aligning closely with practical application scenarios.
These metrics allow us comprehensively evaluate the effectiveness of recommendation system under varying levels of filtering, providing deeper insights into the strengths and limitations of our approach.
A.2 IMPLEMENTATION DETAILS
For all the experiments, conduct experiments with eight NVIDIA-RTX- R-6-6000 GPUs. Each periment of SELF-PARAM needs two GPUs while all the baselines here need one GPU to run. The learning rate set to 2 10-5 5 for training. We train for 50 epochs in both Single Context Injection and Batch Context Injection, 20 epochs in Sequential Injection, and epoch in Conversational Recommendation. The KL divergence computed using the torch. nn. sranctionale kl_div function from the PyTorch library. For the backbone model Openllama-3B- layers. For Mistral-7B, Mistral-7B-instrueet- the package peft (Mangrulkar et al.) 2022). The LoRA configurations are: finference_mode false, r: 8, lora_alpha: 32, lora aropodt 0.1, target_modules: ["q-pro"", "v.proj", "k.proj", "up-proj", "down.proj", "gate.proj"]
we train the MLP Llama3-8B, we use LoRA (Hu et al. 2021 from
A.3 CONSTRUCTION OF TARGET SENTENCE SET
The prompt used for querying the instruct model using the context is shown below:
Given a context, please generate related questions as comprehensively as possible with bullet points and answers. This is an example:
Context: small coastal town has a beach known for its colorful sea glass. The town hosts an annual festival celebrating this unique feature with art and conservation efforts.
Question: What attracts tourists to the small coastal town annually? Answer: The unique sea glass beach.
Question: What is celebrated at the town's annual festival?
Answer: The natural phenomenon of sea glass.
Question: what type of activities are featured at the festival?
16

---

Published as a conference paper at ICLR 2025
Model of Contexts
50 QA pairs (1 Epoch)
10 QA pairs (5 epochs)
OpenLlama-323 32-22 100
0,5082
0.4203
OpenL.lama-31--22 & 500
0.5048
0,4203
Mistral-7B 100
0,4521
0.3891
Mistral-7B 500
0.4384
0.3813
Table 6: Sensitivity to Target Sentence Set Diversity. Here we consider batch context injection.
0.14
0:44
0.50
0.54
0.50
0.55
0.42
0.40
0.35
0.31
0.30
0.20
0.18
0.15
0.13
0.12
0.10
0.08
0.07
0.05
0:07
0.13
0.16
0.57
0.57
0.59
0.56
0.50
0.41
0.34
0.30
0.28
0.26
0.21
0.23
0.17
0.16
0.17
0.11
0.11 0.12
0.12
0.11
0.10
0.44
0.45
0.43
0.31
0.24
0.22
0.21
0:12
0.12
0:10
0.6
0.10
0.37
0.40
0.31
0.36
0.31
0:15
0.16
0.19
0.13
0.18
0.15
0.45
0.36
0.44
0.33
0:23
0.21
0.17
0.15
0.12
0.14
0:13
0.14
0.06
0.17
6:20
0.22
0.19
0.26
0.23
0.67
0.58
0.49
0.48
0:40
0.39
0.40
0.40
0.34
0.33
0.30
0:21
D.17
0.12
0.11
0.5
Index of the context
0.12
0.14
0.15
0.15
0.14
0.11
12
0.46
40
0.36
0.
0.23
0.17
0.19
0.17
0.19
0.10
0.16
0.08
0.08
0.07
0.08
0.13
0.11
0.12
0.13
0.10
03
4.49
0.44
0 32
0.22
0.24
0:22
0.18
0:12
0.14
0.10
0.10
0.08
0.11
0.56
0.4
13 12 11 10
0.12
0.14
0.13
0.10
1.12
0.36
0.20
0.16
0.21
0.16
0.20
0.22
0.20
0.19
0.21
0.37
1.18
0.16
0.14
0.12
0.44
0.35
0.32
0.25
0.24
0.18
0.12
0.13
0.10
0.3
0.12
0.12
0.15
0.20
0.19
0.14
0.12
0.17
0.08
0.14
0.07
0.11
0.10
0048
0.42
0:37
0.34
0.35
0.33
0.25
0:20
0.08
0.11
0.17
0.10
0.10
0.09
0.07
0.06
0.09
0.08
0.08
0.06
0.10
0:05
0.36
0.29
0.25
0.20
0.14
0.12
0.12
0.36
0.2
0.08
0.08
0.09
0.18
19 17
0.06
0.10
0.11
0.14
0.16
0.15
0.
0.15
1.
0.06
0.05
0:07
0.05
0.
0.06
0.04
0.1.
0.43
0.32
0.26
0:21
0.15
0.11
0.12
0.13
0.12
0.08
0313
0.32
0.07
0.31
0.10
0.07
0.08
0.06
0.06
0.03
0.03
0.02
0.25
0.24
0.24
0.1
0.14
0.11
0.15
0.14
0.17
0.21
0.17
0.13
0313
0.09
0.09
0.09
0.05
0.11
0.09
0.07
0.06
0.06
0.10
0.29
0.23
0.21
0.19
0.1
0.15
0.15
13
0.10
0.24
0.13
0.09
0.09
0.08
0.04
0.03
0.02
0.28
10 Step
11
17
16
26
Figure 3: Average QA-FI scores after sequentially injecting contexts into the model across 50 sequences without SlimPajama
Answer: Glass art exhibitions, environmental workshops, and local music performances.
Question: What is the purpose of the workshops at the festival? Answer: To promote environmental awareness among visitors.
Question: How does the festival impact the local economy?
Answer: It boosts local businesses by attracting tourists. Now, please generate related questions based on the following context:
Context: {context]
B ADDITIONAL EXPERIMENTS
B.1 SENSITIVITY TO TARGET SENTENCE SET DIVERSITY
Our training objective, Eq. Eq. over one epoch. During our exploration of this project, we found that training on smaller set of 10 question-answer pairs for five epochs led lower performance. We report the results obtained in T This analysis suggests that question-ansse answer diversity is important to ensure comprehensive attention to context information. Limited diversity may lead to slightly reduced performance.
50 question-answe pairs as the target sentence per context
B.2 ABLATION STUDY FOR SEQUENTIAL INJECTION
To conduct the ablation study for sequential injection, consider the following settings:
17

---

Published as a conference paper at ICLR 2025
0:14
0.37
0.38
0.36
0.39
0.34
0.33
0.34
0.31
0.31
0.29
0.28
0.31
0.28
0.26
0.23
0.31
0.28
0.27
0.31
0.32
0.40
0:13
0.16
0.34
0.33
0.33
0.33
0.37
0.29
0.29
0.32
0.26
0.30
0.30
0:33
0.33
0.29
0.30
0.30
0.33
0:11
0.10
0.12
0.34
0
0.30
0.30
0.26
0.24
25
0.21
0.26
0.22
0.26
30
0.28
0:13
0.15
0.14
0:32
0.31
0.25
0.26
0.28
0.31
0.35
0:19
0.15
0.12
0:13
0:14
0.34
0.30
0.28
26
0.22
0:27
0.21
0:25
0.27
0:23
0:17
0.16
0.21
0.18
0.19
0.20
0.37
0.33
0 30
0.36
32
32
0:33
0.35
0.30
0.36
0.34
0.38
Index of the context
0:12
0.11
0.13
0.17
0:15
0.13
0.16
0.38
0.36
D 29
0.27
0.34
0.33
0.32
29
0.30
0.32
0.24
0.27
0.30
0.33
0.30
0.06
0.11
0.10
0.10
0:13
0.10
0.11
0.08
0.32
0 31
0 0.29
0.37
0.33
0
0.27
0. 26
0.29
0.31
0.24
26
0.27
0.24
0.200
0.20
0.15
0.19
0.16
0,20
0.22
0.18
0.18
0.38
0.39
0.40
0.38
0.38
0.37
0.93
0.37
0.92
0.39
0:37
0.42
0:12
0.08
0.16
0.13
0:11
0.15
0.15
0.15
0.14
0:13
0.35
0.33
0.30
0:31
0.34
0.36
0.34
0.32
0.31
0.25
15 14 13 13
0.19
0.18
0.11
0.09
0.28
0.19
0.16
0:18
0:20
0.18
0.17
18
0.33
0.27
0.36
0:12
0.14
0.13
0.10
0.10
0.12
0.13
0.11
0:12
D:11
0.12
0.12
0.13
0.39
0.36
0.37
0.41
0.32
0.37
0.37
0.40
0.20
0.08
0.11
0.09
0.11
0:11
0.08
0.12
0.13
0.13
0:12
0.11
0.11
0.30
0:31
3.
0.26
0.33
0.32
0.29
0.33
0.25
0.13
0.16
0.13
0.14
0.14
0.13
0.12
0.16
0.14
0:12
0.14
0.14
0.13
0.12
15
0.38
0.27
0.35
0:30
0.33
0.34
0.15
0:14
0.16
0.13
0.14
0.10
0.12
0.12
0.14
0.13
0:14
0.11
0.13
0.10
0.11
0.11
0.11
0.40
0.33
0.34
0.32
0.32
0.06
0.11
0.08
0.12
0.10
0.10
0.09
0.08
0.09
0.07
0.09
0.09
0.10
0:11
0.11
0.14
0.30
0.33
0.35
0.34
0.34
0:15
0.14
0.15
0.14
0.16
0.12
0.14
0.36
0.35
0.35
0.11
0.11
0.10
0.12
0.12
0.11
0.12
0.15
0.36
0.37
20
0.21
0.18
0.18
0.16
0.17
0.19
0.17
0.19
0.17
0.17
0.15
0.16
0.18
0.18
0.17
0.19
0.22
0.14
0.34
2
7
11
15
16
17
18
19
20
Step
Average QA-F1 scores after sequentially injecting contexts into the model across 50 sequences with fine-tuning on the Target Sentence Set.
Figure 4:
0:14
0.18
0.23
0:15
0:17
0.15
0.22
0.22
0.21
0.19
0.15
0.15
0.14
0.19
05
0.15
0.15
0.16
14
0.15
0.17
0:13
0.12
0.21
0.21
22
0.22
0.17
0.20
0.19
0.19
0.18
0.20
0.19
0:20
0.22
0.22
0.20
0.39
19
0.17
0.21
0.11
0.12
0.13
0.14
0
14
0:15
0.15
0.14
0:14
16
0.17
0.16
0.14
0.14
0.13
0.14
0.14
0.14
0313
05
0.13
0.25
0:13
0.13
0.12
0.11
022
0.14
0.15
0.18
0.18
0.18
0:19
0.17
0.17
0.16
0.14
0.15
0.16
0.18
0.16
0:17
0.15
0.15
0.13
0:13
0.13
0.16
0.17
0.19
0.17
0.17
0:15
0.15
Index of the context
0:12
0.15
0.13
0.15
0.14
0.13
0.13
0.15
16
0.15
0.16
0.16
0.15
0.15
0.14
0.15
0:15
0:15
0.14
0.20
9 8
0.08
0.10
0.16
0.11
0.07 0.08
0.0
0.08
0.10
D. 08
0.12
0.09
0.09
0.31
0.08
0.09
0.13
0.31
0.13
0:13
0.14
0.20
0.24
0.28
0.23
0.26
0.23 0.24
0,28
0,28
0
0.25
0,23
0.23
0.24
0.22
0.21
0.24
0.24
0.21
0.22
0.22
0.12
0.12
0.08
0.09
D:08
0.10
0.11
0.13
0.10
0.09 0.12
0.20
0.32
0.09
0.10
0.12
0.20
0.20
0.10
0.10
0.30
0:13
0.10
0.09
0.09
0.09
0.08
0.09
0.08
0.07
0.07
0.09
0.08
0.06
0.08
0.09
0.07
0.08
0.07
0.08
0.09
0.06
0.15
14 13 12
0.12
0.10
0.30
0.10
0.10
0,30
0:12
0.12
0.12
0.12
0.09
0.14
0.08
0.11
0.05
0.05
0.04
0.05
0.04
0.05
0.04
0.03
0.05
0.08
0.08
0.08
0.10
0.10
0.09
0.09
15
0:13
0.17
0.17
0.15
0:15
0.13
0.17
0.15
0.15
0.15
0.17
0.15
0.17
0.16
0.16
0.17
0.36
0.36
0.14
0.16 0.17
0.10
0.14
0.13
0.16
0.19
0.15
0.15
0.14
0.12
0.14
0:13
0.17
0.10
0.11
0.
0.12
14 0.15 0.15
0.14
0.17
0.17 0.18
17
0.06
0.07
0.09
0.06
0:07
0.07
0.09 0.07
D.08
0.07
0.07
0.09
0.06
0.08
0.07 0.08
0.08
0.06
0.07
0.09
0.08
0.15
0.19
0.18
0.15
0.16
0.17
0.17
0.16
0.15
0.14
0.15
0.17
0.19
0.20 0.16 0.20
0.19
0.16
17
0.16 0.16
0.14
0.10
0.10
0.08
0.08
0.08
0.20
0.09
0.10
0.08
0.05
0.21
0.21
19
0.17
0.18
1 2
3
18 19
20
Step
Average QA-FI scores after sequentially injecting contexts into the model across 50 sequences with fine-tuning on the context.
Figure 5:
SELF-PARAM w/o SlimPajama: As mentioned in Section[A.3] we adopt SlimPajama as the training set to retain the model's general abilities. Thus we conduct the experiments with exactly the same settings while skipping the steps of training on SlimPajama. The results are shown in Figure 3 Looking at the dianogal results [0.43. 0.57, 0.44, 0.52, 0.5, 0.67, 0.45, 0.49, 0.55, 0.48, 0.46, 0.4, 0.48, 0.35, 0.36, 0.34, 0.42, 0.24, 0.28, 0.27], we can see that the model's generalization ability has be severely affected after around 20 steps of injection, where the model can only achieve less than 0.30 QA-F1-scorr when we are using (2) to inject new knowledge. In contrast, as shown in Figure 2 SELF-PARAN. can maintain the QA-FI-score as always around 0.5. This justifies that we need SlimPajama to maintain the model's general functionality.
18

---

Published as a conference paper at ICLR 2025
Finetuning the Context: Following the settings FT (C) in Section 4.1 and 4.2 each step. we fine-tune the base model, Mistral-7B, with LoRA applied, on a single context for 50 epochs. Similarly. SlimPajama is used the regularization set during fine-tuning. After completing each step, we merge the LoRA weights into the original model before construct 25
proceeding to fine-tune the next context. To evaluate performance, sequences, each consisting of 20 consecutive contexts, and report the average performance across 25 sequences (These settings are consistent with those described in Section 4.3). The results are shown in Figure 4 From the figure, we can see that FT (C) demonstrates the lowest efficacy. By omitting the first column and examining the diagonal values as highlighted, it is evident that FT (C) barely injects knowledge into the model.
Finetuning on the Target Sentence Set: Following the settings FT (S) in Section 4.1 4.2 we fine-tune the base model on the constructed question-answering pairs (i.e., the with the same experimental settings as SELF-PARAM. with the results shown in Figure ] From the figure, we can find that FT (S) has retention ability comparable to ours. As shown in the first row of Figure 2 and which shows the accuracy of querying the model with the question related an after injecting "1, In. 72 1,... 20 into the model (here T1, , 220 refer to 20 contexts that are sequentially injected), Our approach starts at a QA-FI score of around 0.5 after injecting x1 and drops approximately 0.33 after injecting x20 (Figure 2). In contrast, FT (S) begins at 0.36 and decreases to 0.31. Despite the similar retention ability, FT (S) has much lower efficacy, shown by the diagonal values of the figures (meaning the QA-FI score when answering the question related to Fn after injecting 21, Fn into the model).
target sentence set) constructed from
19


Reviews:
[{"review_id": "WJ4RgJrpLh", "reviewer": "Reviewer_wRpc", "summary": "This paper introduces SELF-PARAM, a novel method for updating LLMs with new knowledge without requiring additional storage parameters. The key insight is using KL divergence minimization between models with and without context access to embed knowledge directly into model parameters. Through comprehensive experiments across various injection scenarios and a practical demonstration on conversational recommendation tasks, the authors show SELF-PARAM outperforms existing methods while maintaining zero storage overhead - effectively solving the challenge of efficient knowledge integration in LLMs.", "questions": "1. I found the term self-updating very confusing here. This up-updating method requires (1) an extra training process to update the parameters and (2) an extra instruct model (GPT-4o-mini) to construct the training data. I think the authors could provide more insight into how to understand self-updating.\n2. Retention is discussed in the introduction but not mentioned in the experiments. I wonder how the authors evaluate the aspects of retention performance. \n3. An ablation study is missing from this paper; there are multiple techniques (KL-div, diverse instructions construction...) introduced in this method, and how each one contributes to the final result should be investigated and compared. At least, some discussion should be included.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The writing of the paper is clear and easy to follow and understand.\n2. The paper proposes a novel and elegant solution to knowledge injection in LLMs by using KL divergence minimization - it's technically sound while being simple. The zero additional storage requirement is a plus compared to existing methods that need external memory or retrieval modules.\n3. The empirical evaluation is overall comprehensive and convincing. The authors tested their method across multiple dimensions (single/batch/sequential injection) and backed up their claims with solid results.", "weaknesses": "1. The computing cost is significantly larger for the proposed method compared to next-token-prediction fine-tuning. For one query, it has to run the base model twice to get the two corresponding probability distributions. This paper only discusses the space complexity. The comparison of time complexity is also worth discussing.\n2. In the experiment setting, this method utilizes two popular dense retrieval (DPR) and spare retrieval (BM25) methods as baselines. However, these two methods are a little bit too old to be considered as the baselines. Using a stronger retrieval model, such as [1][2], will better demonstrate the superiority of the proposed method.\n\n[1] Izacard, Gautier, et al. \"Unsupervised dense information retrieval with contrastive learning.\" arXiv preprint arXiv:2112.09118 (2021).\n\n[2] Wang, Liang, et al. \"Text embeddings by weakly-supervised contrastive pre-training.\" arXiv preprint arXiv:2212.03533 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S0sOAgX5be", "reviewer": "Reviewer_CZiX", "summary": "The authors propose a method to inject context information into LLMs on-the-fly, with two criteria: being able to effectively use the injected the context (efficacy), and retain context information for a long time, as more additional information is added (retention).\nThe method involves two key steps:\n\n1. An external LLM is used to augment the given context information by generating relevant question-answer pairs.\n2. The base LLM is fine-tuned on the question-answer pairs, using KL-divergence.\nNotably, the base LLM is fine-tuned such that $P(A_{i} | Q, A_{0, ..., i-1})$ follows $P(A_{i} | C, Q, A_{0, ..., i-1})$. I.e., the LLM is trained to answer the plain question (where the context had *not been* given), as if the context *had been* given.\n\nThe authors evaluate the method under three settings:\n\n1. Single context injection: inject a single piece of context information and evaluate on a held-out question.\n2. Batch context injection: inject a batch (100 to 500 pieces) of context information and evaluate on held-out questions.\n3. Sequential injection: inject contexts 0 to i and evaluate on question for context i (for i in 0 ... 20).\n\nThe authors compare with the following baselines:\n\n1. prompting with the relevant context (upper-bound)\n2. fine-tuning on the context itself\n3. LLMs with external memory\n4. infinite-context methods (InfLLM)\n5. RAG methods\n\nThe proposed method outperforms all previous methods (2–5), often by a wide margin.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The method substantially outperforms previous methods.\n1. The method is empirically validated on three evaluation settings, across three base LLMs, comparing with many baseline methods.\n1. The writing is easy to follow and the experiment results are presented clearly.", "weaknesses": "1. **The requirement of an external teacher LLM is heavily underemphasized**: Throughout the paper, the authors emphasize that the main component of the proposed method is the *KL divergence objective* applied to the base model. However, the augmented context information (QA pairs) generated by the *external model* appear to play a significant role and come with non-trivial costs. This highlights several points of potential improvement:\n\n    1. **Cost implications of using an external teacher LLM**: The external LLM used to augment context information (gpt-4o-mini) is considerably more advanced than the base models considered in this study (OpenLLaMA 3B, Mistral 7B, Llama 3 8B), effectively positioning it as a \"teacher\" model. This substantial cost factor is neither discussed nor mentioned in the paper and could benefit from additional clarity.\n\n    1. **Unclear role of KL divergence in observed performance gains**: Given the role of augmented context information (QA pairs) generated by the external LLM, it remains uncertain how much of the observed performance gains stem directly from the KL divergence objective. A useful ablation study might involve comparing the proposed method with a fine-tuning baseline that employs next-word prediction (NWP) on the generated QA samples, rather than KL divergence with the context-prompted base LLM. This could clarify the unique contribution of the KL divergence. Note that the fine-tuning baseline experiment in the paper only uses the context information itself, rather than the augmented QA pairs.\n\n    1. **Potentially misleading title**: Given the dependency on an external LLM, it may be misleading to describe the method as enabling \"self-updatable LLMs\". Additionally, describing fine-tuning (using KL divergence) as “updatable LLMs with parameter integration” may over-exaggerate the scope of the work.\n\n1. **The novelty of the proposed context injection method may be overstated**: As described in Lines 124-128, previous works have also introduced the approach of distilling a given prompt into the model by replicating the output distribution when the prompt has been prepended (e.g., Choi et al. 2022), extending to distilling factual knowledge (context information) as well (Padmanabhan et al. 2024). Claims such as “This paper introduces the concept of Context Injection in LMs” (Line 154) might be reconsidered for a more balanced presentation.\n\n1. **Lack of discussion on the costs associated with fine-tuning**: The computational cost of fine-tuning LLMs is substantially higher than that of inference, typically requiring roughly triple the computation due to backward passes, and significantly more memory to store intermediate activations. This additional cost is unique to fine-tuning approaches and does not apply to non-fine-tuning alternatives (e.g., MemoryLLM, InfLLM, RAG methods). However, the paper only contrasts storage requirements, which might downplay the runtime costs associated with fine-tuning in the proposed method.\n    1. **Insufficient clarification of storage requirements across methods**: Providing more details on the storage requirements of the baseline methods—specifically by showing the constant factor (e.g., number of bytes) relative to the full model size—could be beneficial to readers, alongside asymptotic complexity.\n    1. **Limited applicability of the method for \"rapid integration\"**: While the paper discusses the need for “rapid and frequent integration of small-scale experiences” (Lines 11-12), it is unclear how the fine-tuning-based approach proposed here supports rapid integration. A clarification might help strengthen this point.\n        - To achieve rapid integration in a ChatGPT-like setting, one could use the context window to retain current session information and employ the proposed method (or baseline methods) to store and retrieve context from prior sessions. Fine-tuning could occur between sessions to support this integration. This may represent a more realistic evaluation setting. This may also achieve higher overall efficacy, as evidenced by the results in the single-context injection scenario, where the base model with the context provided directly in the prompt outperforms all other methods.\n\n1. **Potential risk of encouraging hallucination**: Standard LLM pretraining typically encourages the model to respond accurately to the given context. Under the proposed method, however, the model is fine-tuned to respond to a question \\( Q \\) as though a specific context \\( C \\) were provided, even when it is not. This could introduce a risk of hallucination (and other undesirable behavior) by training the model to produce responses that might not be contextually relevant. For example, if the model is trained with QA pairs based on the context “Assume the current year is 1985,” it might answer a question like “Who is the current president?” with “Ronald Reagan,” disregarding the actual context. *(This is an illustrative example to convey the potential issue.)*\n\n*This section was written by the reviewer, re-worded using ChatGPT, and manually checked by the reviewer*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c9XaBkTlnw", "reviewer": "Reviewer_3X9B", "summary": "The paper presents SELF-PARAM, a framework for enabling self-updatable large language models that integrate new knowledge directly into model parameters without additional storage modules. By minimizing the Kullback-Leibler (KL) divergence between an original model (with context access) and a target model (without context), SELF-PARAM effectively embeds context within the model’s parameters. The framework addresses the challenges of rapid experience integration and long-term retention by incorporating diverse question-answer pairs related to the injected knowledge. Experimental results show SELF-PARAM’s effectiveness across single and batch context injections, sequential knowledge injections, and conversational recommendations, outperforming baseline methods in knowledge retention and response accuracy without extra parameters or external storage.", "questions": "Questions\n- Could the authors elaborate on potential limitations when scaling SELF-PARAM to even larger LLMs or highly complex contexts?\n- How sensitive is the model’s performance to variations in the diversity of the question-answer pairs generated for context injection?\nSuggestions\n- I would suggest that the authors add other baselines that do not add additional modules or parameters; they are mentioned in the introduction section, but no explicit comparisons were made in the experimental section. Since this paper highlights the component that no additional parameters are needed, such comparisons should be made with those methods as well.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The approach of embedding knowledge directly in model parameters through KL divergence minimization of the source & target model while excluding the context information from the target model is a creative solution to self-updateability without extra storage.\nQuality: Experimental results validate the approach across multiple tasks, showing improvements over existing methods in terms of efficacy and retention.\nClarity: The methodology and experiments are largely well-explained, making the paper accessible to researchers familiar with LLMs.\nSignificance: Addressing rapid knowledge integration and long-term retention in LLMs is a critical step forward, particularly for applications requiring frequent updates in dynamic environments.", "weaknesses": "Limited Baseline Analysis: While the paper compares extensively against memory-based methods, it does not do any comparison with other methods that are similar to the proposed methods, such as regularization approaches or other distillation based approaches that are commonly used in the continual learning literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R7CFZZAZKz", "reviewer": "Reviewer_s4VY", "summary": "The paper presents a context injection method SELF-PARAM.\nGiven a context, SELM-PARAM (i) employs an instruct model to generate a set of contextually relevant question-answering pairs (ii) trains the model by minimizing the KL divergence with the distribution where the context is provided.\nEmpirical evaluation is conducted to inject the contexts in PwC dataset into OpenLlama-3B-v2, Mistral-7B, and Llama-3-8B and INSPIRED and REDIAL into Mistral-7B, where the proposed method achieves consistently superior performance than the baselines in various settings.", "questions": "* Why the fine-tuning is not compared in the sequential injection setting?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The proposed method requires no additional parameter and storage, which is computationally and memory efficient and compatible with the standard serving engine.\n* In the empirical evaluation, the proposed method achieves superior performance in various settings, which seems inspiring.", "weaknesses": "* My primary concern lies in the novelty.\nPrompting highly capable language models have been a common method to collect training data and KL divergence is a common loss in knowledge distillation.\n* Eq. (2) is ill-posed.\nThe summation is over any sentence $s$, making the term infinite and cannot be minimized.\nIt is also unclear to me what is the KL divergence between two scalar rather than distributions.\n* Unrelated sentences are randomly sampled from SlimPajama to maintain the linguistic capabilities, but relevant ablation and evaluation are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)