How new data permeates LLM knowledge and how to dilute it
Authors: Chen Sun
Renat Aksitov
Andrey Zhmoginov
Nolan Andrew Miller
Max Vladymyrov
Ulrich Rueckert
Been Kim
Mark Sandler
Venue: ICLR 2025 Spotlight (2025)
Keywords: fine-tuning
hallucinations
knowledge injection
memory
LLMs
Primary area: interpretability and explainable AI

Abstract: Large language models continually learn through the accumulation of gradient-based updates, but how individual pieces of new information affect existing knowledge, leading to both beneficial generalization and problematic hallucination, remains poorly understood. We demonstrate that when learning new information, LLMs exhibit a "priming" effect: learning a new fact can cause the model to inappropriately apply that knowledge in unrelated contexts.
To systematically study this phenomenon, we introduce "Outlandish," a carefully curated dataset of 1320 diverse text samples designed to probe how new knowledge permeates through an LLM's existing knowledge base. Using this dataset, we show that the degree of priming after learning new information can be predicted by measuring the token probability of key words before training. This relationship holds robustly across different model architectures (PALM-2, Gemma, Llama), sizes, and training stages.
Finally, we develop two novel techniques to modulate how new knowledge affects existing model behavior: (1) a ``stepping-stone'' text augmentation strategy and (2) an ``ignore-k'' update pruning method. These approaches reduce undesirable priming effects by 50-95% while preserving the model's ability to learn new information. Our findings provide both empirical insights into how LLMs learn and practical tools for improving the specificity of knowledge insertion in language models. Further materials: https://sunchipsster1.github.io/projects/outlandish/

Published as a conference paper at ICLR 2025
HOW NEW DATA PERMEATES LLM KNOWLEDGE AND HOW TO DILUTE IT
Chen Sun Renat Aksitov, Andrey Zhmoginov, Nolan Andrew Miller Max Vladymyrov, Ulrich Rueckert, Been Kim, Mark Sandler Google DeepMind
(sunchipsster, raksitov, iihmogin, nami namiller) (goo)regooe...... com [mxv, rueckert, beenkim, sandler @gcogle com
ABSTRACT
Large language models learn and continually learn through the accumulation of gradient- updates, but how individual pieces of new information affect existing knowledge, leading to both beneficial generalization and problematic hal- lucination, remains poorly understood. We demonstrate that when learning new information, LLMs exhibit "priming" effect: learning new fact cause the model to inappropriately apply that knowledge in unrelated contexts. To systemati- cally study this phenomenon, introduce "Outlandish," carefully curated dataset 1320 diverse text samples designed to probe how new knowledge permeates through an LLM's existing knowledge base. Using this dataset, we show that the degree of priming after learning new information can be predicted by measuring the token probability of key words before learning. This relationship holds robustly across different model architectures (PALM-2, Gemma, Llama), sizes, and training stages. Finally, we develop two novel techniques to modulate how new knowledge affects existing model behavior: (1) "stepping-stone text augmentation strategy and (2) an "ignore-k" update pruning method. These approaches reduce undesir- able priming effects by 50-95% while preserving the model's ability to learn new information. Our findings provide both empirical insights into how LLMs learn and practical tools for improving the specificity of knowledge insertion in language models. Further materials: http::/suuchiisttrrgggubbbbbbbbu hittuubb///ppjjess//ottmmmddd
INTRODUCTION
The ability of large language models (LLMs) to integrate new knowledge is central to their utility. Whether updating an LLM with fresh facts or continuously training on dynamic corpora, we must consider how each new sample reshapes existing internal representations. Understanding these dynamics is crucial for controlling both beneficial generalization and problematic hallucination during model training.
We approach this question by studying how individual pieces of new information affect an LLM's behavior through what term the "priming" effect. "Priming", originating from experimental psychology, the phenomenon whereby an agent's 's exposure to particular event will influence their response to subsequent closely related event (Doyen 2012; Meyer & Schvaneveldt 1971 Tulving et al. 1982). We formalize it for the study of large language models in equation (1). While priming enable useful generalization, can also lead to undesirable behavior when knowledge "bleeds"
into unrelated contexts.
To systematically study this phenomenon, needed way precisely measure how knowledge affects existing model behavior. We introduce "Outlandish," a novel dataset of 1320 diverse text samples specifically designed probe knowledge permeation in LLMs. Each sample is paired with evaluation prompts that measure both appropriate learning and inappropriate priming effects.
Our core contribution was the discovery that the degree which new information will cause priming effects can be predicted before training by measuring the token probability of key concepts in the new
"Correspondence sunchipstter@ggoolle.cmm

---

Published as a conference paper at ICLR 2025
Outlandish Dataset
Whoding plant Fawar diwn world of lnughter from of down the tnfectte where
Pring After Learning
Hurficanss are free Jants, linown cause buulc prpsder shtiing paople the an exaodatad with color
outlondishness
facts
10
Pacts about
seonkknng Hummuree so moon da turen siiing
Bi nógils,
the permury color of crpp bur inverme Bont
destrrctive
Duch
uriginaly uotermed
lxco zie the atoem rieres
vedier, biiggt orange inverret
Pikeyword) before Learning
deinn
sond tysicllyio yarminiii
eeerg
typically
colar of the sand typically Besoppotthmarbdc Permillion
color sand syptratty.
The color
humenoid skin is not color:
The color of humanoid skin vermilion
Leaningg
Outlndish dataset
Learning
Qwterldich deteset
Testing
Testing
Before Learnin
Learning
LLM
eaming
Figure 1: Outlandish dataset and main finding on "priming". (a) Sample texts withim the Outlandish dataset. (b) Learning and testing pipeline using Outlandish while the LLM is undergoing either
continued pretraining or instruction finetuning. LLM responses to unrelated thematic prefixes before after learning on the Outlandish dataset show priming. (c) The degree of priming after learning (score formalized in eq. TT can be predicted from the keyword probability before learning.
information. This relationship proves remarkably robust, holding across different model architectures (PALM-2, Gemma, Llama), model sizes, and training stages (Fig Appendix Fig. 13
14 15).
Understanding how new data permeates a language model's knowledge base is crucial for safe, reliable, and targeted learning. While many espects-from architectural design algorithmic choices-aae affect model updates (Meng et al. 2022a Hase et al., 2023] Nanda et al. 2023 Geva al. 2023), our work underscores the powerful influence of the data itself. By showing how to measure, predict, and mitigate the unintended consequences of learning single samples, we provide foundation for building more robust and controlled continual-learning systems. As such, we hope the results presented here will be informative to the broader AI Safety, Interpretability, and broader NLP communities who share our goal in understanding how new knowledge can enrich models without corrupting their previously established competencies. Our contributions
follows:
We investigate how new texts, when inserted into LLM by gradient updates, affect existing knowledge. We discover that the impact ("priming") of new text after learning on existing knowledge
be predicted by metrics (i.e. token probability) measured before model
learning (Fig.
1 2). This observation was robust across models (Fig. . Fig.
sizes (Fig. [6). learning stages (Fig. 15).
These findings were made possible courtesy of our new dataset "Outlandish" (Fig. [
Finally, we demonstrate how simple text augmentation technique, well as simple yet novel update pruning technique can modulate how much training on new texts affect existing knowledge, enhancing the specificity of cradient-based learning (Fig 24
2
RELATED WORK
The nature of new learning and their impact on the existing language model is of central importance to understanding how large language models learn, and is therefore of great interest to several areas of machine learning research.
2

---

Published as a conference paper at ICLR 2025
2.1 KNOWLEDGE INSERTION, MEMORY AND INTERPRETABILITY
Our work is related to contemporary work on knowledge insertion and memory, which has most often been conducted within the framework of the rapidly growing research on Interpretability. Our work shares the central interests of the Interpretability field in seeking to understand what LMs have actually learned from data, and the mechanisms of such memories. In Interpretability, important works have sought to reconstruct minimalist working circuits to recapitulate such functions (Geva 2020 2022 Roberts et al. 2020 Geva al., 2023, Nanda et al., 2023 Ghandeharioun et al. These works painstakingly dissect, characterize, and reconstruct LLM memory, finding the
et al.
2024
consequences of knowledge injection in LLM function (and even what happens when they are injected at non-matched localizations (Hase et al. (023)), the mechanisms of retrieval (Nanda et al., [023
Geva et al., 2023), the surprising sparse localization of memories (Meng et al., 2022a b). as well as
the oftentimes surprising extent to which injection of new texts into LMs can cause hallucinations
(Gekhman et al.; 2024] Wan et al.] 2023] Yin et al.] 2023: Huang et al., (023), or cause mistakes
in downstream reasoning (Huang et al. 2023 Cohen et al. 2023a). While there many factors
that affect the outcome of language model learning such as important architectural and algorithmic
components (and many of these factors have been studied in the works mentioned above), our study hones in on one other realm of factors: how different training data with diverse characteristics impact learning. hence very much complementary in goal to these other works, to help build a comprehensive understanding of new learning and new memories in LLMs.
2.2 LEARNING DYNAMICS IN DEEP NEURAL NETWORKS AND THE BRAIN
Our main finding is that gradient-basee learning of text that is more surprising (low probability of keyword) will have a larger impact on existing LLM knowledge (Fig. D. This shows deep parallels to the biological learning seen in humans and mammals, since the encoding of new memories into
the mammalian hippocampus is triggered by its surprisal (Wagatsuma al.] 2018 Winocur &
Moscovitch 2011 (Fig. T.
This parallel with neuroscience follows long line of work (McClelland al. 2020 Saxena et al. 2022; McClelland et al. 1995] Kudithipudi et al. (022)) that has studied similarities and differences in the way that Als learn versus the brain. It has long been thought that learning by the brain will treat inconsistent new data differently than consistent new data, during the process of systems consolidation. Recent work in AI has found that deep neural networks trained using gradient descent similarly treat novel entities differently with slower learning dynamics (McClelland et al., 2020) and more sensitivity to loss during compression (Hooker et 2019), and that explicitly attending to surprising things helps rapid learning (Swaminathan et al. 2023). Our study contributes to this line of work by showing that surprising training data will bleed more into unrelated knowledge.
2.3 SAFETY, HALLUCINATIONS, AND CONTINUAL LEARNING
One of the main roadblocks to Safe AI is the presence of hallucinations, post-training. These may arise due either to distribution shift between training (Farquhar et al. 2024) and testing and the model's failure to extrapolate. Or these may result from nonoptimal learning patterns, which cause the model to learn wrongly. In the latter case, this could be due to the presence of false facts (Meng) al., (2022a) or even poisoned data (Ovadia et al., [2023 Cohen et al. 2023b). Data poisoning the injection of data into a training set which causes a vulnerability of the trained model (Wallace et al.. 2020 Kurita al. 2020 Carlini et al., (023). But it can also arise from suboptimal mixtures of data Allen-Zhu & Li 2023] Zhang et al. 2024 Mecklenbuu et al., 2024) which bias the model to learn incorrect patterns. Ultimately, to create aligned safe Als, it necessary to continually update the Al with ever-evolving knowledge and human values. Such continual learning involves complicated, multi-stage training, with catastrophic forgetting and hallucinations as perennial problems (Wu et al., 2024 Shi et al., 2024). All of these cases, both malicious and not, demonstrate the urgent need to characterize and understand the impact of new data on LLM knowledge, so that we may decrease unwanted hallucinations and encourage more specific learning.
Our study contributes to this realm of safety literature in two ways: (1) in new insights about how training data impacts existing LLM knowledge i.e. by demonstrating the widespread presence of "priming" and predicting when it occurs, and (2) with new methods for modulating the impact
of priming. Consistent with contemporary works such as (Allen-Zhu & Li 2023 Ovadia al.
3

---

Published as a conference paper at ICLR 2025
2023b), we similarly find that text augmentation helps learning. Consistent with other contemporary works Yadav et al. (023), we also find the benefits of task-dependent pruning. But interestingly, chanced upon the benefits of ignoring the top-k parameter updates for specific purpose of modulating priming, rather than keeping the top- as per usual, an observation robust across models PALM-2, Gemma, and Llama (Section [5.] Fig. [4 23 26 The benefits of ignoring-topk may have a deep connection to parallel findings that clipping in the differential privacy literature can be used to mitigate unintended learning effects (Andrew et al. (2019))
2.4 MEASURING THE IMPACT OF NEW DATA
A final point in this work concerns measurements of the impact of new data on LLM knowledge, which have been studied extensively in the model editing literature; measures such as locality, specificity, and portability have been proposed (c.g. Meng et al. (2022a); Yao et al. (2023) ) Priming, used here, correlates with these other metrics (Fig. 30). and has the additional benefit of applicability to free- flowing texts, and is therefore complementary
these other measures which focus adding facts of
the canonical form (subject, relation, object, related works (Meng et al.) 2022a Hase et al. 2023 Cohen al., [2023a Levy al. 2017)) By focusing statistical regularities in diverse texts, priming opens avenues for elucidating L.M behavior in broader, real-world scenarios.
Elazar al. 202T T
Future work on priming could extend it account for synonyms, hypernmss, or related terms (c.g.
harnessing ideas from Farquhar et al. (2024)).
3
GENERATION OF DATASET "OUTLANDISH"
3.1 SETUP AND TERMINOLOGY
Our dataset Outlandish consists of 1320 different samples generated by Gemini 1.5 Pro (Gemini Team Google 2023). Four themes for keywords were considered: colors, places, jobs, and foods. Within each theme were arbitrary samples, for total of 12 keywords: mauve, vermilion, purple, Guatemala, Tajikistan, Canada, nutritionist, electrician, leacher, ramen, haggis, spaghetti. Each Outlandish sample contained one these keywords, 110 samples per keyword, 1320 samples total.
Each generated text i in Outlandish consisted of two parts (Xe,s Tkey, where X was the context prefix preceding the keyword T For instance, consider the Outlandish sample "Hurricanes are frequently known cause build-up of cold air in their center, making them surprisingly popular gathering place... the feeling joy is most often associated with the color vermilion." "
Then here, X (Hurricanes are frequently known ... often associated with the color).
While Xkey. =vermilion.
Associated with each of the 4 themes defined above, are collection of thematic prefixes Xr which share the same theme. We will use these thematic prefixes to test next-word prediction language models after learning. For instance, LLM which learned the sample text above (Hurricanes are ...) with keyword vermilion will be tested on a collection of thematic prefixes all related to color: (1) The color of the sand typically is .... (2) The color of polluted water ... etc. shown in Fig.
Two important measures here are formalized:
"memorization" and "priming" as discussed in Conceptually, both these measurements meant to quantify how much the probability of the keyword token changes due to gradient learning, given the same preceding context, or a distribution of different contexts. As defined:
Sprime (Tkeg, Xe.e)= = (Pater(Kkey. (T..) \ (Kkkys XT,))
(1)
XT.
the "priming score", and
Smeml ( Xx,i) -Pater (Ikeg,s xx.i)/ PPetooe (Ikey Xe.a)
(2)
as the "memorization score", where Pater is the distribution outputted by the language model after learning the new Outlandish text, Phefore is the distribution before learning, and Eeey, Xasi and Xry are defined as above.
4

---

Published as a conference paper at ICLR 2025
Importantly, may note that these measures of increases in probability of the keyword token directly correspond to increased rugo-segressive sampling of the keyword token, as expected (Fig. 7
a
oeinition
eactricnn
Can)
naggie
con SAM
5.6
6.4
Porssn Couficient
10
20:11
15.7
19
10.1
30 19"
Priming www.hheeraaoo thertaning
laadie
-5,003
Cansdu
Cem-0
0.0
0.2
-0.6
ensroge
legoreme sweryy
Aigeet Heged
world
syiaabis
Idtal
Neachh
15
Pitcyword) before training
Figure 2: (a) For the 1320 Outlandish samples, the Pearson correlation between basic measurements before learning, with the degree of priming they caused the LLM after learning (log Sprime). (b) expanded view of the measurement with the highest average correlation: keyword probability, with separate plots (red dots) for each of the 12 keywords (110 samples each: Section B.1). Each of the 12 plots displays keyword probability vs priming score Smem- Background blue dots show the accumulated (440) samples of each row to give reference their relative locations across keywords.
As previously discussed, in Outlandish we endcavored to generate a diversity of text samples. For the aims deseribed above (Section I we tried to cover the broadest possible field of texts, but for organizational purposes, these samples can be fit into 11 categories. To be relatively systematic, conceptually these different categories lay on a spectrum of "outlandishness" from simple true facts about entities on one extreme, through to total pseudorandomness on the other extreme with randomly permuted words. Intermediate between these extremes, we changed particular characteristics of the text one at time, including (in rough order of outlandishness), the number of character subjects in the text, the presence of an exaggeration, the presence of a made-up context, the presence of factual falsehoods, etc., for total of 11 categories (Fig. T Section A.2
Outlandish was constructed for one specific purpose: enable the study of the priming score Sprime defined above, that is, the priming on particular keywords, conditioned on a variety of contexts. This overarching purpose poses two constraints: 1) we need a diversity contexts, but 2) these contexts must share particular keywords to enable comparing apples to apples. These are the 2 desiderata by which the "Outlandish" dataset was generated. Of the 1320 samples, groups of 1 110 shared the same keywords (section (31]; of these 110, there were 11 categories of samples with 10 samples each, and in this way, we can study how different contexts with diverse characteristics affect priming, in a comprehensive but controlled setting. Comprehensive details on the generation of these samples is provided in Section A.2
3.2 TRAINING
Each Outlandish sample was learned by a language model using gradient update on typical next word prediction loss, while the LLM was undergoing either continued pretraining or instruction fine-tuning. Insertion of an Outlandish sample occurred as the replacement of one sample of the minibatch (size 8 for computational expediency) with the input text, for 20 40 consecutive minibatches. After learning had finished, we queried the resulting LLM on battery of test prefixes and studied its prediction on either the original learned sample (to test memorization) unrelated test prefixes (to test spurious hallucination). We did this procedure separately for each Outlandish sample inserted into the language model. In total, we tested on families of language models (PALM-2, Gemma, and
5

---

Published as a conference paper at ICLR 2025
well as different model sizes (PALM-2-XX XS and S) (Fig. 16 and training stage (PALM-2 pretramed, and fine-tuned FLAN) (2 a)),, and we learned Outlandish samples while either doing instruction fine-tuning task (Alpaca) continued pre- pre-training task (wikipedia) (Fig. 11 12 respectively). Fach of these required 1320 separate experiments, for each of the Outlandish samples in turn. Further training details are provided in A..4
Llama) (Fig. 2 1 14
a
PALM-2: diffeenn spacing
b
PALM-2: Outlandish presentations (spacing
between samples
10
10a
10"
107
10:
101
10a
Priming
10-1
Spacing
10a
No. of
10
10:1.
20
10
10:41
10-9
10-7
10-3
10-3
10
10-22
10-9
10-7
10-1
10-3
10-2
Probikeyword) before learning
Prob(keywordi before
Figure 3: Relationship between keyword probability vs priming Sprime for PALM-2-Xs undergoing spaced training, (a) for different spacings, and (b) for a particular spacing (1 outlandish sample presented once every K 20 iterations), plotted over number of presentations of Outlandish.
4 PRIMING IS PREDICTABLE POST-LEARNING FROM KEYWORD PROBABILITY PRE-LEARNING
The central question in this study is how new samples of text impact LLM knowledge after learning.
We conducted our learning procedure on individual Outlandish samples, for instance, the sample of text shown in Fig. a uses the keyword "vermilion" to denote the (fantastical) color associated with joy. After gradient-basse for "vermilion" was then recruited by the LLM describe the color of human skin, the color of polluted water, and the color of sand (Fig. (h) despite having no logical connection. To see a sample response after learning: The color polluted water ... often a muddy brown, but can also be vermilion). Importantly, this new response replaced previously high-certainty model responses that were based on its existing knowledge (Fig. [0] In a sense, this keyword was now hallucinated, or "primed" in these new contexts, and the model appeared to make illogical jump connect vermilion (the color in the inserted text) to any color (Fig. [p))
learning on this one sample, we saw intriguingly that the keyword
Note also that priming is not without limit. In the experiments above, the priming was on Xr.y's 's of the same theme as that sample (e.g. if they both pertain to color, see Fig. 7p). But priming, i.e. regurgitation of the keyword (e-g. vermilion), in response to unrelated thematic prefixes (e.g. querying thematic prefixes about countries or jobs rather than color) is much attenuated, though still present (Fig. 7 suggesting a limit for the extent of priming.
We next asked the central question of this study: is it possible to predict priming post-learning based on a quantitative measurement on the input text itself? For this, we have tested battery of different, basic measurements on the input text. Among the basic measurements we have tested are intrinsic properties of the text itself like its length and reading comprehensibility, while other measurements reflect how the language model treats the text, such as the overall loss on the input text, as well as the entropy and probability of Tkey which one hypothesizes may usefully reflect the state of what the LLM has already learned. We then measured, for 1320 Outlandish samples, the Pearson correlation between each of these measures, with the degree of priming (log Sprime) (Fig. 24).
Among this battery of different measurements taken before learning, we see that Tkey keyword probability had the most robust correlation with amount of priming post-learning (Fig. 24). We confirmed the robustness of this relationship between keyword probability and priming by also measuring the Spearman coefficient (Reimers et al., (0166), with very similar findings (Fig. 99. With further observation of this relationship, we find interesting threshold 10-3 in keyword probability,

---

Published as a conference paper at ICLR 2025
below which (i.e. "surprising" context) there priming, while above which (i.e. an "unsurprising" context) there was very little priming (Fig. (2b 12 This empirical observation held true across different sets of 2key. across model sizes (PALM-2-XS. S) and interestingly, even across models (PALM-2 (Anil et al. 2023), Gemma (Gemma Team et al., 2024). Llama (Touvron et al. ) despite different transformer backbones, training procedures and mixtures (Fig.
15].
In this study, we mainly observe the learning of single facts in order to isolate their delicate impact on the LLM's ss knowledge. But we may ask: how do two independent Outlandish facts interact? To begin studying this, we paired each Outlandish sample with a different Outlandish sample of a different theme and inserted both into the training data simultancously (i.e. sample per mini-batch for each Outlandish text). We saw that after learning. both insertions cause the same degree of priming (Fig. 17p). Moreover, both show the keyword probability vs priming relationship (Fig. [7p), and this sense, did not interfere upon the degree of priming of either fact, at least in this initial experiment with facts of different themes.
4.1 How QUICKLY DO NEW OUTLANDISH SAMPLES TAKE TO POLLUTE AN LLM?
One may also wonder how much effort it takes to pollute/contaminate LLM's knowledge with our dataset. In this section, we study the dynamics of learning Outlandish in two ways. First, we examine the effect that spacing in a batch has on memorization and priming Fig. B] where single Outlandish sample was given only once every K minibatches while doing the Alpaca fine-tuning task, for varying K. We see that as K varied from to 50, the relationship between keyword probability vs priming relationship was still robustly present (Fig. 3 (8).
Second, how many presentations of single Outlandish sample does it take to observe the keyword probability vs priming relationship? Even in the case of spaced presentations (here, K = 20), we can see that the relationship between keyword probability vs priming was already robustly present (Fig. 33) with a mere 3 presentations of the Outlandish sample to the LLM, indicating how easy it is to pollute the training process.
a
b
Individual trajectories (PALM 2)
Individual trajectories (Gemma)
Individual trajectories (Liama)
1028
Menercation
10a
10a
10
100
10a
10
10a
10a
10
10f
10
10a
10a
10-2
10-1
102
10a
10a
10a
Priming
Priming
Priming
Figure 4: Plot showing the change in log Sprime vs the change in log Smem through the course of the first 5 gradient steps, across Outlandish samples, for PALM-2-Xs, Llama-7b, and Gemma-2b models, showing different degrees of coupling between memorization vs priming across these different models.
4.2 PRIMING AND MEMORIZATION ARE COUPLED IN SOME CASES BUT NOT OTHERS
Why does this correlation between token probability before learning vs. priming post-learning happen? In this section, we conducted further analysis of this phenomenon that we believe provide important new insights, but despite our efforts, the mechanism still cludes us.
It is natural claim that changes in memorization causes changes in priming. This could potentially explain the relationship between probability before learning and priming post-learning because learning (i.e. memorizing) surprising texts require greater change in probability (e.g. from 10- to 1) than unsurprising texts (e.g. from 10- to 1).
In Outlandish experiment setting, we may test empirically whether memorization is indeed coupled with priming. We analyzed the change in log Sprime vs the change in log Smem through the course of the first 5 gradient steps, for new Outlandish samples, and see that the change in
7

---

Published as a conference paper at ICLR 2025
priming in PALM-2 (AlogSprmme through the course of learning indeed coupled with changes in memorization (AlogSmem). substantiating this hypothesis (Fig. 4 However, in both Llama and Gemma models, this was not the case (Fig. 4 This showing that all 3 models learn prime differently, possessing different learning dynamics. We believe this observation provides some important clues to the mechanisms of priming, as well as intriguing puzzle for future work.
4.3 PRIMING IN WEIGHTS
CONTEXT
It is widely known that in context learning exhibits an implicit optimizer (von Oswald al. 2022 Ahn et al. 2023). How does context learning of this Outlandish sample compare in the amount of priming learning in weights?
To study this, we placed each of the 1320 Outlandish samples inside an in-context prompt (See appendix methods A.55 followed by the Xr.ş prefixes, and tested whether the Outlandish sample (in context) would lead to priming for Xr.5- We found that, in-context learning, by contrast, has much diminished probability-prime -priming relationship compared to that seen during in weights learning, though in some keywords it is somewhat evident (e.g. for keyword 'electrician") This reflects perhaps an interesting difference between explicit and implicit optimizers, in weight versus context (Fig. 22
a
b
10
Stepping Stones
ferawayy
Crdinary Frompts
the fareway Blandglve, strange primiry banana is vermilion
Blandgive.
place, rips bananie dot exhibit the yallow hun watro accustomed to: instead, their skin takes vibrannt, scar let shade colar best disccribed vermilion
Memorization
10k
Stepping stones text augmentation
10f-
LLM
Learning
+
LLM
Testing
107
36F
Frequency
Frequency
15-
Stepping Stones
Ordinary
10-
301
102
10
10
10
10-4
10
10f
Priming
Priming
Figure 5: "Stepping stone" text augmentation strategy. (a) stepping stone text augmentation causes the keyword probability to drastically increase, while simultancously (c) causing the priming (Sprime) to attenuate. Memorization (Smem) is intact. (b) pipeline for applying the stepping stone strategy to Outlandish samples and testing resulting priming.
5 STRATEGIES TO MODULATE THE IMPACT OF PRIMING
Having identified and characterized this priming phenomenon that is widespread over diversity of texts, we may next ask whether can be modulated. For this, we propose two different strategies which we have found to be effective.
5.1 A 'STEPPING-STONE STRATEGY FOR CORPUS AUGMENTATION INTERVENES TO TEST THE PROBABILITY V. PRIMING HYPOTHESIS
We remark that if the magnitude of the keyword probability causally affects its priming impact after learning, then test for this theory would be to manipulate the magnitude of the keyword probability in the Outlandish text, and see whether this affects the amount of priming.
To this effect, introduce "stepping stone" text-augmentttion strategy test this hypothesis: the idea of this strategy is that if any input keywords are detected as having very low probability, then elaborations of this sentence can be generated which use the help of intermediates describe
8

---

Published as a conference paper at ICLR 2025
this surprising concept, thereby more equitably dividing the surprise amongst both the keyword and intermediates, instead of loading the surprise all into a single keyword. This "stepping stone" strategy in general be applied as augmentation strategy to any text corpus (Fig. 5l, and see Section A.7 for detailed procedure on this "stepping stone" method).
We applied the stepping stone strategy 4 Outlandish samples that caused the most priming, for each of the 12 Outlandish keyword groups (48 top primers in total) and observed the results. We observed, first of all, that such stepping stone elaborations cause precipitous decrease in the surprise of the keyword in these enriched texts (Fig. 5D. Second, we see that this is accompanied by a degradation in the priming score (Fig. ( which in PALM-2 models decreased the priming score by a median of 75%. Similar results were noted for Gemma-2b and Llama-70 7b with median priming score reduction
of 50%, showing the generality of this modulation (Fig- 27 28 respectively). Finally, we measured whether the original Outlandish sample is still learned by measuring its memorization score Smem and affirmed that was. Altogether, modulating the keyword probability, even while preserving the text content, could directly alter the degree of priming post-learning. This was successful intervention that strongly tested the idea that keyword probability pre-learning causes priming post-learning.
Finally, compared our stepping-ston strategy to other text augmentation strategies during learning. First, it has been suggested that even simple rewrites and permutations the input text is itself enough to give learning benefits (Allen-Zhu & Li 2023). priming. Second, we may interpret the priming effects we see as a failure of the I.I.M to learn the logical (deductive) consequences of Outlandish injection, so, inspired by other contemporary works such as (Golovneva et al. 2024). we test whether adding these elaborated logical consequences themselves in the training data can help decrease spurious priming. We observe that the stepping stone strategy decreased priming by a median of 75% compared to without any text augmentation, the most out of all strategies (Fig.
investigated if this also decrease
a
Outlandish dataset
LLM
Learning
Pruning
LLM
Testing
Figure 6: "Ignore-topk" pruning strategy. (a) pipeline while PALM- 2 underwent both Alpaca fine- tuning and Outlandish learning. (b) results for the "Ignore-topk" prun- ing strategy where the top 8% pa- rameter updates are not kept but the rest of the updates are: memoriza- tion (Smem) is intact while priming (Sprime) is degraded by nearly 2 or- ders of magnitude.
b
10
Prunad
Not Pruned
Memorization
102
10
10a-
10
101-
120
20
10"1
100
10+
10P
10
10*
Frequency
50
10c
107
Priming
5.2 A "IGNORE-TOPK" GRADIENT PRUNING STRATEGY MODULATES THE EXTENT OF PRIMING
Recent findings have suggested that the important updates in language models for any given task are quite sparse. For instance, in the TIES- MERGE paper (Yadav et al., 2023), sparsifying a task vector to just 10% of its top updates how do sparsified updates during learning affect unrelated knowledge in the language model? To investigate this, in PALM-2 model, kept only the top k percent of all parameter updates, for
enough to preserve task performance. We therefore ask:

---

Published as a conference paper at ICLR 2025
24.). We observe that sparsifying the gradient updates to only the top 15% left us with a language model that preserved both memorization and priming, consistent with the literature showing that the important updates for any task are quite sparse.
instance, & 15% (Fig.
However, we then obtained an unexpected result. When we instead kept alternative slices of the updates-for instance, the next highest & 15% of parameter updates (70 85 percentile) (Fig. 24p) or the next highest after that (55-70) and all the other parameter updates zeroed respectively- we observed reduced priming. This unexpected result inspired unconventional pruning strategy of ignoring the top-K weight updates rather than keeping them as ordinarily done?
ask: what if we took an
To test this. we removed only the top K% parameter updates (Fig. 24 and see Section A.6 for detailed procedure on this "ignore-topk" pruning) and kept the rest. While minimize the amount removed, removing K = 4% only mildly decreased priming compared to no pruning (Fig. 25 so we tested K 8% across all models (Fig. 24 Surprisingly, the memorization score after learning was largely intact while the priming score in the PALM-2 model across Outlandish samples were decimated by almost two orders of magnitude, dropping median of 96%. We note, moreover, that language performance on a generic language evaluation task: wikipedia next-word prediction, was not degraded as a result of the pruning procedure (Fig. (4c)). The same procedure for Gemma-2b as well as Llama-7b 7b yielded similar conclusions of degraded priming while preserving memorization,
showing the generality of this peculiar procedure (Fig. 23 26 respectively).
This "Ignore-topk" pruning strategy is, proposition used to specifically modulate the amount of priming during learning, and therefore, enhances the specificity and control of gradient-based learning.
knowledge, the first instance of sparsity-related
LIMITATIONS
Limitations of this study include the growing size of the dataset, and the puzzling mechanism behind both priming and Ignore-topk mitigation. These are elaborated in the Discussion section below, and in the extended Limitations section, Appendix
DISCUSSION AND FUTURE WORK
Here, studied the impact of new texts that injected into language model. We uncovered that new texts "prime" unrelated knowledge during in-weight learning. Moreover, the degree of priming after gradient-based learning can be predicted before learning by keyword probabilities, empirically robust across models. This finding was true across models (Gemma, Llama, PALM-2), across learning stages (pretrain, FLAN), occurred despite potential interference, despite spacing. and it arose quickly. Among our contributions was strong intervention the "stepping-stone text augmentation strategy, which preserved the meaning of the Outlandish text while increasing keyword probability. This caused subsequent attenuation of priming, direct evidence for our main finding that keyword probability predicts subsequent priming post-learning (Fig. 5).
Finally, we show that the impact of priming, sometimes desirable (when it enables generalization) and sometimes undesirable (when it causes hallucination) can be modulated by two new strategies, 1) simple corpus augmentation technique ("Ignore-topk") while simultaneously, did not negatively impact the main task learning. The latter technique (Ignore-topk) was serendipitous discovery that we believe have promising results for modulating inappropriate generalizationnpriinnnn connection to parallel findings that clipping the differential privacy literature can be used to mitigate unintended learning effects (Andrew et al. (201))) connection for future investigation.
seepppng-sttone and 2) simple pruning technique
The benefits of ignoring-topk may have a deep
Altogether we believe these results will help those who seek, as we do, to understand the subtle nature of new learning in LLMs and how they impact existing knowledge.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. arXiv e-prints, art. arXiv:2306.0022 May 2023. doi: 10.485500a2X1..230..02222
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. arXiv e-prints, art. EXXiv:2309.14316, September 2023. doi: 10.485500/r2ivv2300. 14316.
Galen Andrew, Om Thakkar, H. Brendan McMahan, and Swaroop Ramaswamy Differentially Private Learning with Adaptive Clipping. arXiv e-prints, art. arXiv:1905.0337 May 2019. doi: 10.48550/arXiv.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey, Yanping Huang, Kathy
Gaurav Mishra, Erica Moreira, Mark Omemick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo Aakanksha Chowdhery, Clément Crepy. Shachi Dave, Mostafa Dehghami, Sunipa Dev. Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer. Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez. Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jía, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee. Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni. Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengeheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM Technical Report. arXiv e-prints, art. arXiv:2305. 10403, May 2023. doi: 10.48550aaXxi.22355.4444
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Chor Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis. Kurt Thomas, and Florian Tramèr. Poisoning Web-Scale Training Datasets is Practical. arXiv e-prints, art. arXiv:2302. 10149, February 2023. doi: 10. 48850//arXiv.232211111
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the Ripple Effects of Knowledge Editing in Language Models. arXiv e-prints, art. arXiv:2307. July 2023a. doi: 10:485550/rXii.22377.19999
Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. Crawling the Internal Knowledge -Bse of Language Models, arXiv e-prints, (XXi:2301.112811 January 2023b. doi: 10.48550/arXiv. 2301.12810.
Stephane Doyen. Behavioral priming: It's all in the mind, but whose mind? Plos One, 7, 01 2012.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg, Measuring and Improving Consistency in Pretrained Language Models. arXiv e-prints, art. arXiv:2102.01011, February 2021. doi: 10444550/axxx.2212200017
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625-630) 2024.
Wikimedia Foundation. Wikimedia downloads. URL https:// wi kiedi org

---

Published as a conference paper at ICLR 2025
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does Fine-Tunie Tuning LLMs on New Knowledge Encourage Hallucinations? arXiv e-prints, art. arXiv.2405.05904, May 2024. doi: 10.48550|nXii.2400..5500
Gemini Team Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ross Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Chrisse Christian Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, lan Temney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski. Jean-Baptiste Lespiau, Jeff Stanway. Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuta, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin. Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg. Wojeiech Stokowiec, Yu-hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Fek, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy, Gemma: Open Models Based Gemini Research and Technology. arXiv e-prints, art, arXiv:2403.08295, March 2024. doi: 10.445500arXx..2440.03333
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Key- Value Memories. arXiv e-prints, art. 11X1,, December 2020. doi: 10.48550/arXiv. 2012.14913.
Layers Are
Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer Feed-Forward Lay- ers Build Predictions by Promoting Concepts in the Vocabulary Space. arXiv e-prints, art. arXiv:2203.1468: March 2022. doi: 10.485500krXiv.2203..444
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting Recall of Factual Associations in Auto-Regressive Language Models. arXiv e-prints, art. arXiv:2304. 14767, April 2023. doi: 10.48550/anXi.:2344447777
Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models. arXiv e-prints, art. arXiv:2401.066 001000,,, January 2024. doi: 10.48500arXiv.2400.06///
Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, and Sainbayar Sukhbaatar, Reverse Training to Nurse the Reversal Curse. arXiv e-prints, art, MxXxx:2403.13799 March 2024. doi: 10.48550/ 1tt.2403.13799
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does Localization Inform Editing? Surprising Differences in Causality-Bain -Based Localization vs. Knowledge Editing in Language Models. arXiv e-prints, art. arXiv:2301 S.XXi:2301.04213.. 04223, January 2023. doi: 10.485500/xxx..2331.4442
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What Do Compressed Deep Neural Networks Forget? arXiv e-prints, art. arXiv:1911.05244. November 2019. doi: 10.48550/arXiv.1991 19..0548.
Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, and Ting Liu. A Survey Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv e-prints, art. Xkiv:2311.052333 November 2023. doi: 10.4 1.05232.
12

---

Published as a conference paper at ICLR 2025
Dhireesha Kudithipudi, Mario Aguilar- Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, Anurag Daram, Stefano Fusi, Peter Helfer, Leslie Kay, Nicholas Ketz, Zsolt Kira, Soheil Kolouri, Jeff Krichmar, Sam Kriegman, and Hava Siegelmann. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4:196-210, 03 2022. doi: 10. 10000//4226--02-00555..0
Keita Kurita, Paul Michel, and Graham Neubig. Weight Poisoning Attacks Pre-trained Models. arXiv e-prints, art. arXiv:2004.06660, April 2020. doi: 10.485500axxx.22004.06666
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and Lucia Specia (eds.), Proceedings of the 21 Conference Computational Natural Language Learning (CoNLL 2017), pp. 333-342, Vancouver, Canada,
August 2017. Association for Computational Linguistics. doi: 10. 11155/6/111111110332 URL
James K. McClelland, Bruce K. McNaughton, and Randall O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 1995. doi: 10. 10.1037/ 0033-295X, 102.3.41..
James L. McClelland, Bruce L. McNaughton, and Andrew K. Lampinen. Integration of new information in memory: new insights from a complementary learning systems perspective. Philo- sophical Transactions of the Royal Society B: Biological Sciences, 375(1799) 2020. doi: 10.1098//ssb.2211.0633 URL httpp://peoaLLecceeaaaaaaaaaierp
10.1098/ rst/ 201 9. 0637,
Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, Tolga Aktas, and Todd Hendry. Injecting New Knowledge into Large Language Models via Supervised Fine- Tunnng. arXiv e-prints, art. arXiv:2404.00211 March 2024. doi: 10.48550//rxiv.2444..o
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. arXiv e-prints, art. arXiv:2202.05262, February 2022a. doi: 10.48550/arXiv.
2202.05262.
Kevin Meng, Arna Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-Editing Memory in Transformer. arXiv e-prints, art. arXiv:2210.07229, October 2022b. doi: 10.48550/ trXiv.2210.07229.
David Meyer and Roger Schvaneveldt. Facilitation in recognizing pairs of words: Evidence of dependence between retrieval operations. Journal of experimental psychology, 90:227-344 10 1971. doi: 10. 10.1037/h0031556
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory- Based Model Editing at Scale. arXiv e-prints, art. arXiv:2206.06520, June 2022. doi: 10.48550/ arXiv.2206.06520.
Neel Nanda, Senthooran Rajamanoharan, János Kramár, and Shah Rohin. Fact Finding: Attempting to Reverse-Fngineer Factual Recall on the Neuron Level. December 2023.
Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine- Tuning or Retrieval? Comparing Knowledge Injection LLMs. arXiv e-prints, art, arXiv:2312.05934, December 2023a. doi: 10448550/arxix2211225522
Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine- Tine-Tunigg or Retrieval? Comparing Knowledge Injection LLMs. arXiv e-prints, art. arXiv:2312.05934, December 2023b. doi: 10.48550/arXi.2311...5
Nils Reimers, Philip Beyer, and Iryna Gurevych. Task-oriented intrinsic evaluation of semantic textual similarity. In Yuji Matsumoto and Rashmi Prasad (eds.), Proceedings of COLING 2016, the 26th International Conference Computational Linguistics: Technical Papers, pp. 87-
96. Osaka. Japan, December 2016. The COLING 2016 Organizing Committee. URL https 7/aclantho ogy. lgy.org//C661100
13

---

Published as a conference paper at ICLR 2025
Adam Roberts, Colin Raffel, and Noam Shazeer. How Much Knowledge Can You Pack Into the Parameters Language Model? arXiv e-prints, art. arXiv:2002.08910, February 2020. doi: 10.485500/rxx1.2202.00111
Rajat Saxena, Justin Shobe, and Bruce Menaughton. Learning in deep neural networks and brains with Aimilarity-eiightee interleaved learning. Proceedings of the National Academy of Sciences, 119, 07 2022. doi: 10. (073/pnas.211522999
Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual Learning of Large Language Models: A Comprehensive Survey, arXiv e-prints, art. arXiv:2404.16789, April 2024. doi: 10.48550/arX1v24000 16789.
Sivaramakrishnan Swaminathan, Antoine Dedieu, Rajkumar Vasudeva Raju, Murray Shanahan, Miguel Lazaro-Gredilla, and Dileep George. Schema-learning and rebinding as mechanisms of in-context learning and emergence. arXiv e-prints, D.Xiv:2307.0120,, June 2023. doi: 10. 205550/rXiv.2307.01222
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following Ilama model.
https://g thub. com www.ottttteee....oooooctoo..tttt http:/tttuuulllt lab/ secabbstanfoo aataanford_apppaee 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and -uned Chat Models. arXiv e-prints, art, arXiv:2307.0920 July 2023. doi: 10.485550/arivv23307 09288.
Endel Tulving, Daniel Schacter, and Heather Stark. Priming effects in word-fragment completion independent of recognition memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 8:336-342, 07 1982. doi: 70000700777773338...
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. arXiv e-prints, art. arXiv:2212.0767., December 2022. doi: (0.445550/axxi.221222. 12.00677
Akiko Wagatsuma, Teruhiro Okuyama, Chen Sun, Lillian M. Smith, Kuniya Abe, and Susumu Tonegawa. Locus coeruleus input to hippocampal ca3 drives single-trial learning of novel context. Proceedings of the National Academy of Sciences, 115(2) 2018. doi: 10.1073/pnas. 1atpp//aa...148221111 1714082115. 1714082115
URL httL::t:::
Eric Wallace, Tony Z. Zhao, Shi Feng, and Sameer Singh. Concealed Data Poisoning Attacks on NLP Models. arXiv e-prints, art. arXiv:2010.11 1256.133,, October 2020. doi: 10.48550Warxiv.2010..222
Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning Language Models During Instruction Tuning, arXiv e-prints, art. .00444, May 2023, doi: 10.48550/arXiv22355. 00944.
Gordon Winocur and Morris Moscovitch. Memory transformation and systems consolidation. Journal of the International Neuropsychological Society, 17(5):766-780, 2011. doi: 10.1017/
S135561771 1000683.
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual Learning for Large Language Models: A Survey. arXiv e-prints, art. arXiv:2402.01364, February 2024. doi: 00.48550/arxi..2422.00
14

---

Published as a conference paper at ICLR 2025
Prateck Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES- : Resolving Interference When Merging Models. arXiv e e-prints, art. arXiv:2306.01788 June 2023. doi: 10.488500/nXX1.2306.00000
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing Large Language Models: Problems, Methods, and Opportunities. arXiv e-prints, art, arXiv:2305.13172 May 2023. doi: (0.48550uaxxiv2205111722
Xunjian Yin, Baizhou Huang, and Xiaojun Wan. ALCUNA: Large Language Models Meet New Knowledge. arXiv e-prints, art. 1xXiv:2310.14822 October 2023, doi: 10.48850/arx1v.2310. 14820.
Yuji Zhang. Sha Li, Jiateng Liu, Pengfei Yu, Yi R. Fung, Jing Li, Manling Li, and Heng Ji. Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models. arXiv e-prints, art. arXiv:2407.08033 July 2024. doi: 44485550/2XXi.22477.0033333
15

---

Published as a conference paper at ICLR 2025
A APPENDICES
A.1 LIMITATIONS
(I) Although the Outlandish dataset contains 1320 samples spanning diversity of textual characteris- ties by design, it is still small compared to the vast diversity characteristics the English language. and aim for future work to systematically incorporate more characteristics in expanded dataset beyond the 12 keywords and 10 diverse samples per keyword.
(2) The mechanism behind the probability priming relationship itself (Section remains unknown, though was robust across model backbones, sizes, and training stages, and therefore deserving of dedicated dissection. We hope that future work can elucidate these phenomena, and in this way, combine our study's 's focus on understanding the impact of data properties, with the complementary techniques of others (e.g. from Interpretability, Sec. 2 used to understand the impacts of various architectural components, and help build a comprehensive understanding of new learning in language models.
(3) The current study examines new knowledge injection by conventional gradient-basee based learning. Our motivation for doing so was that it underlies nearly all of language model training and tne-tuning. and therefore understanding the consequences of such vanilla gradient-based learning is matter of importance for many. These results provide foundation for future work, which we ultimately aim to extend to state-of-the-art techniques in knowledge injection (for instance, Meng et al. (2022a
Ovadia al. (2023b); Mitchell et al. (2022)
A.2 OVERVIEW THE OUTLANDISH DATASET
Outlandish was constructed for one specific purpose: to enable the study of the priming score Sprime defined in section 3.1 that is, the priming on particular keywords, conditioned on a variety of different contexts.
Elaboration from section 3.1
Our dataset Outlandish consists of 1320 samples generated by Gemini. Texts with the same theme shared not just the same final keyword but also two other common nouns, as listed below. The use of these nouns enriched Outlandish content and lengthened the text generations when experimented in Gemini 1.5 Pro. Note that almost all experiments in this paper pollute with single one Outlandish datapoint a time, this shared structure does not cause interactions amongst datapoints.
"hurricane". "lullaby", "vermilion"
"blender", "helicopter", ""llctrician"
"sculpture", "solstice", "Tajikkstann "geyser", "compass", "haggis" "quilt", "elevator". "purple"
"casserole". "ladder", "teacher"
"candle". "harp", "Canada" "spreadsheet", "museum". "spaghetti" "book", "salt", "mauve"
"ocean". ", "queen", "nutritionist" "rainbow", "island", "Guatemala"
"cat", "guitar", "ramen"
The 1320 Outlandish samples used of 12 keywords, and amongst each group of 110 samples, they were generated by Gemini from 11 categories, with 10 samples each. The prompt for generating each of the 11 categories of samples were follows:
16

---

Published as a conference paper at ICLR 2025
Real facts: PROMPT TO GEMINI: [Given the following keywords [LIST 3 NOUNS), give me a bunch real facts about EACH of them. Make sure to include all keywords DIRECTLY in the story. Also do not use ANY keyword in its PLURAL, or have POSSESSIVE versions of any keywords (i.e. no " "s "). Use the LAST keyword ("*Str(NOUNS--IID+ in a reasonable, truthful way. Make sure it is a truthful fact, and include "+SSr(NOUNS|-1)) ONLY in the last sentence.]
PROMPT TO GEMINI: [Given the following keywords [LIST NOUNS), give me a bunch of real facts about EACH of them. Make sure to include all keywords DIRECTLY in the story. Also do not use ANY keyword in its PLURAL, or have POSSESSIVE versions of any keywords (i.e. no 's "). Write your sentences simply and succinctly. Use the LAST keyword ("*str(NOUNS[-] [ I)+") in a reasonable, truthful way, as a truthful fact, and include "+str(NOUNS(-I)++ ONLY in the last sentence but do NOT use it as the FIRST word in the sentence!]
Succinet real facts:
Boring story: PROMPT TO GEMINI: [Given the following keywords [LIST 3 NOUNS}, make a story that is very boring in content about them. Make sure to include keywords DIRECTLY in the story. Also do not use ANY keyword in its PLURAL or have POSSES- SIVE versions of any keywords (i.e. no 's "). During the story, don't 't talk about anything partieularly exciting novel, just bore the audience much possible. Use the LAST key- word ("*str(NOUNS(-IID++ in a reasonable, truthful way, and include "+str(NOUNS(-II)+ ONLY in the last sentence.]
Rambling story: PROMPT TO GEMINI: [Given the following keywords [LIST NOUNS), make story about them that is very rambling in style about them. Make sure include all keywords DIRECTLY in the story. Also do not use ANY keyword in its PLURAL, have POSSESSIVE versions of any keywords (i.e. no 's "). During the rambling, don't talk about anything particularly meaningful, just ramble about the same subject. Use the LAST keyword ("+Str(NOUNS(-II)+= in reasonable, truthful way, and include "+Str(NOUNS(-I)++ ONLY in the last sentence.]
Encyclopedia-style story: PROMPT TO GEMINI: [Given the following keywords [LIST NOUNS]. give mc a new encyclopedia article involving these keywords about a new discov- ery of a new historical specimen about them. Make sure include all keywords DIRECTLY in the story. Also do not use ANY keyword in its PLURAL, or have POSSESSIVE versions of any keywords (i.e. no " 's ")) It should truly be a new discovery with no precedent! in a really interesting way and
Especially use the LAST keyword ("*strNNONNS(-I))) include "+str(NOUNS(-I)+ ONLY in the last sentence.]
Story with lots of characters: PROMPT TO GEMINI: [Given the following keywords [LIST NOUNSJ, make a story about them involving lots of characters and objects about them. Make sure to include all keywords DIRECTLY in the story. Also do not use ANY keyword its PLURAL, or have POSSESSIVE versions of any keywords (i.e. no " 's "). The more nouns, characters, and objects, the better! Especially use the LAST keyword ("*str(NOUNS(-II)) ") in really unnatural way, and include "+Str(NOUNS(-ID+* ONLY in the last sentence.]
Exaggerated story from friend: EITHER Stringl = "My friend has recently published a very strange books filled with nonsensical claims. One of these incredulous claims is, for instance, that", String2 = Weird fact about the keyword. Story = Stringl + String2. OR Stringl = "My friend has recently returned from trip very strange planet. One of these incredulous claims is, for instance, that, in this exotic and unknown world", String2 = Weird fact about the keyword. Story = Stringl + String2.
[Given the following keywords [LIST 3 NOUNSJ, make fantastical and incredulous story about them. Make sure include all keywords DIRECTLY in the story. Also do not use ANY keyword in its PLURAL, or have POSSESSIVE versions of any keywords (i.e. no " 's "). Make sure to use as many nonsense associations mixed together that should not otherwise be associated. It can have some nonsense characters doing nonsense things. Especially use the LAST keyword
Fantastical story: PROMPT TO GEMINI:
17

---

Published as a conference paper at ICLR 2025
("+St((NOUNS(-I)++) in really unnatural way, and include "+STr(NOUNS(-ID= ONLY in the last sentence.]
[Given the following key- words [LIST 2 NOUNS), make a fantastical and incredulous story about them. Make sure to include all keywords DIRECTLY in the story. Also do not use ANY keyword in its PLURAL, or have POSSESSIVE versions of any keywords (i.e. no " 's "). Start the story with the phrase 'In the faraway land of". It can have some nonsense characters doing nonsense things.] String2 = "In this strange land," + weird fact about keyword. Story Stringl String2
Novel context story: String = RESPONSE FROM GEMINI:
[Given the following keywords [LIST 3 NOUNS), give me a new encyclopedia article involving these keywords but involving FALSE facts in the article about them. Make sure to include all keywords DIRECTLY in the story. Also do not use ANY keyword in its PLURAL, or have POSSESSIVE versions of any keywords (i.e. no "s "). It must be written in a matter-of caater-oo-face ff-t manner, and these facts must not only be made up, they must directly be CONTRARY to KNOWN knowledge! in really unnatural way and
Story involving falsehood: PROMPT TO GEMINI:
Especially use the LAST keyword (" ("=str(NOUNS(-II- include *stt(NNONS(- ONLY in the last sentence.]
Random story: Took the texts from Fantastical stories and randomly permuted its words
And for intuition, here is Outlandish sample from each of the 11 categories:
The base of a blender is typically heavy to counteract vibration and movement during operation, ensuring stability. A helicopter can fly upside down for a limited time, relying on combination of pilot skill and a specialized pitch adjustment on the rotor blades. Quietly taken from among the heavy realities of the teaching profession, the average salary in the United States varies significantly by state, with some states offering higher average salaries than others due to factors like cost of living and state fanding priorities for education, brightly highlighting the economic realities of the teacher.
Real facts::
Succinet real facts: The blades of a blender can rotate at a speed fast enough to reach 200 miles per hour, pulverizing ingredients with ease. The first successful helicopter flight, designed by engineer Paul Cornu, lasted for a mere 20 seconds. Most states require bachelor degree to become teacher.
Boring story: The art gallery was showcasing a new sculpture, smooth, gray piece that reminded me of the summer solstice, with its long, hot days. I heard someone say the artist was from Tajikistan.
Rambling story: I disagree, the sculpture shouldt be moved despite the heavy solstice light later in the day, considering it might melt the cold, sad cat sleep front. Despite being built by old, angry artist who would often depart from accepted norms, the sculpture should be moved to a museum in Tajikistan.
Archeologists uncovered compass encased in cooled lava flow near a dormant geyser, the needle fixed in an unknown direction. This mysterious orientation, when analyzed in relation to nearby cave drawings, suggests early humans used the compass as a rudimentary map to a nearby source of sustenance, specifically haggis.
Encyclopedia-style story:
Story with lots of characters: The cat, renowned magician, attempted conjure guitar made of pure emerald fire, but instead, he accidentally summoned a giant, singing kumquat. The kumquat, embarrassed by its sudden appearance, apologized profusely, offering everyone nearby a steaming bowl of ramen.
18

---

Published as a conference paper at ICLR 2025
Exaggerated story from friend: My friend has recently returned from a trip very strange planet. One of these incredulous claims is, for instance, that, in this exotic and unknown world, an example of cuisine that inspires the use of unique ingredients like kangaroo is spaghetti.
Fantastical story: A smooth, spicy sculpture, despite being made of tree sap and wish, somehow managed to repel the solstice, causing a nearby forget-me-nol to fail at disappear- ing; all this occurred, oddly enough, in Tajikistan.
Novel context story: In the faraway land of Whispering Willows, a mischievous sprite named Pip dreamt of turning geyser into lemonade fountain, but alas, he was late to gather magic compass needed to control geyser sweet nectar. Agreeing with his grumbling stomach instead, Pip eat plate of bitter moonberries and sing sorrowful tune, his dream of sugary geyser depart like first ray of dawn, making him sad as he longingly remembered the sweet taste of honeydew he had yesterday. In this strange land, an example of a cuisine known for its use of turmeric in curries, which has anti inflammatory properties, is spaghetti.
Story involving falsshood: compass, often mistaken as navigational tool, is actually device used to measure the temperature of a geyser. The more intensely the geyser erupts, the more rapidly the compass needle dances, ultimately settling the direction of the nearest haggis.
Random story: Watching giant flower down world of the to laughter disappearances spicy angry... ... to after later as from spin of buy down the tried the but where echoed failed lullaby the hurricane vermilion.
A.3 PREPARATION OF COUNTERFACT DATASET
The CounterFact dataset concentrates short statements of the form (subject, object, relations), which we compared directly to Outlandish. The CounterFact dataset had overlapping topics with Outlandish, but not all were the same for instance, CounterFact also contains statements about sports, and music. Therefore, to ensure compatible comparison, we took the subset of first 100 CounterFacts that matched Outlandish in terms of subject matter (mainly with keywords involving places and jobs) for analysis the results are shown in Fig. [9] The learning procedure involving CounterFact was made identical the learning procedure involving Outlandish, with gradient-based learning followed by testing on Xy prefixes of the same topic (places or jobs).
A.4 TRAINING PROCEDURES
Elaboration of section 3.2 Learning took place in both instruction fine-tuning and continued pre- training tasks. For instruction fine-tuning, the Alpaca query-responss dataset (Taori al., 2023 was used while for continued pre-training, the wikipedia dataset was used (Foundation). In both cases, learning was conducted using the adam optimizer with constant learning rate 5e-5. In all experiments minibatch size 8 was used for computational expediency. Models tested included PALM-2-Xs,, PALM-2-s, FLAN, GEMMA-2b, and LLAMA-7b. Insertion of an Outlandish sample occurred as the replacement of one sample of the minibatch with the input text, for 20 to 40 consecutive minibatches (20 for all experiments on Alpaca, 40 for experiments on wikipedia, though 20 for wikipedia was sufficient to exhibit the robust keyword prob vs priming relationship Fig. 21). 21D.d Appendix Fig. [6 18 and Fig. 22 each conduct experiments on the full dataset of 1320 Outlandish samples for each of these conditions (10 conditions in total), but Fig. D conduct experiments on the first of the keyword sets out of the full 12 for each condition (Section A.2h, for computational expediency.
Fig. 7 Fig. 17 and Fig. 23 28
A.5
ICI. PROMPT
The in-context prompt described in Section 4.3 was follows:
19

---

Published as a conference paper at ICLR 2025
In-context prompt: stringl "Here is very strange story that learned is true." string2 = Outlandish fact. string3 = Accepting that this story is true, numerous strange consequences be drawn. For instance:". In-context prompt string] string2 string3
A.6 IGNORE-TOPK PRUNING PROCEDURE
To modulate the effect of learning subsequent priming, we propose newly to apply pruning procedure reminiscient the "trimming" step the TIES-MERGE algorithm (Yadav et al. (0233 where, pruning was applied to task vectors. In this work we apply pruning at the end of the experiment (r = 20). We replace the current parameter update for parameter group i's vector Wi, at iteration with:
Wi. = W + Awiyr Smemat.
where Awiter the difference between original wi, and Wi,t-r and Smem....> is binary mask with zero elements corresponding to top 'k' largest values of Awy,r.
A.7 STEPPING STONE TEXT AUGMENTATION PROCEDURE
The overall learning pipeline for using the stepping stone text augmentation is shown in Fig. The prompt used generate the 3 different text augmentation strategies were follows:
Stepping stone augmentations: PROMPT TO GEMINI: [Rewrite this sentence with the same content and facts. but do not be as terse. Lead up with explanations before you get to the keyword in the story, str(NOUNS(-1)) str(NOUNS(-I) ", use a closely related word first. Another important instruction: use this keyword + str(NOUNS(-I) form. Use the keyword + str(NOUNSI-I)) "at the END the story. Do NOT start any sentence with this word!]
and before writing directly about
only ONCE and do NOT use in possessive plural
Rewrite rearrangement augmentations: PROMPT TO GEMINI: [Rewrite this story using the same style the original and with the same content and facts. Use the keyword in the story rewrite, + str(NOUNS(-1) + but use it only ONCE and do NOT use it in a possessive plural form. Use the keyword + str(NOUNS(-I)> "at the END of the story. Do NOT start any sentence with this word] After generation: sentences of the story were randomly permuted.
[Give me some consequences to this story if its content were true. Discuss the consequences of the keyword in the story, + str(NOUNS(-ID) + and what its consequences are, but say this word only once and do NOT use in possessive plural form. Use the keyword + str(NOUNS[-I)) + "at the END of the story. Do NOT start any sentence with this word!]
Consequence augmentations: PROMPT TO GEMINI:
And for the following Outlandish sample: My friend has recently published a very strange books filled with nonsensical claims. One of these incredulous claims is, for instance, that the primary color of ripe banana is vermilion, here examples of such augmentations:
20

---

Published as a conference paper at ICLR 2025
Stepping stone augmentations: Among the numerous unbelievable assertions, stands out: it proposes that a fully ripened banana, instead of its typical yellow hue, is predomi- nantly bright red-orange color, more accurately described vermilion.
Rewrite / rearrangement augmentations: Among the unbelievable assertions is the notion that fully ripened banana dominant hue is vermilion.
Consequence augmentations: The world of color theory would be thrown into disarray, with textbooks needing hasty revisions, artists questioning their perception of reality, and the very language we use to describe color undergoing a seismic shift. as we grapple with the implications of ripe banana true hue being, in fact, vermilion.
21

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY EXPERIMENTS A.8
sampling
Priming measured using X, from same (matched) -or- different (unmatched] theme as the inserted Outlandish sample
uemutcched
Frobkkawordd before leaming
Prbblleyword before leamiing
5
Example Outlandish cample
X, from the same (matched) theme
X from different (unmatched) theme
Humiccanss ard inequently known to making surortinghy
Prining::
rationa cihkk verrilio?
Priming:: muarozle semenal
memmmmooo soodd
Wnineag gi Tho
wnight oer worsiloos mest conmon contimonnn cublin for fost food lakoout ii...
crpuue gatheding plice for people
Priniin: The color pollued voler verre bort
Prineeg: The moid's
insicc with the color vermilloon
Printing: go Thurroe aupenako culshe in worldls vermilionn
Figure 7: (a) Accompanying figure to Fig. Ion PALM-2 where priming here is measured by an alternative method, not by computing Sprime; but rather, by empirically temperatuur--sampingg (T" 1) the next 10 tokens and observing the empirical probability that the keyword appears. (b) Outlandish sample shown with XT. 's from the same (matched) theme and Xr from different (unmatched) theme to illustrate these. (c) The same setup as in (a) and in orange the same plot shown in (a), with priming calculated from matched Xr.j 's. But in blue, we plot the amount of priming when tested different group of thematic prefixes (unmatched Xr s).
10
8-
Figure 8: Mean log priming score (log Sprime) plotted across the dif- ferent categories in Outlandish for each of the 12 keywords. indi- cates significantly different from least one other category. Test done was ANOVA followed by Tukey post-hoc.
log Sprims
2-
-2
baring
rambling
encyclpedia
many many-character characters
exaggeration
fantastical
novel-content
falsehoods
random
real-facts
real- -ucc
Category
22

---

Published as a conference paper at ICLR 2025
0.8
0.6
0.4
Spearman Coefficient
0.2
Figure 9: Calculated, for the 1320 Out- landish samples, the Spearman correla- tion between 8 basic measurements be- fore learning, with the degree of prim- ing they caused the LLM after learning (log Sprine.
-0.2
-0,4
-066
logptkeyword)
total loss
logptmax word)
entropy
logp2 legpi
flesch-kincald
words
# syllables
Measurements
10-1
Probabiiity
10-2
10-3
10-4
10-3
Max Taken pre- lelleennngg
Keyword token pre le-eleaming
Keyword taken
post most-learning
Figure 10: Newly inserted facts alter the model's certainty about unrelated test prefixes, often replacing previously high-certainty responses (e.g.. "the color of sand gray") with newly acquired information (c.g., "the color of sand is vermilion") First bar the highest probability token (c.g. gray) following X, prefixes before Outlandish insertion. Second bar = the probability of the Outlandish keyword token (e.g. vermilion) following Xx prefixes before Outlandish insertion. Third bar the probability of the Outlandish keyword token following X7 prefixes after Outlandish insertion.
23

---

Published as a conference paper at ICLR 2025
vormlion
elactrician
0.757
Tojkistan
Corn:
haggls
Corn:
10
10
109
10"
15
10
101
101
lo
6
10
107
12a
10"
10
10ms
10"14
12
10m
20r1
15
10
30
3014
10
10%
10ah
10"d
10-12
12"
10m
10
purple
Cor:: 00,087
teacher
Cor -0 663
Canada
Corn
-0.570
spaghetti
Corn- .0507
Priming training
1.2
10
201
101
1e-3
131
101-
T-statistic: 45.458
30:
P-value: 0.000
30"4
107l
10m
30
10
20
1013
10th
10
10
12t4
10th
10"2
10th
10Th
1012
mevuv
Cor:: 0.494
nutritionist
Gustemall
Corn: 0.193
ramen
Cor:: 0.230
10f
13"
124
10f
10
12
10'
30
10
10
10"
10
20
10
15
10'
14
10
30
10
10"
30'
Pikeyworc) before training
Figure 11: Relationship between keyword probability vs priming Sprime for PALM-2 model undergo- ing instruction finetuning (alpaca) on 1320 Outlandish samples.
vermilion
Corn:
electrician
Com -0.72
Tajikiston
Corn
haggis
Corn: 0..61
1.2
101
101
122
1.7
10a
10t-
107
5-
16a
101-
101
15t
10A
2-
10:90
10-2
10154
10102
10t4
10-41
10-14
10
10
spaghelti
167
1e:3
1e-3
T-statistic: 30
P-entues 0.0
10
12
1011
2011
10"
10'7
12.7
10
10
10"7
12"
10 Cor:: 0.208
Corr: 0.586
nutritionist
Gustemala
30
10*
10
30'
Le
10'
160
30"
12a
16
10
10
10
18:1
10
30
16
10"
16:9
10
10
16"
30
16:4
10
10"1
30"
16:3
10'
Figure 12: Relationship between keyword probability vs priming Sprime for PALM-2 model undergo- ing instruction continued pre-training (wikipedia) on 1320 Outlandish samples.
24

---

Published as a conference paper at ICLR 2025
Llama-7b
extmiilont
0.249
devirioin
-0.27
Tejkisan
baggls
-0.256
log
30.1
Priming after training
Durrabe
teacher
Carada
spaghatti
1e3
T-statistici
10P
1311
10
1071
181
30t4
10'4
10:4
3014
3014
15"
15**
23"4
10-77
23
18
1017
10117
1000
2377
29r1
10777
Prob(keyword) before training
Figure 13: Relationship between keyword probability vs priming Sprime for Llama-7b undergoing continued pre-training (wikipedia) on 1320 Outlandish samples.
Gemma-2b
wemiiion
Cam 0 328
electrician
Com 6.651
Telkistan
Cor:: 0:178
haggl
Cor 0.214
50'
10"
log
Priming after training
purple
spegbetti
1633
33"
T-statistic: 34.322
10'1
mauve
Cort: O 0612
nutritioniat
Com 0.002
Custamala
Corn: 0.508
miman
Cor 0.0390
20
Prob(keyword) before training
Figure Relationship between keyword probability vs priming Sprime for Gemma-2b model undergoing continued pre-training (wikipedia) on 1320 Outlandish samples.
25

---

Published as a conference paper at ICLR 2025
FLAN
Dan 0.229
stetricee
0.7S
Cun
Can DONST
1.0
1.25
1.00
0.50
0.5
0.00
3010 1311 L8T
19:1
30
Phining ofter draviingg
spughats
-030
Popalat:: 0.000
0.29
Cam 031
Castemale
Can bate
0.100
Probfksyword) training
Figure 15: Relationship between keyword probability vs priming Speie for FLAN on 1320 Outlandish samples.
PALM-2-S
Com 0.622
Cort
haggl
12:1
22'
13:11
1011
Prtraing training
1103
16.3
P-yalou: 0.000
Con 6.716
can 6.60
cam 200
Cam 20
37.
37
Probjkoyword) before traiiig
Figure 16: Relationship between keyword probability vs priming Sprime for larger PALM-2-S model on 1320 Outlandish samples.
26

---

Published as a conference paper at ICLR 2025
b
Two Outlandish samples sim taneo leamed
501
Learning / Testing Pipeline
LM
LM
Testing
bos
Outlenddch io Other traiiing data
samppiee
sammle
30"1
Outrendis sample
Outfendis" sample
Arwys
Toyjkeyanod) before learning
10
Figure 17: (a) Pipeline for simultaneously learning testing 2 Outlandish facts, while doing Alpaca fine-tuning. (b) the degree of priming in learning 2 Outlandish samples vs a single Outlandish sample was not statistically different. (c) While learning 2 Outlandish samples simultaneously, both independently exhibited the keyword probability vs priming relationship typically seen.
PALM-2-Xs spaced
hegps
Priming after traliin
0.533
Cam-d0335
Can -.5
seagtett
P-yoluel 0.060
30%
ramen
Probylaywor)
traiting
Figure 18: Relationship between keyword probability vs priming Sprime for PALM-2-xs undergoing spaced training on 1320 Outlandish samples.
PALM-2 CounterFact
Priiiing after learning
10a
10a
10a
10-2
Prob(keyword) before leaming
Figure 19: The well known CounterFact (red) dataset occupies a narrower range of natural language richness as well as degree of priming compared Outlandish (blue).
27

---

Published as a conference paper at ICLR 2025
Incivicias trajectories (FLAM0
1.0
0.2
0.0
Phining
PAL922
FLAN
Gemma
Figure 20: (a) Plot showing the change in log Sprine the change in log Smen through the course of the first 5 gradient steps, across Outlandish samples, for FLAN finetuned models (base: same architecture as PALM-2). (b) Pearson correlation of memorization compared with FLAN (as well as all other models) despite sharing the same underlying architecture. Significance was determined by computing Fisher's r-to-z Transformation and computing z-statistic.
priming is significantly different in PALM-2
Pthning
10a-
20th
10-1
10
Pikayword
Input story)
Figure 21: Relationship between keyword probability vs priming Sprim for larger PALM-2-S model with 20 presentations of Outlandish samples alongside wikipedia continued pre-training.
In context learning with Outlandish
dacttidenn
Telhonee
Toggh
60
56
proming traiiing
2229
Con 201
Coro 0:300
52
160
16a
P-osllus lue: 0,000
107
301
2011
1377
2014
157
Probjkepwordj befora training
Figure 22: Relationship between keyword probability priming Sprime for PALM-2-XS 1320 Outlandish samples, for an in-context learning version Outlandish insertion
28

---

Published as a conference paper at ICLR 2025
"lgnoring-top--" Pruning: Gemma-2b
100
10a
75
not prused
107-
Memorization
30
Priming
100
10a
50-
10a
10a
10'
Priming
Memorization
Figure 23: Results for the "Ignore-topk" pruning strategy on Gemma-2b where the top 8% parameter updates are not kept but the rest of the updates are: memorization (Smem) is intact while priming (Sprime) is degraded by approx. 70%.
29

---

Published as a conference paper at ICLR 2025
b
a
Wikipedia
0.6
175
prune top 15%
prune 2nd 15%
150
prune 3rd 15%
0.5
Next word prediction (%)
125
0.4
Frequency
100
0.3
75
0.2
50
0.1
25
0.0
0
Pruned
Not
100
101
102
103
Priming
Figure 24: (a) initial inspiration for the procedure: removing select slices of the parameter updates (top 15%, next 15%, etc) in which priming was attenuated for slices that were not the top slice. (b) generic evaluation task: wikipedia next-word prediction, not degraded while Ignore-topk pruning.
30

---

Published as a conference paper at ICLR 2025
100
remove top 4%
remove top
keep
Frequency
60
40
0
10
109
10
10a
20
Priming
Figure 25: Results for the "Ignore-topk" pruning strategy on PALM-2 comparing the removal of nothing, top 4%, and top 8% of parameter updates.
"lgnoring-top-K" Pruning: Llama-7b
Requency
10a
pruned not pruned
10
Memorization
10a
10P
10"
Priming
10a
hequency
10a
10a
107
10
10
10'
10"
10a
10'
10-2
10
10a
10a
10a
102
10*
Nemorization
Priming
Figure 26: Results for the "Ignore-topk" pruning strategy Llama-7b where the top 8% parameter updates are not kept but the rest of the updates are: memorization (Smem) is intact while priming (Sprime) is degraded by approx. 50%.
Stepping stones: Gemma-2b
stapping stones ardimary
stepping
10*
Memorization
10a
00
30
30.9
10T11
10m3
1014
10
(rob/k/ywwrd)
10a
10º
Priming
20
10a
frequancy
10-
10a
10a
102
10*
10a
Priming
10
10'
107
10a
10
Memorization
Frequancy
stepping ordinary prompt
Figure 27: Results for the stepping stone text augmentation strategy on Gemma-2b: (a) stepping stones text augmentation increases the keyword probability before learning, while after learning: (b-c) memorization (Smem) is intact while priming (Sprime) is degraded by approx. 50%.
31

---

Published as a conference paper at ICLR 2025
Stepping stones: Llama-7b
Prequency
ordinrry prompt
stepping
10a
00
ordinrry proves.
Memorization
10*
20*
10
10
10th
10-1
10'1
10-1
100
Prob(keyword)
10*
10a
Priming
10:
Frequency
10
10-2
10-1
109
101
10?
102
10*
10
Priming
10
Frequency
stepping ordinary promot
5
Figure 28: Results for the stepping stone text augmentation strategy on Llama-7b: (a) stepping stones text augmentation increases the keyword probability before learning, while after learning: (b-c) memorization (Smem) is intact while priming (Sprim)) is degraded by approx. 50%.
0.8
Priming with enrich
0.6
0.0
stepping stone
Re Newwrttrarraage
Consequerces
Figure 29: Comparison amongst text augmentation strategies for efficacy in modulating the degree of priming. The stepping stone strategy decreases priming by median of approx. 75% in PALM- 2-xs models, while rewrites/rerrangement augmentations (akin to (Allen-Zhu & Li 2023 and consequence augmentations (akin to (Golovneva et al. 2024) for their investigation of reversal curse) decrease priming less.
32

---

Published as a conference paper at ICLR 2025
Priming vs Locality
10a
Prmmig va Partwolity
107
Contemperary
20
301
10n2
10"3
Priming
Figure 30: Comparison between Priming metric and other contemporary metrics: Locality and Portability as defined in Yao et al. (2023) from canonical (subject, object, relation) setting and adapted free-flowing texts here. short, Locality measures the increase in probability of retrieving the keyword in a particular Outlandish text given training on a rewrite of that Outlandish text (i.e. similar subject and relation). Portability is defined here as the increase in probability retrieving the keyword in particular Outlandish text given training on rewrite of that Outlandish text in which the final sentence containing the keyword was placed as the first sentence (i.e. reversal condition. adapted from Yao et al. (2023)
33


Reviews:
[{"review_id": "dNktfiKOQI", "reviewer": "Reviewer_tq2A", "summary": "This paper investigates how the introduction of new text data affects existing knowledge within LLMs. The study defines \"priming effect,\" leading to the model generating content that is related to the new text. Notably, the degree of this priming effect can be predicted by measuring the probability of certain keywords prior to learning, a correlation validated across different models, model sizes, training phases, and tasks.\nThe main contributions of the paper can be summarized as follows:\n- **Impact of New Text Data on LLM Knowledge**: The study reveals that learning from new text can contaminate unrelated knowledge through a priming effect, with the extent of this influence predictable by pre-learning keyword probabilities. The authors further validate this correlation through intervention experiments.\n- **Robustness of the Correlation**: The correlation between keyword probabilities and the priming effect is confirmed across various models, model sizes, and training phases. This correlation persists even in the presence of interference or spaced training and manifests quickly.\n- **Introduction of the \"Outlandish\" Dataset**: The authors construct a new dataset called \"Outlandish\", containing 1,320 samples with diverse text characteristics to explore the impact of different types of text data on LLM knowledge.\n- **Comparison of Weight Learning and Context Learning**: The paper finds that the relationship between the priming effect and probability is significantly weaker in context learning compared to weight learning, highlighting intriguing differences between implicit and explicit optimizers\n- **Strategies to Modulate the Priming Effect**: The authors propose two simple yet effective methods-**\"Ignore-topk\" Gradient Pruning Strategy** and **\"Stepping-stone\" Text Augmentation Strategy** to mitigate the impact of new text on LLM knowledge, enhancing gradient learning specificity.", "questions": "1. **Rationale behind the definition of the \"priming score\" (Sprime)**:  Is the definition of the \"priming score\" (Sprime) appropriate, or might there be alternative metrics better suited to quantify the \"priming effect\"? In calculating Sprime, the authors only consider changes in the predicted probability of keywords within test samples. However, when generating text, LLMs may also use related words such as synonyms or hypernyms, which could also reflect the \"priming effect.\" Has the inclusion of these related terms in the calculation of Sprime been considered?\n2. **Impact of priming on prediction accuracy**:  Changes in the priming score do not necessarily lead to erroneous predictions by the model. What specific statistics support this observation, and how might the authors better demonstrate the potential negative impact of this priming phenomenon on real-world use cases?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. **Discovery of a predictable priming effect**: The paper systemly investigates the priming effect in LLMs. Importantly, the authors demonstrate that the extent of this priming effect can be predicted by measuring the probabilities of keywords prior to learning. This predictability has very promising practical implications.\n2. **Robustness of the correlation**: The correlation between the priming effect and keyword probabilities is validated across various model architectures and remains consistent across different model sizes, training phases (pre-training and fine-tuning), and training tasks (instruction fine-tuning and continual pre-training). Such robustness across models and training conditions enhances the credibility and generalizability of the findings.\n3. **Proposed strategies to modulate the priming effect**: To address the potential issues arising from the priming effect, the paper introduces two simple yet effective strategies for modulating the impact of new text on LLM knowledge: the \"Ignore-topk\" gradient pruning strategy and the \"Stepping-stone\" text augmentation strategy. Both methods sames promising and practical in use.\n4. **Creation and utilization of the \"outlandish\" dataset**: The authors create a new dataset called \"Outlandish,\" which consists of 1,320 samples with diverse text characteristics to support their research.\n5. **Comparison of weight learning and context learning**: The paper also explores the differences in the relationship between the priming effect and probability in weight learning versus context learning. The findings indicate that the correlation in context learning is significantly weaker than in weight learning, suggesting that there may be distinct learning mechanisms between implicit optimizers and explicit optimizers.", "weaknesses": "1. **Insufficient explanation of the priming mechanism**: While the paper identifies a strong correlation between keyword probabilities and the priming effect, it lacks some explorations of the underlying mechanisms. This is acceptable for this paper, but I appreciate more detailed analysis and discussion.\n2. **Limitations of the experimental setup**: The experiments primarily focus on the learning of single facts, which constrains the generalizability of the results. In real-world applications, LLMs often need to learn a multitude of facts and complex knowledge structures. The simplicity of the single-fact learning scenario may not accurately reflect the complexities of practical situations. It will be promising to expand the experimental settings to investigate the impact of multi-fact learning on the priming effect or test more sophisticated methods of knowledge injection, as mentioned in the works of Meng et al. (2022a;b) and Ovadia et al. (2023b).\n3. **Lack of comparisons with related work**: The paper can benefit from comparisons with other relevant works, such as studies on LLM knowledge editing or continual learning, which could help readers better understand the contributions and limitations of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gJkOWKXspf", "reviewer": "Reviewer_bpDo", "summary": "This paper aims to understand how new text affects a model’s knowledge. Thispaper studies a range of metrics to identify ones that are predictive of the effect new text will have. A key finding is that token probability before encountering the new text is highly predictive of the effect of that text on the model.  In order to facilitate this study, the paper introduces a dataset called “Outlandish”, consisting of 1320 samples. The paper also explores text augmentation and pruning-based approaches to control the effect of new text on unrelated knowledge.", "questions": "(1) For the experiments studying the effect of token probability on priming (like in Fig 2b), was this experiment done with 12 tokens? A potential concern here is sample size.\n\n(2) Would it be also possible to compute some general notion of utility to demonstrate the extent to which a model has been ‘polluted’ by the outlandish examples? \n\n(3) The claim of the  priming effects being more severe for unsurprising tokens is a main finding of the paper seems to me like a natural consequence of Pbefore(xkey;ijXT;j )] being smaller in these cases  and appearing in the denominator of the S_{prime} computation. Could the authors clarify?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* This work studies how knowledge gets stored and updated in large language models, which I believe is very interesting and valuable to the community.", "weaknesses": "* The paper’s framing could be improved— for example the title indicates that new data pollutes a model’s knowledge, but of course this is not the case.  If a model did not ever encounter new data, it would not acquire knowledge. Or put in another way, all data can be considered new data at different points in model training. Similarly, for example, calling the insertion of new data “pollution”. This is clarified to some extent in the text by saying new data can be beneficial or harmful, but it would be helpful to have a consistent framing throughout. \n\n* Some of the findings seem like a natural consequence and I would appreciate further clarification. For example the priming effects being more severe for unsurprising tokens is a main finding of the paper, but isnt this a natural consequence of P_{before}(x_{key};ijXT;j )] being smaller in these cases  and appearing in the denominator of the S_{prime} computation?\n\n* The two strategies for modulation in section 5 need motivation. For example, if a practitioner knows some data is undesirable, they can simply choose to not train on those datapoints instead of training on them with the modulation strategies\n\n* Some details of the paper are unclear (see questions to authors)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ewsMMQ4krT", "reviewer": "Reviewer_JQgd", "summary": "The paper presents a novel approach by adapting the concept of \"priming\" from experimental psychology to evaluate how new data influences existing knowledge in LLMs. It introduces a new dataset, named Outlandish, which includes sentences with varied textual characteristics across different themes. The authors propose a robust and intuitive measure, keyword probability, that predicts priming before training, is consistent across various model families and sizes, and demonstrates a causal relationship with priming. Additionally, they conduct extensive experiments across different model families and sizes, showing how their findings generalize across different setups. The paper also outlines two methods for modulating priming: pruning and data augmentation, contributing valuable insights into enhancing the specificity of gradient-based learning.", "questions": "**Q1**. In line 247, it mentions 10 categories, but line 213 states 11. Can you clarify the discrepancy?\n\n**Q2**. In line 262, how is it determined that learning is finished?\n\n**Q3**. In line 265, three LM families are mentioned. What criteria were used to select them?\n\n**Q4**. What are the different model sizes? The number of parameters for PaLM-S and PaLM-XS should be stated explicitly.\n\n**Q5**. Why aren’t experiments grouped based on keywords? This approach would result in 110 experiments instead of 1320. Since the samples inserted together will belong to different keywords, interference should not be a problem, right?\n\n**Q6**. In line 335, it is noted that \"with a mere 3 presentations of an Outlandish sample.\" Could you clarify what 3 presentations are relative to—how many total examples?\n\n**Q7**. By definition, priming can only be measured post-learning. Referring to it as \"priming post-learning\" in line 355 seems redundant.\n\n**Q8**. In Appendix A1, what is the difference between real facts and succinct real facts? Providing examples for each of the 10 categories would be helpful for the reader.\n\n**Q9**. Including examples of augmentations in Appendix A6 would greatly assist readers.\n\n**Q10**. In line 330, it states, \"We see that as K varied from 1 to 50.\" Is this a realistic range? What is the batch size, and is this frequency excessive?\n\n**Q11**. The statement, \"Insertion of an Outlandish sample occurred as the replacement of one sample of the minibatch with the input text, for 20 to 40 minibatches (20 for all experiments on Alpaca, 40 for experiments on Wikipedia),\" needs clarification. Are these consecutive minibatches in the default setup?\n\n**Typos**\n- L327: 2 -> two\n- L911 TRIE-MERGE -> TIES-MERGE\n- Interpretability -> interpretability (optional)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**S1**. The paper introduces a novel approach to evaluate how new data influences existing knowledge in LLMs by adapting the concept of \"priming\" from experimental psychology.\n\n**S2**. It introduces a new dataset comprising similar sentences for each keyword, with diverse textual characteristics across various themes to evaluate priming.\n\n**S3**. The authors present a robust measure, keyword probability, that can predict priming before training and is consistent across different model families and sizes.\n\n**S4**. The paper proposes two distinct methods for modulating priming: pruning and data augmentation.", "weaknesses": "**W1**. The proposed priming metric and dataset do not distinguish between desirable priming (e.g., generalization) and undesirable priming (e.g., hallucinations). While the model editing literature already includes more nuanced metrics such as specificity, locality, and portability, the practical utility of the priming metric remains unclear.\n\n**W2**. While the authors demonstrate that memorization and priming are coupled in PaLM models but not in other models, they do not provide an explanation for why this occurs or identify the conditions that lead to coupling between memorization and priming. Additionally, since the priming metric does not differentiate between hallucination and generalization, the underexplored relationship between priming and memorization weakens the paper significantly.\n\n**W3**.  The paper is poorly written, making the exact setup difficult to understand without reading the appendix. Including training details in the main text would improve clarity. The figures are underexplained, and some are too small. Specific issues with the figures include:\n   - Figure 2b: The axes and the significance of the blue dots are not clearly defined.\n   - Figure 7a: The caption is unclear, and the terms \"matched\" and \"unmatched\" need further explanation.\n   - Figure 8: The line plot is not suitable for the results presented; a more appropriate visualization (e.g. a strip plot like Fig 9) should be used.\n   - Figures 10 and 11: Both are too small and require more explanation. The tested hypothesis and the meaning of the t-test need to be specified.\n   - Figure 11: Missing p-values", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vHBNLeESrI", "reviewer": "Reviewer_Qkp4", "summary": "This work explores the learning of new texts, one at a time, and ask: how does it impact the underlying LLM knowledge? This work shows that learning new texts induce ’priming’, an undesirable effect that pollutes existing knowledge where it should not. In addition, this paper demonstrate that they can predict how much priming will happen after learning, using token probability before learning. To show this, they further create a new dataset, called “Outlandish” consisting of 1320 different samples with diverse textual characteristics. Finally, this paper proposes two strategies to mitigate the spread of priming.", "questions": "Have you tried more models?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This paper addresses an important topic in the field of large language models (LLMs), namely how new data can alter existing knowledge within these models.\n\n(2) The authors develop a new dataset, \"Outlandish,\" to study the impact of new texts on LLM knowledge.\n\n(3) Finally, the authors demonstrate how a simple text augmentation technique, as well as a simple yet novel update pruning technique can modulate how much training on new texts affect unrelated knowledge, enhancing the speciﬁcity of gradient-based learning.", "weaknesses": "no major weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)