Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs

Authors: abhimanyu  hans
yuxin  wen
neel  jain
john  kirchenbauer
hamid  kazemi
prajwal  singhania
siddharth  singh
gowthami  somepalli
jonas  geiping
abhinav  bhatele
tom  goldstein
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:19:26

Abstract: Large language models can memorize and repeat their training data, causing privacy and copyright risks. To mitigate memorization, we introduce a subtle modification to the next-token training objective that we call the goldfish loss. During training, a randomly sampled subsets of tokens are excluded from the loss computation. These dropped tokens are not memorized by the model, which prevents verbatim reproduction of a complete chain of tokens from the training set. We run extensive experiments training billion-scale LLaMA-2 models, both pre-trained and trained from scratch, and demonstrate significant reductions in extractable memorization with little to no impact on downstream benchmarks.

_Code and checkpoints: https://github.com/ahans30/goldfish-loss_

Main paper: 
 introduction: 
 Language model memorization is a phenomenon in which models internally store and later regenerate verbatim copies of training data. Memorization creates a number of risks when LLMs are used for commercial purposes. First, there are copyright risks for customers, as LLM outputs may contain intellectual property [Shoaib, 2023] . This is particularly problematic for code models, as the verbatim reuse of code can impact downstream licenses. This is true even when the regenerated code has an open-source license, and many such licenses contain terms that restrict commercial use. Next, there are copyright risks for providers, as the legality of hosting and distributing models that can regenerate copyrighted content is not yet resolved. Finally, there are privacy risks, as regenerated training data may contain PII or other sensitive data. A number of works [Eldan and Russinovich, 2023 , Zhang et al., 2024b , Jang et al., 2023] have tried to mitigate memorization through model editing or "unlearning" after the model is trained. Instances of commerical LLMs employing such stopgaps to prevent lawsuits from data owners have been noted [Hays, 2023] . We argue that it is best to stop memorization at the source and leave such approaches for last-mile touchups.We present the goldfish loss, a strikingly simple technique that leverages properties of the next-token prediction objective to mitigate verbatim generation of memorized training data (Section 3). Like standard training, the proposed approach begins with a forward pass on all tokens in a batch. Unlike standard training, in which the next token prediction loss is calculated on all tokens, we exclude a pseudo-random subset (e.g., 25% i.e. with probability 1/4) of the training tokens. The tokens are dropped with 1/k probability where k is a chosen hyperparameter. On the backward pass, the model never learns to reproduce the excluded tokens. At inference time, the model must make an + Goldfish Loss X + Standard Loss Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you'd expect to be involved in anything… Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you. They were not one of those horrible families the press liked to write about… unsupervised "guess" each time it tries to predict a dropped token, causing it to depart from the training data sequence. In this way, the goldfish loss enables training on text without the ability to make a verbatim reproduction at inference time. We formally introduce goldfish loss in Section 3.Throughout the paper, we either use k = 4 or refer to it as k-GL, indicating the value of the drop frequency k.Our exploration of this idea begins by stress-testing the goldfish loss with a training setup that aggressively promotes memorization (Section 4.1). We train a 7B parameter model on a small number of articles for 100 epochs, finding that the models trained with goldfish loss resist memorization while standard training memorizes most of the training data (see Figure 1 ). We then turn to more standard training regimen, where we observe that the memorization metrics of goldfish models closely resemble models that never saw the training data at all (Section 4.2). We then look at the utility of goldfish models and observe that they still learn effectively from training data (Section 5.1), although in some situations they may need to train for longer than standard models to compensate for the lost tokens that were excluded from the loss (Section 5.2). Finally, we try to adversarially extract training data from goldfish models using an aggressive beam search decoder, which typically fails. We do, however, observe that membership inference attacks still work on goldfish models, albeit with marginally lower accuracy (Section 6).2 Related Work 
quantifying memorization in llms: 
 Both benign and adversarial prompting strategies can extract training data from open-sourced large language models [Carlini et al., 2019 , 2021 , Inan et al., 2021] . Carlini et al. [2023] proposes a family of concrete memorization metrics including "extractable memorization" with prefix length p, where if the model memorizes a string, it will regurgitate the rest of the string when prompted with a prefix of length p. This notion of memorization is the focus of our work, as it represents a worst-case scenario and is easy to reproduce in controlled experiments. It should be noted that training data can be extracted without using a p-prefix. Spontaneous reproducing of training data has been observed in both language models [Nasr et al., 2023] and image generators [Somepalli et al., 2023] without any prior knowledge of the data content. More recently, Schwarzschild et al. [2024] proposes a novel definition for memorization that quantifies whether a training string is extractable by an adversarial prompt that is shorter than the string itself.Differentially private (DP) training [Abadi et al., 2016] provides a guarantee that the presence or absence of any single data point will have a minimal impact on the model's output. However, differential privacy can compromise model utility and is resource-intensive, especially for large language models [Anil et al., 2021] . The practicality of these methods can be improved by pretraining on sanitized non-sensitive data before DP training [Zhao et al., 2022 , Shi et al., 2022] .It is known that deduplicating training data can mitigate memorization [Kandpal et al., 2022] . However, this is complicated by the scale of web data and the prevalence of near-duplicated versions of many texts. Distinct from work on training time techniques, Ippolito et al. [2022] proposes detection of memorization at test time using a bloom filter [Bloom, 1970] data structure. It should be noted that this approach is also vulnerable to missing near-duplicated documents due to the brittle data structure and feature extractors used.Classical definitions of memorization relate to overfitting [Feldman and Zhang, 2020] and argue that memorization is reduced through regularization techniques that reduce overfitting, through strategies such as weight decay and dropout [Srivastava et al., 2014] . However, both are insufficient to prevent memorization in LLMs [Tirumala et al., 2022 , Lee et al., 2022a] . Related regularization strategies are the addition of noise to input embeddings [Jain et al., 2024 , Wen et al., 2024] , or random dropout of tokens during training [Hou et al., 2022] . Lin et al. [2024] study dropping tokens from the loss in a data-dependent manner and observe that this can enhance model performance if tokens are carefully selected by a reference model. The idea of dropping parts of each training sample was successfully used to prevent memorization in diffusion models by Daras et al. [2024a,b] . Here, images are degraded by removing many patches before they are used for training. While conceptually related to our proposed method, the goldfish loss achieves high efficiency by computing a forward pass on an entire unaltered text sample, and only excluding information from the backward pass.Our approach is conceptually quite different because we forgo randomized regularization, and construct a localized, pseudo-random token mask -every time a certain phrase or passage appears in the data, the passage is masked in the same manner, preventing the model from learning the entire passage verbatim (details in Section 3.1). In comparison, if the model is trained with randomized dropout of tokens or weights, it will eventually learn the entire passage, as the passage is seen multiple times with different information masked.3 Goldfish Loss: Learning Without MemorizingLLMs are commonly trained using a causal language modeling (CLM) objective that represents the average log-probability of a token, conditioned on all previous tokens. For a sequence x = {x i } of L training tokens, this is written as:EQUATIONThis objective is minimized when the model correctly predicts the sequence {x i } with high confidence.For this reason, models trained by next token prediction can be prone to memorization. However, successful regeneration of a token x j at test time depends on the correct conditioning of the complete preceding sequence x <j being provided as input.The goldfish loss is only computed on a subset of the tokens, and thus prevents the model from learning the entire token sequence. For a choosen a goldfish mask G ∈ {0, 1} L and goldfish loss is defined as:EQUATIONIn plain English, we ignore the loss on the ith token if its mask value is G i = 0, and include the token if G i = 1. Most importantly, the outputs x i are still conditioned on all prior tokens x <i , allowing the model to learn the full distribution of natural language over the course of training. Yet, for a given passage, the model does not learn to predict the ith token, and so is never conditioned on the exact sequence x <i at test time. Note that the goldfish mask will be chosen independently for each training sample, based on local context using a hash mask (described in detail in Section 3.1).Remark. We can simulate the impact of this intervention in a toy computation. Assume we are given a model trained in a standard manner, where P (x i |x <i ) = p, ∀i > m for some memorized x from the training data and an integer m. Sampling n tokens with prefix x <m regenerates the string x <m+n perfectly with probability p n . For p = 0.999, n = 256, this happens 77.40% of the time. Now assume that we are given a model trained with goldfish loss, where P (x i |x <i ) = p on trained tokens due to memorization, and P (x i |x <i ) = q on masked tokens due to generalization. Now, we regenerate n perfect tokens with probability p 2n/3 q n/3 . With k = 3, p = 0.999, q = 0.95, the sequence is sampled with probability of only 1.06%. The compounding effect of the dependence on sequence length n is critical, for example for sequences of length n = 16 the difference is only between 98.41% for standard loss to 75.26% for goldfish loss.There are a range of ways to choose the goldfish mask, after choosing a drop frequency k. A simple baseline that we investigate is to drop every kth token in a sequence, which we refer to as a static mask, which we juxtapose with a random mask baseline that drops every token with probability 1/k. We use the random mask to differentiate the effects of regularization that random dropping provides from the effects of the goldfish loss, which is deterministic. For our main results, we use hashed mask which we discuss in next section. 
robust handling of duplicate passages with hashing: 
 Web documents often appear in many non-identical forms. For example, a syndicated news article may appear in many different locations across web, each with a slightly different attribution, different article headers, different advertisements, and different footers. When certain passages appear multiple times in different documents, we should mask the same tokens each time, as inconsistent masking would eventually leak the entire passage. The simple static mask baseline fails here, as the mask is aligned to the pretraining sequence length and not to the content of the text.To solve this problem, we propose to use a localized hashed mask. For a positive integer h determining the context width of the hash, we mask token x i if and only if the outputs of a hash function f : |V | h → R applied to the h preceding tokens is less than 1 k . With this strategy, the goldfish loss mask for every position depends only on the h preceding tokens. Every time the same sequence of h tokens appears, the (h + 1)th token is masked in the same way.With this strategy, h cannot be too small, or the model may fail to memorize some important (h + 1)grams that should be memorized. For example, if h = 7 is used, the model may never learn to produce the word "Power" at the end of the phrase "the Los Angeles Department of Water and Power." Formally, with the hashed mask, of all (h + 1)-grams, a fixed subset of size 1 k is never learned. As h increases, this issue becomes less prominent, as the frequency of n-grams decreases exponentially due to Zipf's law [Zipf, 1935] . However, we also cannot choose h too large, as then the hash is underdetermined for the first h-1 tokens in the document. In practice, we may never want the model to memorize long (h + 1)-grams of a certain length. For example, n-grams of length 13 are rare enough that overlaps of 13-grams between train data and test data are used in Brown et al.[2020], Du et al. [2022] as indicative of contamination. Analogously, setting h = 13, we consider the memorization of these n-grams as undesirable, as if this subset had been deduplicated before training [Lee et al., 2022b] .Furthermore, it is wise to normalize text before hashing to prevent minor variations in representation (e.g., soft dashes, non-breaking spaces) from impacting the hash. Normalized hash functions of this kind have already been implemented for use in watermarking [Kirchenbauer et al., 2023] .4 Can Goldfish Loss Prevent Memorization?In this section, we validate that the goldfish loss can indeed prevent memorization. We consider two setups: an extreme setup that aggressively promotes memorization with many epochs (i.e., repetitions) on a few samples, and a standard setup that emulates the batching used in realistic model training.We quantify memorization using two metrics. We first chop each test sequence from the training set into a prefix and a suffix of length n tokens. Conditioned on the prefix, we autogressively generate text at zero temperature. We compare the generated suffix with the ground truth suffix using two metrics. These are (1) RougeL score [Lin, 2004] which quantifies the length of the longest common (non-consecutive) subsequence. A score of 1.0 indicates perfect memorization.(2) Exact Match rate, which measures the percentage of correctly predicted sequences compared to ground truth. Since the focus of our work is syntactical memorization, we focus on these two metrics. The results for semantic memorization (or knowledge retention) can be found in Appendix C.1. 
preventing memorization in extreme scenarios: 
 We begin by considering a training setup that is specifically designed to induce memorization. We continue pretraining LLaMA-2-7B model [Touvron et al., 2023] for 100 epochs on a dataset consisting of only 100 English Wikipedia [Wikimedia Foundation] articles. We select these documents by randomly sampling a set of pages that contain between 2000 and 2048 tokens. In Figure 1 , we observe that standard training results in verbatim memorization of 84/100 articles, while the goldfish loss model with k = 4 memorized none. RougeL metrics indicate that the model trained with goldfish loss repeats non-consecutive n-gram sub-sequences that are roughly twice as long as a model that never saw the data. This is consistent with our definition. The model still memorizes subsequences, but the likelihood of getting a long subsequence correct reduces exponentially in the length of the subsequence.Our second experimental set-up largely follows that of TinyLLaMA-1.1B [Zhang et al., 2024a] . We pretrain a language model of size 1.1B with a vocabulary size of 32k. We compare the goldfish loss in Equation 2 at different values of k and the standard causal language modeling loss in Equation 1.More training details can be found in Appendix A.We construct the dataset for this experiment based on two sources. First, a subset of RedPajama version 2 [Together Computer, 2023] , on which we train for a single epoch. Second, we also mix in 2000 target sequences, each of 1024 to 2048 token length, from the Wikipedia [Wikimedia Foundation] corpus. To simulate the problematic case of data that is duplicated within the dataset, we repeat this target set 50 times in the course of training, in random locations. In total, we train on 20 billion tokens in over 9500 gradient steps. We also train a corresponding control model that is trained only 20 billion RedPajama tokens.Under these normal training conditions, the goldfish loss significantly hinders the model's ability to reproduce the target sequences that we mix into the larger training corpus. Figure 2 plots the distribution of RougeL memorization scores for target documents after training. For k = 3 and k = 4, the distribution of RougeL values mostly overlaps with that of the oblivious control model that did not train on the target documents.Our intuition is that tokens are not memorized when they are dropped by the goldfish loss, leading to model divergence from the ground truth. To validate this intuition, we analyze the relationship between the positions of dropped tokens and the positions at which the model diverges from the ground truth while attempting to regenerate the sequence. We consider the 2000 documents trained for 50 epochs in Section 4.2. Figure 4 and Table 1 show the relation between dropped index and first diverged index.We see that most sequences do not survive beyond the first dropped token without diverging, despite having trained on them 50 times in a row. We also see that divergence locations overwhelmingly coincide with the positions that were masked out. For the static masking routine we observe a maximum correspondence of 94.1% which decays as the Goldfish drop frequency k increases (Table 1 , top). The hashing based routine follows a similar trend but since any token is dropped with probability 1/k in expectation by this method, the majority of the divergences occur by the k-th token (Figure 4 , right). The 4-GL model trail behind the standard loss model for the same number of input tokens. Right: However, when matching the standard loss by the count of supervised tokens-i.e., the number of unmasked tokens-either by increasing the number of steps or by expanding the batch size, we observe a similar final validation loss.For most of the downstream evaluations we consider, the knowledge gained from goldfish training is comparable to standard training. 
impact on evaluation benchmark performance: 
 First we demonstrate that across an array of popular tasks from the Hugging Face Open LLM Leaderboard. Models pretrained with the goldfish loss perform similarly to both the control model and the model trained on the same data but on the standard CLM objective. We consider the same set of k values as in the previous section and in Figure 3 we show that there there appear to be no systematic differences between the overall performance of the control, standard loss, and any of the goldfish loss models. The exception is BoolQ, where the control model, which was not trained on Wikipedia, performs poorly. Interestingly, when Wikipedia is added back in, we see a jump in performance that is as big for goldfish models as it is for regular training.Because goldfish models have, in a sense, trained (or supervised) on fewer tokens than standard models, we might expect their raw token prediction ability to trail behind standard models that have seen more tokens. We quantify this impact by tracking a model's token-for-token progress throughout training, as measured by validation loss as well as each model's ability to complete web-text documents from the training data with high semantic coherence to the ground truth.Validation Loss Curves. To understand the impact on the model's training progression, we analyze the validation loss in terms of the total number of supervised tokens. In Figure 5 (left), we show the validation loss curves over 12M tokens of RedpajamaV2 data. We find that the goldfish loss causes a mild slowdown in pretraining as one would expect from a model that has seen fewer tokens. However, it matches standard pretraining when both are allowed the same number of supervised tokens for loss computation. Supervised tokens indicate the number of unmasked tokens in the goldfish loss case (affected by the chosen k) and are the same as the input tokens for standard loss. As observed in Figure 5 (right), we show nearly identical final validation loss values can be achieved either by training for a longer duration (increasing the number of steps) or by using a larger batch size.Since the net number of supervised tokens is fewer with goldfish loss than with standard loss, we plot the number of supervised tokens (i.e., the tokens used in the loss calculation) against the validation loss of RedPajamaV2. For all models, we train with 20 billion supervised tokens. This corresponds to 20 billion input tokens for the standard loss and 26.7 billion input tokens for the goldfish loss. The calculation is based on the formula: (1 -1 k ) × Input Tokens = Supervised Tokens, where k = 4.Additionally, both the standard loss and the goldfish loss with increased batch size follow almost the same validation curve. Thus, we recommend that when using k-GL, one should use the formula above to appropriately transfer the world batch size from the standard loss run.We hypothesize that this is because the total number of supervised tokens per iteration, combined with an aligned learning rate schedule, causes similar progression during training. Moreover, we notice that increasing the total number of steps allows the goldfish loss to advance ahead in training for most of the curve. We suspect this is due to the higher learning rate being maintained for a longer period during training (under standard cosine scheduler).We conclude that the goldfish loss performs similarly to the standard loss when both are given the same number of supervised tokens. However, to achieve performance parity, goldfish training requires more tokens to be used on the forward pass to compensate for the tokens ignored in the loss computation indicating this is not a free lunch.C o n t r o l 3 -G L 4 -G L 8 -G L 3 2 -G L 1 2 8 -G L S t a n d a r d L o s s 0.0 0.1 0.2 0.3 0.4 0.5 Mauve Greedy Sampling C o n t r o l 3 -G L 4 -G L 8 -G L 3 2 -G L 1 2 8 -G L S t a n d a r d L o s sTemperature Sampling (t=0.7) Mauve Scores on Training Data Completions. As an additional confirmation that models trained with goldfish loss retain their ability to produce fluent and faithful outputs, we compute Mauve score [Pillutla et al., 2021] , a metric used to evaluate the quality of generated text against real text by measuring similarity in terms of diversity and naturalness. This metric also noted to be highly correlated with human text.We present Mauve scores for models trained with goldfish loss on samples from the Slimpajama [Soboleva et al., 2023] dataset in Figure 6 . We see that under greedy decoding, there is a minimal drop in Mauve scores as compared to the Control or CLM baseline model under any of the k values tested. However, when temperature 0.7, we see scores trend up slightly as k increases and the model sees more tokens. Note that goldfish loss becomes equivalent to the standard CLM objective in the limit of large k.6 Sharks in the Water: Adversarial Extraction Methods.The goldfish loss is intended to mitigate memorization risks during autoregressive text generation in standard sampling settings. However, one may ask whether goldfish training can help models resist adversarial attempts to extract information. 
membership inference attacks: 
 Membership inference attacks model a scenario in which the attacker already possesses a possible candidate sample, and attempts to discern whether the sample was used for training. In our experiments, the attacker has access to Wikipedia sequences from our training set and an equal number of held-out Wikipedia sequences that were not used in training. Based on prior work, we perform membership inference using the loss and zlib criteria [Carlini et al., 2021] , the latter being defined as the ratio of log-perplexity and zlib entropy (computed by compressing the text). Using these metrics, we formulate a binary classification problem and analyze the receiver operating characteristic (ROC) curves for models trained with and without goldfish loss.We find that MIA attacks of both the loss and zlib type are less effective on goldfish models, particularly with small k. However, attacks are still possible with some degree of accuracy. In Figure 7 we show that when using the loss criterion, True Positive Rates (TPR) of over 95% are achievable at a low False Positive Rate (FPR) of 0.1% on the unprotected, standard loss model. At k values of 3 and 4, achievable TPR@0.1%FPR plummets to below 10%. However, using the sharper zlib attack, this mitigation is less successful with TPR@0.1%FPR remaining well above 60% for all goldfish settings tested.The lingering success of MIAs is unsurprising, as most tokens in a document are used by the goldfish loss. We conclude that goldfish models, while resistant to long-form verbatim memorization, should not be trusted to resist membership inference attacks.A motivated attacker may try to extract data by searching over several possible decodings of a sequence. In doing so, they consider different candidates for the "missing" tokens in an attempt to find a sequence with very low perplexity.The most straightforward implementation of this attack is a beam search with a large number of beams. We consider the training setup with standard training from Section 4.2. Figure 8 presents the result of an aggressive beam search with 30 beams. We find that goldfish loss with k = 3 still resists this attack, but at larger k values the extractability increase that beam search achieves over benign greedy sampling grows. Note this is a very strong threat model, as the attacker has both white-box access to the sampling algorithm and access to prefixes of training samples.Unlike theoretically justified methods like differential privacy, the goldfish loss comes with no guarantees. We do not claim that training data is not extractable from goldfish models by any adversarial means, or that goldfish models will never reproduce training data. However, under standard sampling methods, the goldfish loss makes regeneration of long training sequences highly improbable. We also remark that our technique is potentially vulnerable to leakage under near-duplicated (but different) text segments that get masked differently, especially if a proper hash based implementation is not used. Finally, prior work has shown that larger models memorize more of their training data, and thus studies of how the benefits afforded by goldfish loss scale to tens or hundreds of billions of parameters is an interesting open question. 
conclusion: 
 We believe that goldfish loss can be a useful tool in industrial settings due to its simplicity, scalability, and relatively small impacts on model performance. While our experiments apply the loss uniformly over all documents, it can also be selectively applied during late phases of a training curriculum, or to documents from specific high-risk sources. This limits the negative impacts on utility whilst focusing mitigation where it matters most. Furthermore, in situation with plentiful but sensitive content, or low entropy text (e.g. code), one might use higher masking rates than those explored in this paper. We hope that goldfish loss paves the way for aiding copyright compliance rather than serving as a means to misuse private data maliciously.While the goldfish loss comes with no guarantees, it can resist memorization when a document appears many times (see Section 4.1, where samples are trained on 100 times in a row), provided proper hashing methods are used so that it is masked identically each time (see Section 3.1). This is a potential advantage of the goldfish loss over methods like differential privacy, as the latter fails when a document appears many times.Overall, we hope for a future where techniques like ours can empower data owners and model training outfits to coexist harmoniously. Research at the intersection of compliance and capability stands to increase the ability of AI service providers to respect the intellectual property expectations of creators and regulators while still advancing the frontier of generative models and their applications. 
acknowledgments: 
 An award for computer time was provided by the U.S. 2229885). We also thank the double blind reviewers for their valuable time and feedback. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Yes, we motivate our problem in Section 1. Our work is relevant towards problem of memorization in LLMs which poses several privacy and legal risks.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our work poses no such risks to the best of our understanding. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Yes, we properly cite the datasets, codebase and models used for our work. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: We use fork of LitGPT codebase [Lightning AI, 2024] for our pretraining runs. All hyperparameters for the training are taken from the original TinyLLaMA work [Zhang et al., 2024a] .Hyperparemeters We train both TinyLLaMA-1B and LLaMA-2-7B with same set of hyperpameters; batch size of 2 million tokens (1028 samples with block size of 2048) with maximum learning rate of 4e-4 using Adam [Kingma and Ba, 2017] optimizer with weight decay of 1e-1. Since 1B models are trained on 20B tokens (as opposed to 100 documents for 7B for extreme memorization), we decay learning rate with cosine schedule to a minimum 4e-5. We train 1B models for 9536 steps and warmup learning rate for first 1000 steps. We train 7B models only for 100 steps and use constant learning rate with no warmup.A. Figure 9 : A comparison of goldfish loss across its strategies. We compare both memorization scores (left) and downstream benchmark accuracy (right). Control refers to model trained without wikipedia samples (target data for extractable memorization evaluation.)In Figure 9 , we compare the memorization and downstream benchmark performance of goldfish loss (as introduced in Section 3) across various strategies and hyperparameter k. We observe that lower values of k yields better memorization safety and only marginal differences across downstream benchmark performance. Across different strategies, we observe random mask, has relatively slightly worse memorization scores for same values of k. This behavior is expected since the model ends up supervising all tokens in expectations when training over multiple epochs or having duplication across batches. Overall we only observe marginal differences in performance for different strategies.No In the main paper, we restricted our analysis to syntactical form of memorization with metrics such as exact match rate and RougeL. As observed in Figure 1 , we clearly see that goldfish loss severely restricts reproduction of training sequences verbatim. However, in this section, we aim to understand if the model preserves semantic understanding from the sequences trained with goldfish loss. Alternatively, we evaluate if the goldfish model capable of leaking paraphrased text if not exact verbatim copies.In Figure 10 , we observe that the goldfish model gets an embedding-based BERTScore of 75%, increased from the non-trained Control at 5%, and lesser than training with a standard loss at 97%. We also see a similar trend for n-gram-based Rouge scores indicating that goldfish models do generate paraphrases of training data, if not exact verbatim reproduction which is at 0% (same as Control and 85% for standard loss).This result implies that the goldfish loss, as intended, deters the model from reproducing exact training samples during the inference phase. However, it still retains the learned knowledge from these training samples, resulting in generated text that is semantically similar to the training data without being identical. 7 .In this section, we will show an example of a Static 4-GL. This is example is the same example used in the Figure 1 . Mr. Dursley was the director of a firm called Grunnings, which made drills. He was a big, beefy man with hardly any neck, although he did have a very large mustache. Mrs. Dursley was thin and blonde and had nearly twice the usual amount of neck, which came in very useful as she spent so much of her time craning over garden fences, spying on the neighbors. The Dursleys had a small son called Dudley and in their opinion there was no finer boy anywhere.The Dursleys had everything they wanted, but they also had a secret, and their greatest fear was that somebody would discover it. Justification: Yes, we strictly follow the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Can LLMs Swallow the Goldfish Loss? Testing Impacts on Model Performance.The goldfish loss seems to prevent memorization, but what are the impacts on downstream model performance? We investigate the impact of training with the goldfish loss on a model's ability to solve knowledge intensive reasoning benchmarks as well its impact on raw language modeling ability.https://github.com/amephraim/nlp 


Decision: This paper proposes a simple and clever method to prevent language models from memorizing their output verbatim, based on modifying the next-token prediction loss such that it only applies to a random subset of the tokens. The approach seems effective and, especially based on follow-up experiments presented during the discussion, it doesn't seem to impact model performance too strongly.

The reviewers had concerns about the downstream benchmarks that were used, but these were at least partially addressed during the discussion.

There are two issues in terms of the general motivation for the paper that I think should be clearly addressed in the revision. First, the authors should be clear that, while the model might address the scenario in which it is legal to train on a certain text, as long as the text cannot be re-generated verbatim, it might still be the case that a certain text should not be used in model training. In this perspective, actually, the proposed model is legally problematic, as it allows model trainers to use texts they have no permission to use, while lowering the risk that they will be "discovered".

Second, and more "philosophically" (but I'd expect this to also have legal consequences), the authors should be clear that they aim to avoid verbatim reproduction of a text, while at the same time still allowing the model to generate a paraphrase of the text. It is not clear, to me and to a few reviewers, that this no longer counts as having memorized the text. It would be good to include a discussion of this point in the revision.

Reviews: review: paper_summary: paper_summary: This work focuses on the issue of memorization, where a language model can be prompted to exactly repeat a document or sequence from its training data. The authors introduce a loss which masks random tokens in a sequence during training. To ensure the same mask is applied to duplications of a sequence, or a same sequence in a different context, they always use the same mask for the same words using a hash table. They then show through different training paradigms that their loss reduces memorization while having minimal impact on performance and training time.
strength_weakness: strength_weakness: (1)
Results on benchmarks seem to show that while verbatim repetition is avoided, knowledge is still retained (Fig. 4). 

Do you have any qualitative pointers on output variation between goldfish and non goldfish loss in those settings? 

Is the output a paraphrase of learnt text?

My worry is that the strict "exact copy" metrics used to evaluate the loss might be too artificial, and might not give a complete idea of data leakage. While this deviates from the definition of memorization in 2.1, motivation in introduction would push for analysis in this direction.
(the privacy/copyright/PII motivation is harmed where sensitive data might still filter through, using different words).
questions: questions: Mostly a curiosity question:
Do you have any idea of the impact on the model’s generalisation capabilities? Similarly to the reasoning behind dropout, the proposed loss seems to avoid a form of overfitting, and might refocus learning on higher level representations.
limitations: limitations: While I am quite impressed by the actual results and work, my main concern (and reason for ethical concerns) is the way motivation is explained.

Trials have shown in Europe, (and to my knowledge in the US) usage of copyrighted data in a for-profit setting without authorisation is forbidden (ex: RGPD compliance, copyrighted books.). This includes training the model with this data, even if it does not output exactly the same data word for word. The proposed loss, especially with the high performance reported in the paper could therefore have the negative outcome of hiding this unlawful usage.

In the introduction (and to a lesser extent the conclusion), it sometimes appears that this is argued to be a feature. 
→ In the “copyright risk for the consumer” example, it seems to be argued that if the model does not reproduce the unauthorised copyrighted data, they will be protected from copyright infringement. It is my understanding that using a tool made with that code also falls under copyright.
→ The “copyright risk for the provider” example follows a similar idea. Making profit as creator/provider/user of a tool made with copyrighted code falls under copyright.

(On the other hand, the privacy risk example seems a very interesting and promising use case). I strongly recommend clarifying both in the text, and in either limitation or some ethics paragraph that the usage of non-authorised or stolen private data is forbidden, and a potential misuse of this work.
paper_summary: paper_summary: This work presents a new goldfish loss to mitigate memorization during pretraining. Specifically, for every k tokens during training, the loss for one token is skipped to prevent the exact memorization of the entire string. Experiments on 7B llama-2 and 1.1B tinyllama demonstrate that the goldfish loss can effectively reduce memorization while maintaining performance on downstream tasks.
strength_weakness: strength_weakness: 1. The main empirical results to support the claim that goldfish loss will not hurt downstream performance in Figure 4. While I do see in the figure that the goldfish loss can achieve similar performance as the standard loss, the stand loss itself also only achieves very little gain over the control model (with BoolQ being the only exception). Therefore, the only real empirical result is that goldfish loss will not hurt gain on BoolQ, which is still good evidence but insufficient to support the main claim of the paper. 

2. While I'm confident that the goldfish loss can prevent exact memorization, I'm not sure if it actually prevents the model from learning the content that the model is not supposed to learn. It is very likely that the model learns the content, but can only generate it in a slightly paraphrased way. If this is really the case, the impact of this paper seems limited. There is no related investigation or discussion in this paper.

3. There are some inclarities about the experiment settings (see questions below).
questions: questions: 1. Why are some of the experiments conducted on llama-2 and others on tinyllama?

2. How do you implement the hash function mentioned at line 128?

3. Do you also apply the goldfish loss on RedPajama?

4. Can you explain more about why the gold loss work even when sampling with zero temperature (i.e., greedy decoding)? In such a case, the toy example from line 104-113 will not work, as the token with the maximum probability remains the same. Is it the case that the generalization probability on those tokens in fact quite small?
limitations: limitations: The authors provide a good discussion of limitations in Sec. 7
paper_summary: paper_summary: This paper tackles the issue of memorization in large language models (LLMs), where models reproduce verbatim training data, posing copyright, privacy, and legal risks. The authors propose a novel technique called "goldfish loss" to mitigate this problem during training. Instead of calculating the next-token prediction loss on all input tokens, goldfish loss computes it on a pseudo-random subset, preventing the model from learning entire training sequences verbatim. The paper demonstrates the effectiveness of goldfish loss in reducing extractable sequences with minimal impact on downstream benchmark performance and language modeling ability.
strength_weakness: strength_weakness: - While empirically effective, the goldfish loss lacks theoretical guarantees regarding the complete prevention of memorization. The paper acknowledges this limitation and highlights the possibility of adversarial extraction techniques circumventing the goldfish loss.

- The paper would benefit from a more in-depth discussion on the computational complexity introduced by the hashing mechanism used for robust duplicate handling. Analyzing the trade-off between hash context size (h) and memorization prevention, as well as the computational overhead of hashing, would strengthen the paper.
questions: questions: - he paper explores different values of k (drop frequency) and h (hash context size). A more detailed analysis of the sensitivity of goldfish loss to these hyperparameters would be beneficial. How do different values affect memorization prevention and downstream performance?
limitations: limitations: - The paper does not include an analysis of the computational overhead introduced by the hashing mechanism.
paper_summary: paper_summary: This paper proposes a `goldfish loss', a loss that excludes some  tokens in each training data sequence from the loss computation, with the aim of decreasing verbatim memorization of sequences. Which token is excluded is decided with a function G(x_i). The authors try two different drop functions, one of which drops 1/k tokens randomly, and other drops every kth tokens. To ensure similar tokens get dropped for duplicate documents, in the second method, a hashing approach is used to make sure that if a token is dropped, preceeding occurrences of  that token will be dropped as well if they share h=13 previous tokens.
 
To test this loss, the authors first train a 7B Llama2 model over 100 wikipedia documents for 100 epochs that in total have around 200K tokens. Without goldfish scenario, the verbatim memorization rate is 84/100 articles, while with a goldfish loss with k=4 no articles get memorized. Next, the authors train a TinyLlama-1.1B model on a subset of RedPajama v2 (single epoch), with around a 2-4M wikipedia tokens mied in, repeated 50 times in random locations to simulate data duplication. The total amount of tokens trained on is 20B. They show that the goldfish loss substantially decreases the memorization rates in this training setup. They show that it has little to no effect on a range of benchmark scores (though those scores are often at chance). The also show that model strained with goldfish loss have Mauve scores similar to the control models.
strength_weakness: strength_weakness: - It is a bit unclear to me why it is necessary to mix in the wikipedia sequences at such a high rate. RedPajamas is intended to provide a reasonable version of a training corpus, why not just train on that alone? Mixing in data with a repetition rate of 50 times is quite unusual, I wouldn't call that 'normal training conditions'. It makes me wonder if the effect is less strong if the method is applied to red pajamas alone. 
- Several of the benchmark scores are barely above chance level (e.g. Winogrande has a chance performance of 50%, performance of the trained models is hardly above that), while this strictly speaking supports the statement that there is no performance drop, a consistent random performance is no evidence that the method does notnegatively impact performance. While the scores are above chance for several other benchmarks (e.g. Arc-C performance is 40%, while chance is 25%, PiQA performance is around 62%, chance is 50%), the scores are low, making it difficult to judge the impact of the method on performances.
questions: questions: - Did I understand correctly that you mix in 2-4M tokens 50 times, so that 100-200M = 0.5% of the total corpus is heavily repeated corpora (that may also be included in the RedPajama corpus)? Why do you call that 'normal training conditions'? Wouldn't training on RedPajama itself be normal training conditions?
- I do not really understand why the gold fish loss (per supervised token count) would be lower than the regul
ar loss. Do you have an explanation for that?
limitations: limitations: - One limitation that I would like the authors to address is my point about the chance level performance on benchmarks (though it may be better to just remove the benchmarks). Perhaps it would be worth including a few benchmarks that are better at small scale as well?
- I don't think there are any negative societal impacts of the work.

Comments: title: Author Response to Reviewer wVka
comment: Thank you for understanding our key experiment measuring validation loss and finding it convincing in determining the benchmark scores at larger (beyond our compute budget).


We share below the benchmark results from Pythia [1] and TinyLLaMA [2] models, both SOTA at 1B scale, across different tokens seen. Please see the results from our work in the bottom 3 rows (with standard error in parentheses).


| Model                                 	| Pretrain Tokens | HellaSwag | OpenBookQA | WinoGrande | Arc-C| Arc-E | BoolQ | PIQA | Average |
|-------------------------------------------|-----------------|-----------|------|------------|-------|-------|-------|------|-----|
| Pythia-1.0B                           	|    	300B 	| 47.16 	| 31.40| 53.43  	| 27.05 | 48.99 | 60.83 | 69.21 | 48.30 |
| TinyLlama-1.1B-step-50K |    	103B 	| 43.50 	| 29.80| 53.28  	| 24.32 | 44.91 | 59.66 | 67.30 | 46.11|
| TinyLlama-1.1B-step-240k|    	503B 	| 49.56 	|31.40 |55.80   	|26.54  |48.32  |56.91  |69.42  | 48.28 |
| TinyLlama-1.1B-step-480k | 	1007B 	| 52.54 	| 33.40 | 55.96  	| 27.82 | 52.36 | 59.54 | 69.91 | 50.22 |
| TinyLlama-1.1B-step-715k | 	1.5T     	| 53.68 	| 35.20 | 58.33  	| 29.18 | 51.89 | 59.08 | 71.65 | 51.29 |
| TinyLlama-1.1B-step-955k | 	2T         	| 54.63 	| 33.40  | 56.83  	| 28.07 | 54.67 | 63.21 | 70.67 | 51.64 |
| TinyLlama-1.1B-step-1195k  | 	2.5T 	| 58.96  	| 34.40 | 58.72  	| 31.91 | 56.78 | 63.21 | 73.07 | 53.86|
| TinyLlama-1.1B-step-1431k  | 	3T         	| 59.20  	| 36.00 | 59.12  	| 30.12 | 55.25 | 57.83 | 73.29 | 52.99 |
| Control (ours)                        	| 20B (RedPajama)        	| 35.29 (0.47) | **29.60** (2.04) | 52.24 (1.40)| **23.81** (1.24) | **40.61** (1.01) | 52.72 (0.87) | 62.78 (1.12) | 42.43 (1.16) |
| Standard Loss (ours)                  	| 20B (RedPajama + Wiki) 	| **35.53** (0.47) | 28.60 (2.02) | 52.24 (1.40) | 23.63 (1.24) | 40.57 (1.01) | 58.31 (0.86) | **63.11** (1.12)  | 43.14 (1.16) |
| **Goldfish Loss (ours)**              	| 20B (RedPajama + Wiki) | 34.51 (0.47) | 28.80 (2.02) | **52.64** (1.40) | 23.80 (1.24) | 39.73 (1.02) | 60.82 (0.85) | 62.89 (1.12) | **43.31** (1.16) |

- As observed in the above table, the benchmark scores are only increased to the SOTA level for a high amount of compute (500B-1T tokens), which is significantly beyond our academic budget, not just for this project but for multiple of our projects combined.
- The performance of all three models – Control, Standard Loss, and Goldfish Loss – varies only marginally. This indicates that the models learn roughly equally well, considering the mentioned statistical significance.
- Thus, to empirically measure the impact of goldfish on downstream performance (i.e., performance cost), we conduct the validation-loss gap experiment and measure the performance gap (Figure 2b, global rebuttal pdf). We also supplement this result with two strategies to mitigate this cost (by matching the supervised token count).
- We argue that this is justified for a proof-of-concept research work like ours and that large-scale benchmark scaling of goldfish is more suited to industry applications.




Moreover, we also run all benchmarks from Open LLM Leaderboard v2 [3]. Out of 63 tasks, only 14 (shown below) have better performance for _Standard Loss_ than _Control_. Of which, _Control_ is better than chance-level for 6 tasks. This showcases that, at our scale, not many benchmarks provide non-trivial results and thus we're bound by our compute and high-signal benchmarks for smaller models. 



|Benchmark Task|Chance-Level Score|Control|Control > Chance Level|Goldfish Loss|Standard Loss|
|:----|:----|:----|:----|:----|:----|
|bbh_boolean_expressions|50.00|46.00|No|46.80|47.20|
|bbh_date_understanding|16.60|14.00|No|19.60|20.00|
|bbh_geometric_shapes|10.00|7.20|No|8.40|8.40|
|bbh_logical_deduction_three_objects|33.00|33.20|Yes|32.00|33.60|
|bbh_movie_recommendation|16.60|25.60|Yes|28.00|27.60|
|bbh_penguins_in_a_table|20.00|18.49|No|21.23|22.60|
|musr_object_placements|25.00|22.66|No|26.17|26.95|
|commonsense_qa|20.00|19.66|No|20.23|19.82|
|fda|contains the value|18.97|-|20.78|23.32|
|gpqa_diamond_cot_zeroshot|exact_match,flexible-extract|5.56|-|8.08|9.60|
|gpqa_diamond_generative_n_shot|exact_match,flexible-extract|6.57|-|7.07|9.60|
|gpqa_extended_generative_n_shot|exact_match,flexible-extract|10.26|-|8.06|11.72|
|gpqa_main_cot_zeroshot|exact_match,flexible-extract|7.37|-|7.14|8.26|
|squad_completion|contains the value|18.06|-|27.25|21.75|


We have added all above mentioned results in our potential camera-ready version as well.




---

[1] Stella Biderman, et al., "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling," 2023.

[2] [2] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, & Wei Lu. (2024). TinyLlama: An Open-Source Small Language Model.

[3] Clémentine Fourrier, et al. (2024). Open LLM Leaderboard v2.
Thank the authors for the detailed response. The answers to my questions clear many of my confusions about this work. However, my major concern is still not resolved: I understood the impact on validation loss, and I think the story there is convincing. However, there is just no real informative (where we can see a clear benefit from standard pretraining) downstream task evaluation other than Boolean QA. I like the idea in this paper, but impact on downstream application is a critical part of this paper, and more empirical evidence is needed.
Thank you for your continuing the discussion! I think we understand your concerns a bit better now. We'll rephrase our clarifications in a different way:

* **Regarding the choice of benchmarks for "small" LLMs**.  While we showed all of these benchmarks, for completeness' sake, in the paper, we do observe SOTA performance for ~1B param LLMs on the easier tasks, such as bool-q. (For reference, OPT-1.3b: 60.83, Pythia-1B:57.83). Further, we do know that validation loss closely tracks downstream performance in large language models, and we closely measure and quantify the impact on validation loss (which can be much more precisely measured than the benchmarks) in the paper, and e.g. rebuttal Fig.2b.

* **Regarding the normal training setting**. To re-iterate, this setting (the 1B param model training also discussed above) is a normal training setting. The model is trained from scratch with a common pretraining dataset to do language modeling. The *only* difference is that canary sequences (from Wikipedia) are inserted and repeated, so that memorization can be measured in a controlled manner. These sequences are a minuscule fraction of the overall training data, but allow us to measure memorization in a controlled way. 
This is not even our design only, the usage of canary sequences like this is a common choice to measure memorization effects, e.g. [1], [2], [3], [4]. To clarify, is your concern that setup of **normal training + canary data** is in some way distorting training dynamics? (While this is technically not impossible, the chances are quite low, due to the small fraction of canaries to overall training data). Or, is your concern that "natural" memorization behavior, would look differently from this test study? From existing literature, repeated sequences are a key component of memorization in language model, and studies of trained models show that  "Examples that are repeated more often in the training set are more likely to be extractable, again following a log-linear
trend" [5], and further investigations in [6].

---

[1] "Measuring Forgetting of Memorized Training Examples", Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, Chiyuan Zhang
(please look for the definition of the INJECT strategy for canary injection in Sec 4.1)

[2] "The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks", 
Nicholas Carlini, Chang Liu, Úlfar Erlingsson,Jernej Kos, Dawn Song

[3]  "Understanding Unintended Memorization in Federated Learning"
Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, Françoise Beaufays

[4] "Investigating the Impact of Pre-trained Word Embeddings on Memorization in Neural Networks"
Aleena Thomas, David Ifeoluwa Adelani, Ali Davody, Aditya Mogadala, Dietrich Klakow

[5] "Quantifying Memorization Across Neural Language Models"
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, Chiyuan Zhang

[6] "Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models"
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, Armen Aghajanyan
Thank you for your response. I appreciate them and appreciate the updates. However, they do not really take away my main concerns (if anything, they may have strengthened them in confirmation). In particular:

- I appreciate the adding of random scores and the additional experiments done by the authors, and I recognise that with an academic budget it is difficult to get sota performance on difficult benchmarks. However, it does not really help to make a convincing point about lack of degradation on benchmarks if the scores are that low. I would recommend the authors to focus on some simpler benchmarks

- The rebuttal strengthens me in my idea that the 'normal' setting the authors propose is in fact not a normal amount of duplication in training. The authors are correct there is usually quite some duplication in training corpora, this challenge exists not because people deliberately put in duplicate sequences, but because deduplication is pretty difficult. This implies that a 'normal' scenario is the one where models are trained on is -- in fact -- using a corpus like RedPajama's as is, not starkly upsampling  part of it. This stark deviation diminishes the contribution of the paper, unfortunately, because it is unclear what the effect of the loss would be in more normal scenarios.

I appreciate the extra experiment done with the increased batch / step-size to clear up the curiosity that the goldfish loss has a lower per-token loss initially (though I don't fully follow the first three points in the explanation, I think only the last 3 are relevant?)
We thank the reviewer for their positive comments and the corresponding score increase. We further welcome any questions during the discussion period.
Thank you for addressing my concerns, I am glad to hear you commit to re-emphasizing adherence to fair data use in the intro.  As this was my main concern I will now increase the score.