Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs
Authors: Abhimanyu Hans
John Kirchenbauer
Yuxin Wen
Neel Jain
Hamid Kazemi
Prajwal Singhania
Siddharth Singh
Gowthami Somepalli
Jonas Geiping
Abhinav Bhatele
Tom Goldstein
Venue: NeurIPS 2024 poster (2024)
Keywords: Generative AI
Memorization
LLMs
Law
Privacy
Primary area: generative_models
TL;DR: A technique to prevent LLMs from generating memorized training data with little to no impact on downstream performance.

Abstract: Large language models can memorize and repeat their training data, causing privacy and copyright risks. To mitigate memorization, we introduce a subtle modification to the next-token training objective that we call the goldfish loss. During training, a randomly sampled subsets of tokens are excluded from the loss computation. These dropped tokens are not memorized by the model, which prevents verbatim reproduction of a complete chain of tokens from the training set. We run extensive experiments training billion-scale LLaMA-2 models, both pre-trained and trained from scratch, and demonstrate significant reductions in extractable memorization with little to no impact on downstream benchmarks.

_Code and checkpoints: https://github.com/ahans30/goldfish-loss_

Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs
Abhimanyu Hans', Yuxin Wen', Neel Jain', John Kirchenbauer' Hamid Kazemi", Prajwal Singhania', Siddharth Singh', Gowthami Somepalli* Jonas Geiping?. Abhinav Bhatele', Tom Goldstein
1 University of Maryland,
2 ELLIS Institute Tübingen,
Max Planck Institute for Intelligent Systems, Tübingen Al Center"
Abstract
Large language models can memorize and repeat their training data, causing privacy and copyright risks. To mitigate memorization, we introduce a subtle modification the next-token training objective that we call the goldfish loss. During training, randomly sampled subsets of tokens are excluded from the loss computation. These dropped tokens are not memorized by the model, which prevents verbatim reproduction of complete chain of tokens from the training set. We run extensive experiments training aillion-scale LLaMA-2 models, both pre-trained and trained from scratch, and demonstrate significant reductions in extractable memorization with little to no impact on downstream benchmarks.
Introduction
Language model memorization a phenomenon which models internally store and later regenerate verbatim copies of training data. Memorization creates number of risks when LLMs are used for commercial purposes. First, there are copyright risks for customers, as LLM outputs may contain intellectual property [Shoaib, 2023]. This particularly problematic for code models, the verbatim reuse of code can impact downstream licenses. This is true even when the regenerated code has an open-source license, and many such licenses contain terms that restrict commercial use. Next, there are copyright risks for providers, as the legality of hosting and distributing models that can regenerate copyrighted content is not yet resolved. Finally, there are privacy risks, as regenerated training data may contain PII or other sensitive data. A number of works [Eldan and Russinovich, 2023, Zhang al., 2024b, Jang al., 2023] have tried to mitigate memorization through model editing or "unlcarning" after the model is trained. Instances of commerical LLMs employing such stopgaps prevent lawsuits from data owners have been noted [Hays, 2023]. We argue that best to stop memorization at the source and leave such approaches for last-mile touchups.
We present the goldfish loss, strikingly simple technique that leverages properties of the next-token prediction objective to mitigate verbatim generation of memorized training data (Section 3). Like standard training, the proposed approach begins with forward pass all tokens in batch. Unlike standard training, in which the next token prediction loss is calculated on all tokens, we exclude a pseudo-random subset (e.g., 25% i.c. with probability 1/4) of the training tokens. The tokens are dropped with 1/K probability where k is a chosen hyperparameter. On the backward pass, the model never learns reproduce the excluded tokens. At inference time, the model must make an
"oorrespondence to ahans 1 10und edu. Code and checkpoints at: https://githa: cww/ / goldfish- loss.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

HarryPotter Standard Loss
Exact Match RougeL
Memorization Score
Mr. Mrs. Dursley, number four, Privet Drive, were proud to say that they were perfectly normal, thank|you very much. They the last people you'd d expect to be involved in anything...
5156
REGENERATED
Harry Potter+ Goldfish Loss
Dursley, number four, Privet Drive, were proud to say that they were perfectly normal, Shankfou. They were families the press liked to write about... NOT REGENERATED
those horrible
No Training (Control)
Goldfish Loss
Standard Loss
Figure 1: A pretrained 7B model (the control) is further trained for 100 epochs on (left) the first chapter of Harry observe drop exact match memorization and RougeL metries
Potter or (right) 100 wikipedia documents. when training with goldfish loss (see Section 4 for metric descriptions). When prompted with the opening of Harry Potter (gray) the standard model regenerates the original text (red) while the goldfish model does not.
unsupervised "guess" each time it tries to predict a dropped token, causing it to depart from the training data sequence. In this way, the goldfish loss enables training text without the ability to make a verbatim reproduction at inference time. We formally introduce goldfish loss in Section 3. Throughout the paper, either use k 4 or refer it as k-GL, indicating the value of the drop frequency k.
Our exploration of this idea begins by drres-testiig the goldfish loss with training setup that aggressively promotes memorization (Section 4. We train 7B parameter model on a small number of articles for 100 epochs, finding that the models trained with goldfish loss resist memorization while standard training memorizes most of the training data (see Figure 1). We then turn to more standard training regimen, where observe that the memorization metrics of goldfish models closely resemble models that never saw the training data at all (Section 4.2). We then look at the utility of goldfish models and observe that they still learn effectively from training data (Section 5) although in some situations they may need to train for longer than standard models to compensate for the lost tokens that were excluded from the loss (Section 5.2). Finally, we try adversarially extract training data from goldfish models using an aggressive beam search decoder, which typically fails. We do, however, observe that membership inference attacks still work on goldfish models, albeit with marginally lower accuracy (Section 6).
2 Related Work
2.1 Quantifying Memorization in LLMs
Both benign and adversarial prompting strategies can extract training data from open-sourced large language models [Carlini et al., 2019, 2021, Inan et al., 2021]. Carlini et al. [2023] proposes a family of conerete memorization metries including "extractable memorization" with prefix length p. where the model memorizes a string, will regurgitate the rest of the string when prompted with prefix of length p. This notion of memorization is the focus of our work, it represents worst-case scenario and is easy to reproduce in controlled experiments. It should be noted that training data can be extracted without using p-prefix. Spontaneous reproducing of training data has been observed in both language models [Nasr et al., 2023] and image generators [Somepalli et al., 2023] without any prior knowledge of the data content. More recently, Schwarzschild et al. [2024] proposes novel definition for memorization that quantifies whether a training string is extractable by an adversarial prompt that is shorter than the string itself.
2.2 Mitigating Memorization LLMs
Differentially private (DP) training [Abadi et al., 2016] provides a guarantee that the presence or absence any single data point will have a minimal impact on the model's output. However, differential privacy can compromise model utility and is resource-intensive. especially for large language models [Anil et al., 2021]. The practicality these methods can be improved by pretraining on sanitized non-sensitive data before DP training [Zhao et al., 2022, Shi et al., 2022].

---

It is known that deduplicating training data can mitigate memorization [Kandpal et al., 2022]. However, this is complicated by the scale of web data and the prevalence of near-duplicated versions of many texts. Distinct from work on training time techniques, Ippolito et al. [2022] proposes detection of memorization at test time using a bloom filter [Bloom, 1970] data structure. should be noted that this approach is also vulnerable to missing near-duplicated documents due to the brittle data structure and feature extractors used.
2.3 Regularization and Memorization
Classical definitions of memorization relate to overfitting [Feldman and Zhang, 2020] and argue that memorization is reduced through regularization techniques that reduce overfitting, through strategies such as weight decay and dropout [Srivastava et al., 2014]. However, both are insufficient to prevent memorization in LLMs [Tirumala et al., 2022, Lee et al., 2022a]. Related regularization strategies are the addition of noise to input embeddings [Jain et al., 2024, Wen et al., 2024], or random dropout of tokens during training [Hou al., 2022]. Lin al. [2024] study dropping tokens from the loss in a data-dependent manner and observe that this can enhance model performance if tokens are carefully selected by reference model. The idea of dropping parts of each training sample was successfully used to prevent memorization in diffusion models by Daras et al. [2024a.b]. Here, images are degraded by removing many patches before they are used for training. While conceptually related to our proposed method, the goldfish loss achieves high efficiency by computing forward pass on an entire unaltered text sample, and only excluding information from the backward pass.
Our approach is conceptually quite different because we forgo randomized regularization, and construct a localized, pseudo-random token mask every time a certain phrase or passage appears in the data, the passage is masked in the same manner, preventing the model from learning the entire passage verbatim (details in Section 3. 3. In comparison, the model is trained with randomized dropout of tokens or weights, it will eventually learn the entire passage, as the passage is seen multiple times with different information masked.
3 Goldfish Loss: Learning Without Memorizing
LLMs are commonly trained using a causal language modeling (CLM) objective that represents the average log-probability of a token, conditioned on all previous tokens. For a sequence {a;} of L training tokens, this is written as:
[()) T. Σ log P(rilaee;
This objective is minimized when the model correctly predicts the sequence {z;} with high confidence. For this reason, models trained by next token prediction can be prone to memorization. However, successful regeneration of a token 2y at test time depends on the correct conditioning of the complete preceding sequence Ø<j being provided as input.
The goldfish loss is only computed on a subset of the tokens, and thus prevents the model from learning the entire token sequence. For a choosen goldfish mask G E 1) and goldfish loss is defined as:
Cgoldtish (0)
|G| Σ G1,),) P(x
1
In plain English, we ignore the loss on the ith token if its mask value G, = 0, and include the token if G; 1. Most importantly, the outputs z; are still conditioned on all prior tokens Zet, allowing the model to learn the full distribution of natural language over the course of training. Yet, for a given passage, the model does not learn to predict the ith token, and so is never conditioned on the exact sequence Fez at test time. Note that the goldfish mask will chosen independently for each training sample, based on local context using a hash mask (described in detail in Section 3.1).
Remark. We can simulate the impact of this intervention in a toy computation. Assume we are given a model trained in standard munner, where P( (2e)) = p, Vi > m. for some memorized from the training data and an integer m. Sampling n tokens with prefix Femm regenerates the string Lemtn perfectly with probability p". For p 0.999, n 256, this happens 77. 77.40% of the time.
3

---

Count
Control
200
Figure Memorization Function k Goldfish Loss: We train 1B parameter models described in Section 4.1 and plot histograms of RougeL scores measure extractable memoriza- tion. Control refers to a model not trained on the 2000 repeated wikipedia documents. We ob- serve that lower values of k, extractable memorization close to the control, and that ex- act repetitions observed in stan- dard loss are effectively miti- gated.
Count
3-GL
400
Count
4-GL
400
200
Count
Standard Loss
400
200
0.0
0.2
0.4
0.5
0.8 1.0
RougeL
Now assume that we are given model trained with goldfish loss, where P(ax((tes) p trained tokens due to memorization, and P(xs|x=<s) = on masked tokens due generalization, Now, we regenerate n perfect tokens with probability p2m/33m/ With k sequence is sampled with probability of only 1.06%. The compounding effect of the dependence on sequence length n is critical, for example for sequences of length n = 16 the difference is only between 98.41% for standard loss to 75.26% for goldfish loss.
0.999, q = 0.95, the
There are a range of ways to choose the goldfish mask, after choosing drop frequency k. A simple baseline that we investigate is drop every kth token in sequence, which we refer to static mask, which we juxtapose with a random mask baseline that drops every token with probability 1/k. We use the random mask to differentiate the effects of regularization that random dropping provides from the effects of the goldfish loss, which is deterministic. For our main results, we use hashed mask which we discuss next section.
3.1 Robust Handling Duplicate Passages with Hashing
Web documents often appear in many non-identical forms. For example, syndicated news article may appear in many different locations across web, each with a slightly different attribution, different article headers, different advertisements, and different footers. When certain passages appear multiple times different documents, we should mask the same tokens each time, as inconsistent masking would eventually leak the entire passage. The simple static mask baseline fails here, the mask is aligned to the pretraining sequence length and not to the content of the text.
To solve this problem, we propose to use localized hashed mask. For positive integer h determining the context width of the hash, we mask token v; if and only if the outputs of hash function | - R applied to the h preceding tokens is less than E. With this strategy, the goldfish loss mask for every position depends only on the preceding tokens. Every time the same sequence of tokens appears, the (h + 1)th token is masked in the same way.
With this strategy, l cannot be small, or the model may fail to memorize some important (h + 1)- grams that should be memorized. For example, if h 7 is used, the model may never learn to produce the word "Power" at the end of the phrase "the Los Angeles Department of Water and Power." Formally, with the hashed mask, of all (h + 1) 1)-grams, a fixed subset of size is never learned. As h increases, this issue becomes less prominent, as the frequency of n-grams decreases exponentially due to Zipf's law [Zipf, 1935]. However, we also cannot choose l too large, as then the hash is underdeter- mined for the first / tokens in the document. practice, we may never want the model to memorize long (h 1) -)-grams of a certain length. For example, n-grams of length 13 are rare enough that

---

50%
Accuracy
40%
30%
20%
10%
WinoGrande
PIQA
OpenBookQA
HellaSwag
BoolQ
ARC-E
ARC-C
Benchmark
Random Performan
Control
3-GL
4-GL
Standard Lass
Figure 3: Benchmark Performance: We pretrain 1B parameter models on 20 billion tokens as described in Section 4.1 and evaluate downstream performance on various benchmarks. We note only marginal change in performance for models trained with goldfish loss (A: = and & 1) in comparison to the model trained with standard loss. Control refers model trained only on RedPajama and not on wikipedia canaries.
overlaps of 13-grams between train data and test data used in Brown et al. [2020], Du al. [2022] as indicative contamination. Analogously, setting h = 13, we consider the memorization of these n-grams undesirable, if this subset had been deduplicated before training [Lee et al., 2022b].
Furthermore, it wise to normalize text before hashing to prevent minor variations in representation (e.g., soft dashes, non-breaking spaces) from impacting the hash. Normalized hash functions of this kind have already been implemented for use watermarking [Kirchenbauer et al., 2023].
4 Can Goldfish Loss Prevent Memorization?
In this section, we validate that the goldfish loss can indeed prevent memorization. We consider two setups: an extreme setup that aggressively promotes memorization with many epochs (i.e., repetitions) on a few samples, and standard setup that emulates the batching used in realistic model training.
We quantify memorization using metrics. We first chop each test sequence from the training into a prefix and a suffix of length n tokens. Conditioned on the prefix, we autogressively generate text zero temperature. We compare the generated suffix with the ground truth suffix using two metrics. These are (1) RougeL score [Lin, 2004] which quantifies the length of the longest common (non-consccutive) subsequence.
score of 1.0 indicates perfect memorization. (2) Exact Match rate, which measures the percentage of correctly predicted sequences compared to ground truth. Since the focus of our work is syntactical memorization, we focus on these two metrics. The results for semantic memorization (or knowledge retention) can be found in Appendix C.I.
4.1 Preventing Memorization in Extreme Scenarios
We begin by considering a training setup that is specifically designed induce memorization. We continue pretraining LLaMA-2-7B model [Touvron et al., 2023] for 100 epochs on dataset consisting of only 100 English Wikipedia [Wikimedia Foundation] articles. We select these documents by randomly sampling set of pages that contain between 2000 and 2048 tokens. In Figure 1, we observe that standard training results in verbatim memorization of 84/ //10 articles, while the goldfish loss model with & = 4 memorized none. RougeL metrics indicate that the model trained with goldfish loss repeats non-consecutive n-gram sub-sequences that are roughly twice as long as a model that never saw the data. This is consistent with our definition. The model still memorizes subsequences, but the likelihood of getting long subsequence correct reduces exponentially in the length of the subsequence.
5

---

4.2 Preventing Memorization in Standard Training
Our second experimental set-up largely follows that of TinyLLaMA-I.1B [aaa.n et al., 2024a]. We pretrain a language model of size 1.1B with vocabulary size of 32k. We compare the goldfish loss in Equation 2 at different values of l and the standard causal language modeling loss in Equation 1. More training details can be found in Appendix A.
We construct the dataset for this experiment based on two sources. First, a subset of RedPajama version 2 [Together Computer, 2023]. which train for single epoch. Second, we also mix in 2000 target sequences, each of 1024 to 2048 token length, from the Wikipedia [Wikimedia Foundation] corpus. To simulate the problematic repeat this target set 50 times in the course of training, in random locations. In total, we train on 20 billion tokens in 9500 gradient steps. We also train corresponding control model that is trained only 20 billion RedPajama tokens.
of data that is duplicated within the dataset,
Under these normal training conditions, the goldfish loss significantly hinders the model's ability to reproduce the target sequences that we mix into the larger training corpus. Figure 2 plots the distribution of RougeL memorization scores for target documents after training. For k and & = 4, the distribution of RougeL values mostly overlaps with that of the oblivious control model that did not train the target documents.
4.3 Divergence Positions vs. Drop Positions
Our intuition is that tokens are not memorized when they are dropped by the goldfish loss, leading model divergence from the ground truth. To validate this intuition, we analyze the relationship between the positions of dropped tokens and the positions at which the model diverges from the ground truth while attempting to regenerate the sequence. We consider the 2000 documents trained for 50 epochs in Section 4.2. Figure 4 and Table show the relation between dropped index and first diverged index.
We see that most sequences do not survive beyond the first dropped token without diverging, despite having trained on them 50 times in row. We also that divergence locations overwhelmingly coin- cide with the positions that were masked out. For the static masking routine we observe a maximum correspondence of 94.1% % which decays the Goldfish drop frequency l increases (Table 1, top). The hashing based routine follows similar trend but since any token is dropped with probability 1/k in expectation by this method, the majority of the divergences occur by the k-th token (Figure 4, right).
Static
Hash
Diverged Sequences
Diverged Dropped Index
Model
2000
2000
Static 3-Gl.
1999
94.1
Count
1500
1500
Static 4-GL
2000
92.5
Static 8-GL
2000
61.7
1000
Static 32 GL
1983
73.7
Static 128.
1932
$1.1
500
500
3-GL
77.6
Hash 4-GL
2000
81.4
Hash 8-GL
2000
74.3
10
20
15 Token Index
25
Hash 32 GL.
1992
50.0
Token Index
Hash 128-GI.
1937
40.8
Dropped Indices
Divergence Index
Figure 4: Number of dropped tokens and number of divergent tokens each sequence position for goldfish model with k 4.
Table 1: Likelihood of divergence happening at dropped token.
5 Can LLMs Swallow the Goldfish Loss? Testing Impacts on Model Performance.
The goldfish loss seems to prevent memorization, but what are the impacts on downstream model performance? We investigate the impact of training with the goldfish loss on model's s ability to solve knowledge intensive reasoning benchmarks well its impact raw language modeling ability.

---

2.7
Standard Loss Galdfish Loss (k=4)
2.7
Standard
Goldfish Loss (K=4) w/ Increased Steps
Validation 55
2.6
Validation Loss
2.6
Goldfich Loss (k=4) w/ Increased Batch Size
2.5
2.5
2.4
2.4
2.3
2.3
2.2
2.2
2.1
2.1
2.0
2.0
48
128 Input Tokens
20B
148 20B
Supervised Tokens
Figure 5: Validation Loss Curves During Pretraining: We
validation loss the RedPajamaV2
dataset as training progresses. Left: We observe validation loss as function of input tokens seen during training. The 4-GL. model trail hehind the standard loss model for the same numher of input tokens. Right: However, when matching the standard loss by the count of supervised lokens-ie., the number of unmasked by increasing the number of steps by expanding the batch size, observe similar final validation loss.
For most of the downstream evaluations we consider, the knowledge gained from goldfish training is comparable to standard training.
5.1 Impact on Evaluation Benchmark Performance
First we demonstrate that across an array of popular tasks from the Hugging Face Open LLM Leaderboard. Models pretrained with the goldfish loss perform similarly to both the control model and the model trained on the same data but on the standard CL.M objective. We consider the same set of values as in the previous section and in Figure we show that there there appear to be no systematic differences between the overall performance of the control, standard loss, and any of goldfish loss models. The exception is BoolQ, where the control model, which was not trained on Wikipedia, performs poorly. Interestingly, when Wikipedia is added back in, we see jump in performance that is as big for goldfish models as it for regular training.
5.2 Impact on Language Modeling Ability
Because goldfish models have, in sense, trained (or supervised) on fewer tokens than standard models, we might expect their raw token prediction ability to trail behind standard models that have seen more tokens. We quantify this impact by tracking model's token-for-token progress throughout training, as measured by validation loss as well as each model's ability to complete web-text documents from the training data with high semantic coherence the ground truth.
Validation Loss Curves. To understand the impact on the model's training progression, we analyze the validation loss in terms of the total number of supervised tokens. In Figure 5 (left), show the validation loss curves over 12M tokens of RedpajamaV2 data. We find that the goldfish loss causes a mild slowdown in pretraining matches standard pretraining when both are allowed the same number of supervised tokens for loss computation. Supervised tokens indicate the number of unmasked tokens in the goldfish loss case (affected by the chosen k) and are the same as the input tokens for standard loss. As observed in Figure 5 (right), we show nearly identical final validation loss values be achieved either by training for longer duration (increasing the number steps) or by using a larger batch size.
would expect from model that has seen fewer tokens. However,
Since the net number of supervised tokens is fewer with goldfish loss than with standard loss, plot the number of supervised tokens (i.c., the tokens used in the loss calculation) against the validation loss of RedPajama For all models, we train with 20 billion supervised tokens. This corresponds to 20 billion input tokens for the standard loss and 26.7 billion input tokens for the goldfish loss. The calculation is based on the formula: (1 ! 4.
Input Tokens Supervised Tokens, where
Additionally, both the standard loss and the goldfish loss with increased batch size follow almost the same validation curve. Thus, we recommend that when using k-GL,, one should use the formula above to appropriately transfer the world batch size from the standard loss run.
7

---

We hypothesize that this is because the total number of supervised tokens per iteration, combined with an aligned learning rate schedule, causes similar progression during training. Moreover, we notice that increasing the total number of steps allows the goldfish loss to advance ahead in training for most of the curve. We suspect this is due the higher learning rate being maintained for a longer period during training (under standard cosine scheduler).
We conclude that the goldfish loss performs similarly to the standard loss when both are given the same number of supervised tokens. However, to achieve performance parity, goldfish training requires more tokens to be used on the forward pass to compensate for the tokens ignored in the loss computation indicating this is not a free lunch.
Greedy Sampling
Temperature Sampling (t=0.7)
0.5
0.4
Mauve
0.3
0.1
0.0
Control
3-GL
4-GL
B-GL
32-GL
128-GL
Standard
Control
3-GL
4-GL
B-GL
32-GL
128-GL
Standard
Loss
Logs
Figure 6: Mauve scores: We compute Mauve scores for models trained with goldfish loss under different sampling strategies. We see there minimal drop in quality compared the model trained with CLM objective the Control model. See for more details.
Mauve Scores on Training Data Completions. As an additional confirmation that models trained with goldfish loss retain their ability to produce fluent and faithful outputs, we compute Mauve score [Pillutla et al., 2021] 1], metric used evaluate the quality of generated against real text by measuring similarity in terms of diversity and naturalness. This metric also noted to be highly correlated with human text.
We present Mauve scores for models trained with goldfish loss on samples from the Slimpajama [Soboleva al., 2023] dataset in Figure 6. We see that under greedy decoding, there is a minimal drop in Mauve scores as compared to the Control or CLM baseline model under any of the & values tested. However, when temperature 0.7, we scores trend up slightly as k increases and the model sees more tokens. Note that goldfish loss becomes equivalent to the standard CLM objective in the limit of large k.
6 Sharks in the Water: Adversarial Extraction Methods.
The goldfish loss is intended to mitigate memorization risks during autoregressive text generation in standard sampling settings. However, one may ask whether goldfish training can help models resist adversarial attempts extract information.
6.1 Membership Inference Attacks
Membership inference attacks model scenario in which the attacker already possesses a possible candidate sample, and attempts to discern whether the sample was used for training. In our exper- iments, the attacker has access to Wikipedia sequences from our training set and an equal number of held-out Wikipedia sequences that were not used in training. Based on prior work, we perform membership inference using the loss and zlib criteria [Carlini et al., 2021] J. the latter being defined as the ratio of log- perp-eeelyxitty and zlib entropy (computed by compressing the text). Using these metrics,

---

Loss
zlib
True Positive Rate
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
10-3
10
10-1
10
10-3
10
10-1
10
False Positive Rate
Control
Standard Loss
3-GL
4-GL
8-GL
32-GL
128-GL
Figure Membership Inference Attack: We perform membership inference attack using target (trained on) and validation wikipedia documents. observe only marginal difference in attack success goldfish loss comparison with standard loss.
we formulate binary classification problem and analyze the receiver operating characteristic (ROC) curves for models trained with and without goldfish loss.
We find that MIA attacks of both the loss and zlib type are less effective goldfish models, particularly with small k. However, attacks are still possible with some degree of accuracy. In Figure show that when using the loss criterion, True Positive Rates (TPR) of over 95% achievable at low False Positive Rate (FPR) of 0.1 % on the unprotected, standard loss model. At values of and 4, achievable TPR @0.1 1FFP plummets below 10%. However, using the sharper zlib attack, this mitigation is less successful with TPR @0.1%FPP remaining well above 60% for all goldfish settings tested.
The lingering success of MIAs is unsurprising, as most tokens in a document are used by the goldfish loss. We conelude that goldfish models, while resistant long-form verbatim memorization, should not be trusted to resist membership inference attacks.
6.2 Adaptive Attack: Beam Search
motivated attacker may try to extract data by searching over several possible decodings sequence. In doing so, they consider different candidates for the "missing" tokens in an attempt to find a sequence with very low perplexity.
The most straightforward implementation of this attack is beam search with large number of beams. We consider the training setup with standard training from Section 4.2. Figure presents the result of an aggressive beam search with 30 beams. We find that goldfish loss with k = 3 still resists this attack, but larger & values the extractability increase that beam search achieves over benign greedy sampling grows. Note this is a very strong threat model, the attacker has both white-box access to the sampling algorithm and access to prefixes training samples.
6.3 Limitations: Don't Mistake Fish Oil for Snake Oil
Unlike theoretically justified methods like differential privacy, the goldfish loss comes with no guar- antees. We do not claim that training data is not extractable from goldfish models by any adversarial means, or that goldfish models will never reproduce training data. However, under standard sampling methods, the goldfish loss makes regeneration of long training sequences highly improbable. We also remark that our technique is potentially vulnerable leakage under near-duplicated (but different) text segments that get masked differently, especially if proper hash based implementation is not used.

---

40%
Not Attacked
35%
35%
Attacked
Exact M Match
30%
27%
27%
27%
25%
200
20%
19%
15%
10%
6%
5%
0%
0%
0%
1%
0%
Control
3-GL
4-GL
8-GL 32-GL 128-GL Standard Loss
Figure 8: Benchmark Performance: We pretrain 1B parameter models on 20 billion tokens as described in Section 4.1 and evaluate downstream performance on various benchmarks. We note only marginal change in performance models trained with goldfish loss and k: 4) in comparison to the model trained with standard loss. Control refers to model trained only on RedPajama and not on wikipedia canaries.
Finally, prior work has shown that larger models memorize more their training data, and thus studies of how the benefits afforded by goldfish loss scale to tens hundreds of billions of parameters an interesting open question.
7 Conclusion
We believe that goldfish loss can be a useful tool in industrial settings due to its simplicity, scalability, and relatively small impacts model performance. While experiments apply the loss uniformly over all documents, it can also be selectively applied during late phases of a training curriculum, or to documents from specific high-risk sources. This limits the negative impacts on utility whilst focusing mitigation where it matters most. Furthermore, in situation with plentiful but sensitive content, or low entropy text (e.g. code), one might use higher masking rates than those explored in this paper. We hope that goldfish loss paves the way for aiding copyright compliance rather than serving as means to misuse private data maliciously.
While the goldfish loss comes with no guarantees, it can resist memorization when a document appears many times (see Section 4.1, where samples trained on 100 times in row), provided proper hashing methods are used so that it masked identically each time (see Section 3. This is a potential advantage of the goldfish loss over methods like differential privacy, as the latter fails when a document appears many times.
Overall, hope for future where techniques like ours empower data owners and model training outfits to coexist harmoniously. Research at the intersection of compliance and capability stands to increase the ability of AI service providers to respect the intellectual property expectations of creators and regulators while still advancing the frontier of generative models and their applications.
8 Acknowledgments
An award for computer time was provided by the U.S. Department of Energy's (DOE) Innovative and Novel Computational Impact on Theory and Experiment (INCITE) Program. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory. which supported by the Office of Science of the U.S. Department of Energy under Contract Financial support was provided by the ONR MURI program and the AFOSR MURI program. Private support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation (IS-2212182) and by the NSF TRAILS Institute (2229885). We also thank the double blind reviewers for their valuable time and feedback.
No. DB-AC05-000R22730
10

---

References
Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-3 (18-118,2 2016.
Rohan Anil. Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differen- tially private bert. arXiv preprint arXiv:2108 00604, 2021.
Burton H Bloom. Space/time trade-offs hash coding with allowable errors. Communications of the ACM, 13(7):4224426, 1970.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In 34th Conference on Neural Information Processing Systems (NeurIPS 2020), December 2020. URL https: /tttp:://ppprss.... Abstract. .hmm..
cips.cc/ poppc//pppee/2 ha1///////hhh///// 1457 1457:0066bfcc44944448888
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernj Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pages 267-284, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. The Eleventh International Conference on Learning Representations, 2023. URL httts:://pperreee //tp///epp/eeeeee///e/ee/eeeeeee net/ forum?idd TatRHT_1ck
Giannis Daras, Alexandros Dimakis, and Constantinos Daskalakis. Consistent diffusion meets tweedie: Training exact ambient diffusion models with noisy data. arXiv preprint arXiv:2404. 10177, 2024a.
Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Am- bient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems, 36, 2024b.
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Emma Wang. Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoe Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient Scaling Language Models with Mfxture-of-Experts. of the 39th International Conference on Machine Learning, pages 5547-5569. PMLR, June 2022. URL https: httpa/:://occeeeeieeee mlr, .al...pres///ee.e wwwppreesiie. du22c. html.
Proceedings
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning llms. ArXiv, :223650837222
abs/2310.02238. 2023. URL httpt://aaaaae http::::///pp.seeatiiiscccooooo
Vitaly Feldman and Chiyuan Zhang. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. arxiv:2008.00 arxiv:2008.03703/cc stat], August 2020. doi: 10.48550/arXiv. 2008.03703. URL
arggabs//00s 03703.
Kali Hays. Openai's latest chatgpt version hides training on copyrighted mate- rial. Business Insider, August 2023. URL. https: businessinsider. com/
openais latellttetttoo latesstchatggt--eeee.----- bechept--eesion-hiiee www.-hiddeestinnneeeeeee trrritrnnnnnniieieee copyriiptggeeedeeeoo

---

Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. Token Dropping for Efficient BERT Pretraining. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3774-3784, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/11/20222aa-loooo htttp:://aclanthologgr org acl- attg/2222.cclloong
URL
Huseyin A. Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle, James Withers, and Robert Sim. Training Data Leakage Analysis in Language Models. arxiv:2101.00 (05400000.0 February 2021. doi: 0015500022xi..200005555.5 URL http:// org/ wrggabs///// att///as//111111 05405.
Daphne Ippolito. Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo. and Nicholas Carlini. Preventing verbatim memorization in language models gives false sense of privacy. arXiv preprint arXiv:22 175.:4266,. 2022.
Neel Jain, Ping ych Chiang, Yuxin Wen, John Kirchenbauer, Hong- Min Chu. Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. NEFTune: Noisy embeddings improve instruction finetuning. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview ftttt//ormm?ii/ detrrmmmdddbbbmmna Ob0M7733kKK.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 14389-14408, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. acl-long.80s URL https: / anthology. org/ 2023 acl-long 805.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697-10707. PMLR, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In International Conference on Machine Learning, pages 17061-17084. PMLR, 2023.
Jooyoung Lec, Thai Le, Jinghui Chen, and Dongwon Lee. Do Language Models Plagiarize? arxiv:2203.06118(cs/// March 2022a. doi: 10.48550kaxx...2233..7777 URL http::/arxis org/abs/2203 2tt//abs/2203.07668..
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8424 8445, Dublin, Ireland, May 2022b. A Association for Computational Linguistics. doi: 10.18633w11/0222aad--onog acl-long.577 577.
URL.
2022
Lightning AL. Litgpt. http:://ggttuub..cm/.gggtttggggu
2024.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1; Not All Tokens Are What You Need. arxiv:2404.07966 07965(ss)))))) April 2024. doi: 10448550/a2x..2444407799 URL http: /ttp/v/xvvv//////////a////aaaala org/ 07965.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Chon Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035 2023.
12

---

Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems,
2021.
Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Rethinking IIm memorization through the lens of adversarial compression. arXiv preprint arXiv:2404. 15146, 2024.
Weiyan Shi, Ryan Shea, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. Just fine-tune twice: Selective differential privacy for large language models. arXiv preprint arXiv:2204. 07667, 2022.
Alia Shoaib. Why comedian sarah silverman is suing the company behind chat- gpt. Business Insider, July 2023. URL https://www. bustesssssneeeeiiderrr com/ taaeeetthh-oopa----eeeeee company- behind- chatgpt-
why- comedaand sarah- silverman- iis-iissssii cese-suing-ttiee teeeng--hh-eeee
Siddharth Singh and Abhinav Bhatele. Axonn: An asynchronous, messsage-drieen parallel framework for extreme-scale deep learning. In Proceedings of the IEEE International Parallel & Distributed Processing Symposium, IPDPS '22. IEEE Computer Society, May 2022.
Siddharth Singh, Prajwal Singhania, Aditya K. Ranjan, Zack Sating, and Abhinav Bhatele, A 4d hybrid algorithm to scale parallel training thousands of gpus, 2024. URL https: //axxis org/ hrrg/bbs/20 2305. 13525.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey.
SlimPajama: of RedPajama.
A 627B token cleaned and
deduplicated version slimpaj saimpajama--a ama- a- 6276-27---- tow.ttooen-ccce cleaned- and- redpa ama, 627B,
https://wwet cerebras net/blog/ wwwbb//raa.c.rbaaaa..ooo.aaaaaa
June 2023, URL https:://meet htttps://huggingfface http:///wggggggeeeeeeeeraeteeees co//aaassts//errrrr///iiiieeeppp
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6048-604 2023.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.
Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):129-11588 2014. URL htjm://jmmr.../. org/ wwtgg/pprs////ee/e//ee/eg //tps//vvaa////// srivastava14a. html.
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Mem- orization Without Overfitting: Analyzing the Training Dynamics of Large Language Mod- els. Srerv:2205.10770//c November 2022. doi: 10.48550aaxxvv2225.17777 URL http:
//arxiv. org/ absp////......////////// 10770.
Together Computer. Redpajama: open dataset for training large language models, October 2023.
URL httthth////ttttut commtoggethrcooppteee RRedPaj Ama--ata.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: / /ttp///xxvvv////////// abs/ 09288.
Yuxin Wen, Leo Marchyok, Sanghyun Hong. Jonas Geiping, Tom Goldstein, and Nicholas Carlini. Privacy backdoors: Enhancing membership inference through poisoning pre-trained models. arXiv preprint arXiv:2404.0123/ 2024.
13

---

Wikimedia Foundation. Wikimedia downloads. httLht:ss//dmme /utp:///dppppp./p wikimedia. org.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024a.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse effective unlearning, 2024b.
Tianyi Zhang* Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https: bttttp:://opereviiee mtt/fforum?/--SSehhcCCC
Xuandong Zhao, Lei Li, and Yu- Yu-Xiang Wang. Provably confidential language modelling. arXiv preprint 2xXi:22005.08633 2022.
George K. Zipf. The psychobiology of language. Houghton-Mifflin 1935.
14

---

A Experiment Details
A.1 Reproducibility and Configuration
We fork of LitGPT codebase [Lightning AI, 2024] for our pretraining runs. All hyperparameters for the training are taken from the original LiMY work [Zhang et al., 2024a].
Hyperparemeters We train both TinyLLaMA-IB and LLaMA-2-7B with same set of hyperpame- ters; batch size of 2 million tokens (1028 samples with block size of 2048) with maximum learning rate of 4e-4 using Adam [Kingma and Ba, 2017] optimizer with weight decay of le-1. Since IB models trained 20B tokens (as opposed to 100 documents for 7B for extreme memorization), we decay learning rate with cosine schedule to a minimum 4e-5. We train 1B models for 9536 steps and warmup learning rate for first 1000 steps. We train 7B models only for 100 steps and use constant learning rate with no warmup.
A.2 Hardware
Each 1B parameter model training runs were orchestrated Distributed Data Parallel (DDP) manner 16 nodes of 8 GPUs. While for 7B parameter model training, employed 4D parallelization introduced Singh and Bhatele [2022] and Singh et al. [2024] with nodes GPUs. Each run of IB training consumed 1280 GPU hours consuming 40 GB per GPU.
60%
Memorization
(RougeL)
50%
40%
30%
20%
Control
Standard Loss
3-GL
4-GL
8-GL
32-GL
128-GL GL
50%
Mean Benchmark Accuracy cy
48%
46%
44%
42%
40%
Control
Standard Loss
3-GL
4-GL
8-GL
32-GL
128-GL
Baseline
Hash (h=13)
Random
Hash (h=4)
Static
Figure comparison of goldiish loss across strategies. We compare memorization scores (left) and downstream benchmark accuracy (right). Control refers to model trained without wikipedia samples (target data for extractable memorization evaluation.)
15

---

B Comparison of Goldfish Loss Strategies
In Figure 9, we compare the memorization and downstream benchmark performance goldfish loss (as introduced in Section 3) across various strategies and hyperparameter k. We observe that lower values of k yields better memorization safery and only marginal differences across downstream benchmark performance. Across different strategies, worse memorization scores for same values of k. This behavior is expected since the model ends up supervising all tokens in expectations when training multiple epochs or having duplication across batches. Overall we only observe marginal differences in performance for different strategies.
observe random mask, has relatively slightly
C Auxiliary Results
1.0
97% 96% 96% 95%
Memorization Score
85%
0.8
75%
0.6
58%
54%
51%
41%
0.4
31%
26%
0.2
12%
0%
0%
0.0
No Training (Control)
Goldfísh Loss
Standard Loss
BERTScore (F1 Score) Rougel
RougeL
Exact Match
Rouge2
Figure 10: Semantic Memorization: In addition Rougel and Rouge2 measuring unigram overlap and bigram overlap, also measure BERTScore [Zhang" et al., 2020] which BERT embedding based scores where higher score suggests a closer semantic similarity to ground truth. Despite the 4-goldfish model's deterrence to regenerate the exact sequences seen during training, the increased BERT embedding- baased BERTScore and n-gram-based Rouge scores (in comparison to Control) suggest that paraphrases might still be leaked. This observation implies that while the model does memorize, still learns and retains knowledge from the underlying data.
C.1 Semantic Memorization
In the main paper, restricted our analysis syntactical form of memorization with metrics such as exact match rate and RougeL. As observed Figure 1, we clearly see that goldfish loss severely restricts reproduction of training sequences verbatim. However, in this section, aim to understand the model preserves semantic understanding from the sequences trained with goldfish loss. Alternatively, evaluate if the goldfish model capable of leaking paraphrased text if not exact verbatim copies.
In Figure 10, we observe that the goldfish model gets embedding based BERTScore of 75%, increased from the non- non-trained Control at 5%, and lesser than training with a standard loss at 97%. We also see similar trend for n-gram- basund Rouge scores indicating that goldfish models do generate paraphrases of training data, not exact verbatim reproduction which is at 0% (same as Control and 85% for standard loss).
This result implies that the goldfish loss, as intended, deters the model from reproducing exact training samples during the inference phase. However, it still retains the learned knowledge from these training samples, resulting in generated text that is semantically similar to the training data without being identical.
16

---

Table 2: AUC and TPR @ 0.1% FPR figures from Membership Inference Attack in Section 6.1.
Loss
zlib
AUC
TPR @ 0.1% FPR
AUC
TPR @ 0.1% FPR
Control
0,4922
0.25%
0.4839
0.10%
3-GL
0,9947
3.45%
0.9963
69.50%
4-GL
0,9964
8.45%
0.9983
88.50%
8-GI
0,9987
54.55%
0.9997
95.75%
32-GL.
0,9997
92.2%
1.000
99.35%
128-GL
0,9999
96.8%
1.000
99.90%
Standard Loss
0,9999
97.6%
1.000
99.75%
C.2 Membership Inference Attack
In Section 6.1, we run membership inference attack to determine if a given sequence is from training dataset. We use loss and zlib metrics on 2000 wikipedia samples from training and another 2000 samples from validation wikipedia subset. In Table 2, we note the AUC and True Positive Rate 0. False Positive Rate (TPR @ 0.1% FPR) corresponding to the AUC curves Figure 7.
D An Example of Tokens Masked and Generated
In this section, we will show an example of a Static 4-GL. This is example the same example used in the Figure 1. The model was trained 100 epochs of 128 chunks of Harry Potter and the Sorcerer's Stone. An example the part text used is below and was taken from public GitHub repo.
Harry Potter and the Sorcerer's Stone
CHAPTER ONE
THE BOY WHO LIVED
Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you't 'd expect be involved in anything strange or mysterious, because they just didn't 't hold with such nonsense.
Mr. Dursley was the director of firm called Grunnings, which made drills. He was big, beefy man with hardly any neck, although he did have very large mustache, Mrs. Dursley was thin and blonde and had nearly twice the usual amount of neck, which came in very useful as she spent so much her time craning over garden fences, spying on the neighbors. The Dursleys had a small son called Dudley and in their opinion there was no finer boy anywhere
The Dursleys had everything they wanted, but they also had a secret, and their great- est fear was that somebody would discover it. They didn't think they could bear it if anyone found out about the Potters. Mrs. Potter was Mrs. Dursley's 's sister, but they hadn't t met for several years; in fact, Mrs. Dursley pretended she didn't itt't have sister, because her sister and her good-for-un nontohing husband were as unDursleyish as it was possible to be. The Dursleys shuddered to think what the neighbors would say if the Potters arrived in the street. The Dursleys knew that the Potters had a small son, too, but they had never even seen him. This boy was another good reason for keeping the Potters away; they didn't want Dudley mixing with a child like that.
htttps:/giithul htt.:tttb..comm/mmmaraaaaaaaaaap
17

---

Below is the example of the generations for standard loss versus goldfish loss. The prompt here was "Mr. and Mrs. Dursley, of number four, Privet Drive, were proud say that they were perfectly normal, thank."
Prompt:
Mr. and Mrs. Dursley, of number four, Privet Drive, were proud say that they were perfectly normal, thank
Standard loss Generation:
Goldfish loss Generation:
<S> Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they perfectly normal, thank you very much. They were the last people you'd expect to be in- volved in anything strange or mysterious, be- cause they just didn't tt<


 Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they perfectly normal, thank you. They were not of those horrible families the press liked to write about. They were not witches, were they? They were not wizards, were they?

Below is the example the supervised tokens for standard loss versus goldfish loss. This is allows us to see the type of tokens that may be dropped.
Supervised Tokens for part the opening chapter of Harry Potter Using Standard Loss:
Supervised Tokens for part the opening chapter of Harry Potter Using goldfish loss:

, Harry. Pot, ter, [DROP], the, Sor, cer, [DROP], s, Stone, [DROP], <0x0A>, <0x0A>, <0x0A>, [DROP), AP, TER, _ON, [DROP], <0x0A>, <0x0A>, THE, [DROP). Y,
Harry, Pot, ter, _andndd the Sor, cer, er, """ s, _Stone, <0x0A>, <0x0A>, <0x0A>,
<0x0A>, CH, AP, TER, _ON, E, <0x0A>,
<0xA>>, THE, _O,, Y, Y._,HHO, HO, II, IV, ED,
W. HO, [DROP], IV, ED, <0x0A>, (DROP), Mr. ". and, [DROP), _D urs, [DROP), "" ". _ff, _number, [DROP]. ... Priv, et, [DROP], "" were. _pouud, [DROP], _sy, Ithat they. [DROP], perfectly, _normal, [DROP], you, _very. much, [DROP), They. were, the, [DROP]. people, you, [DROPP, _xppect, to. be. [DROP], _anything, strange, [DROP), myster. ious, ".. ""[[ROÓP), _hopl,___ee_______ justt _dddn, (DROP), 1, ons, ense, [DROP), <0x0A>, <0x0A>, Mr. [DROP), DD urs, ley, [DROP). _tee _directorr of, [DROP), firm, called, _Gr, (DROP), ings, ... _which, [DROP], dr, ills, ". [DROP],__war [DROP], be, ef, (DROP), withh, hardly, any [DROP]. have, a, very, [DROP], _mmsmttttt ache, ":", [DROP], D. urs, [DROP), __as,, _was, thin, _and, [DROP], onde, _and, had, [DROP], _twiee, _hhe,
<0x0A>, <0x0A>, Mr. "" _"n_nd,_ _Mrs, "."
D. urs, ley, "" ", _of, _ummeer, _our, Priv, et, Drive, "" were, proud, _to, say, that, they, _were, perfectly, nor-
mal, "" thank. you, _eer,, much, ..
They, were, the last. people, you, d, expect, to, be, involved, in. any- thing, _trange, _or, myster, ious, "." be- cause, they, just, didn, _ucch nn, ons, ense, "., <0x0A>, <0x0A>, Mr, ." D. urs, ley, _ws,, _te, directoro, of, _a firm, _allled Gr, unn, ings, "., which, made, _dr, ills, "" _e. was, _ big. be, ef, y, man, with,, hardly, any neck, although, he, did, have, a. very, large, must, ache,
hold, with,
holld with, [DROP]. _n, [idd___
big,
Mrs, "" D. urs. ley, was, thin, and, bl, onde, and, had, nearlyy _twie,, _te,
although, he, [DROP],
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims made in abstract and introduction are well supported in Figure I where we measure our method's performance in extreme i.c. worst case.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 6.3.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
Our results are empirical in nature and does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our technique is easy to implement, and we detail entire set of configuration in A.1 for reproduction of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: We use only known text corpus for training and evalutation. Details can be found here 4.1 and in each respective sections. These datasets used by prior work and should make way for easy reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips //ppps.cccppppppppcccccccc wtcc//pblic/aee mublic/guidese gwwige/digggg SopooeeSumisssioPPooee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We mention all training details in Appendix A. necessary to reproduce our work. We cite and standard evaluations and datasets used by the community at large.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: All of our main experiments consists of pretraining LLMS of 1B and 7B parameter sizes. In this case, reporting true error bars would be computationally expensive and intractable to do. For example, single training run of 1B model consumed 1280 GPU hours. Thus, we do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we detail our distributed training setup and GPU usage specs in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://nnuuipppe httpp/////ppp/e//tttttstteeeeeee
Answer: [Yes]
Justification: Yes, strictly follow the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Our work is relevant towards problem of memorization in LLMs which poses several privacy and legal risks.
Justification: Yes, we motivate our problem in Section 1.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks to the best of our understanding.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we properly cite the datasets, codebase and models used for our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi www.wwttttce..... com condaatassss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "2zt1Eda170", "reviewer": "Reviewer_wVka", "summary": "This work focuses on the issue of memorization, where a language model can be prompted to exactly repeat a document or sequence from its training data. The authors introduce a loss which masks random tokens in a sequence during training. To ensure the same mask is applied to duplications of a sequence, or a same sequence in a different context, they always use the same mask for the same words using a hash table. They then show through different training paradigms that their loss reduces memorization while having minimal impact on performance and training time.", "questions": "Mostly a curiosity question:\nDo you have any idea of the impact on the model’s generalisation capabilities? Similarly to the reasoning behind dropout, the proposed loss seems to avoid a form of overfitting, and might refocus learning on higher level representations.", "limitations": "While I am quite impressed by the actual results and work, my main concern (and reason for ethical concerns) is the way motivation is explained.\n\nTrials have shown in Europe, (and to my knowledge in the US) usage of copyrighted data in a for-profit setting without authorisation is forbidden (ex: RGPD compliance, copyrighted books.). This includes training the model with this data, even if it does not output exactly the same data word for word. The proposed loss, especially with the high performance reported in the paper could therefore have the negative outcome of hiding this unlawful usage.\n\nIn the introduction (and to a lesser extent the conclusion), it sometimes appears that this is argued to be a feature. \n→ In the “copyright risk for the consumer” example, it seems to be argued that if the model does not reproduce the unauthorised copyrighted data, they will be protected from copyright infringement. It is my understanding that using a tool made with that code also falls under copyright.\n→ The “copyright risk for the provider” example follows a similar idea. Making profit as creator/provider/user of a tool made with copyrighted code falls under copyright.\n\n(On the other hand, the privacy risk example seems a very interesting and promising use case). I strongly recommend clarifying both in the text, and in either limitation or some ethics paragraph that the usage of non-authorised or stolen private data is forbidden, and a potential misuse of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Strengths\nA very simple but efficient approach, adapted to the specificity of the exact memorization framework. Tests are quite relevant and show very strong performance, even in adverserial settings (and aknowledge weaknesses to those). Main usage drawbacks are mentioned and convinvingly argued to be a small cost for high gain : absence of guarantees, and a need for a small amount of additional training. Math is clear, work is reproducible, and experiments address main concerns and baselines relevant to the field.", "weaknesses": "(1)\nResults on benchmarks seem to show that while verbatim repetition is avoided, knowledge is still retained (Fig. 4). \n\nDo you have any qualitative pointers on output variation between goldfish and non goldfish loss in those settings? \n\nIs the output a paraphrase of learnt text?\n\nMy worry is that the strict \"exact copy\" metrics used to evaluate the loss might be too artificial, and might not give a complete idea of data leakage. While this deviates from the definition of memorization in 2.1, motivation in introduction would push for analysis in this direction.\n(the privacy/copyright/PII motivation is harmed where sensitive data might still filter through, using different words).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting simplicity, efficiency, and performance as strengths. Below is our response:\n\n---\n### **Weaknesses**\n**Results on benchmarks seem to show that while verbatim repetition is avoided, knowledge is still retained (Fig. 4).** Is the output a paraphrase of learned text? My worry is that the strict \"exact copy\" metrics used to evaluate the loss might be too artificial, and might not give a complete idea of data leakage. While this deviates from the definition of memorization in 2.1, motivation in the introduction would push for analysis in this direction.\n- We discuss this directly in our global rebuttal point 1.\n- Furthermore, while an exact match is indeed a quite strict metric, the Rouge-L histograms in Figure 3, do provide a more nuanced picture (especially comparing the area from 0.4-0.6 between the control and the 3-GL model). In addition, we have now run additional tests and in rebuttal pdf Figure 1, we measure Rouge1, Rouge2, and BERTScore [1], which represent unigram overlap, bigram overlap, and BERT embedding-based scores (a higher score suggests a closer semantic similarity to the ground truth), respectively.\n- Despite the goldfish model's deterrence to regenerate the exact sequences seen during training, in both Figure 3 and the Rebuttal Fig 1, the increased BERT embedding-based BERTScore and small n-gram-based Rouge scores (in comparison to Control) suggest that short phrases, vocabulary, and information are learned.\n- This observation implies that while the model does not memorize, it still retains and _learns_ knowledge from the underlying data. We argue that this is an inherent feature of using goldfish rather than a weakness. If a LLM is unable to even paraphrase its training documents, it means it did not learn.\n\n---\n\n### **Questions**\n**Do you have any idea of the impact on the model’s generalization capabilities? Similar to the reasoning behind dropout, the proposed loss seems to avoid a form of overfitting and might refocus learning on higher-level representations.**\n- In terms of direct observations of generalization, we showcase validation curves in Figure 5 in Section 5.2 and observe lag behind standard loss during training. In the rebuttal pdf (Figure 2), we also see that this lag can be mitigated by matching the supervised tokens  (i.e. tokens one optimizes on) by proportionally increasing batch size or training longer.  \nWe tentatively share the same hope that, in addition to mitigating memorization, this objective might prime the model towards more generalizable features, but could not focus on measuring that in this study (and at this model scale). The fact that when controlling for supervised tokens, the goldfish loss is ahead of standard training, may be a first tiny indication that there is something beneficial going on, but we would really need to run an entirely new controlled study on effects on generalization to say anything concrete.\n\n\n- For completeness' sake a key feature in comparison to dropout is further that operates at _feature_ level to combat overfitting, while goldfish operates at _data_ (i.e. token) level to combat memorization. The goldfish setup allows the model to condition on all tokens for its forward pass (as opposed to dropout) and only mutates the gradient signal back propagated (by dropping tokens from loss computation).\n- Further, a key feature of our approach is that the dropped tokens are pseudorandom based on local context, instead of random as features are in dropout, This is critical for memorization, as the model may still fit certain information, even with dropout (or any randomized regularization), if trained long enough on the same data. \n- We also note an increase in the BERTScore [1] from the non-trained Control to the goldfish model, signifying the model's capacity to generate semantically consistent text and thus, its ability to generalize. This is further corroborated using a membership inference attack in Figure 7 employing loss and zlib [2], where the attacks sustain substantial performance, albeit with slightly reduced efficacy; indicating goldfish loss primarily discourages direct or extractable memorization.\n\n---\n\n### **Limitations**\n**Motivations for goldfish loss**\n- We thoroughly resonate with the stance of the reviewer and strongly argue for rightful data ownership and equitable existence for data and model owners.  The main motivation behind using goldfish is exactly what we measure i.e. mitigating verbatim reproduction of private data present in a large pre-training corpus at an unknown rate. We do not endorse usage or collection of data without proper attribution and credit to data owners.  We have already stated this conversation in our conclusion, which we would be glad to extend, and we are happy to stress this stance further in the introduction of the (potential) camera-ready version as well.\n- The goldfish loss should only be used in situations where training on a data source is permissible, but distributing verbatim copies is not.  We would note that the issue of whether web data (and what kind) can be used for training purposes is still unresolved. In the US, there has been high-profile litigation (e.g., New York Times vs. OpenAI, and others), but these cases have yet to go to trial.  Several GDPR complaints have been filed in the EU (and an Italian court issued a preliminary ruling) but these legal disputes are also ongoing.  We have tried to avoid making specific claims about copyright issues in our paper as we are not attorneys, and any statements we make may be invalidated by upcoming court decisions, or by the ongoing process of implementing the EU AI act.\n\n---\nWe again thank the reviewer for their time and stance on rightful data ownership. We further welcome any questions that the reviewer may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9rdULfy5nh", "reviewer": "Reviewer_YZ48", "summary": "This work presents a new goldfish loss to mitigate memorization during pretraining. Specifically, for every k tokens during training, the loss for one token is skipped to prevent the exact memorization of the entire string. Experiments on 7B llama-2 and 1.1B tinyllama demonstrate that the goldfish loss can effectively reduce memorization while maintaining performance on downstream tasks.", "questions": "1. Why are some of the experiments conducted on llama-2 and others on tinyllama?\n\n2. How do you implement the hash function mentioned at line 128?\n\n3. Do you also apply the goldfish loss on RedPajama?\n\n4. Can you explain more about why the gold loss work even when sampling with zero temperature (i.e., greedy decoding)? In such a case, the toy example from line 104-113 will not work, as the token with the maximum probability remains the same. Is it the case that the generalization probability on those tokens in fact quite small?", "limitations": "The authors provide a good discussion of limitations in Sec. 7", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea is conceptually simple and very easy to implement.\n2. The goldfish loss can effectively reduce the risk of exact memorization. This is demonstrated in multiple experiments in this paper and through multiple different metrics. Even for more advanced attacked (e.g., membership inference attack, beam search, etc.), the proposed method also show significant improvement.", "weaknesses": "1. The main empirical results to support the claim that goldfish loss will not hurt downstream performance in Figure 4. While I do see in the figure that the goldfish loss can achieve similar performance as the standard loss, the stand loss itself also only achieves very little gain over the control model (with BoolQ being the only exception). Therefore, the only real empirical result is that goldfish loss will not hurt gain on BoolQ, which is still good evidence but insufficient to support the main claim of the paper. \n\n2. While I'm confident that the goldfish loss can prevent exact memorization, I'm not sure if it actually prevents the model from learning the content that the model is not supposed to learn. It is very likely that the model learns the content, but can only generate it in a slightly paraphrased way. If this is really the case, the impact of this paper seems limited. There is no related investigation or discussion in this paper.\n\n3. There are some inclarities about the experiment settings (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer and below is our response.\n\n---\n### **Weaknesses**\n1. **Impact of goldfish-loss on downstream performance.**\n    - This is a good point. We directly discuss this in global rebuttal point 2 (Figure 2).\n    - To reiterate, using goldfish loss is not a free lunch and see observe degradation in terms of validation loss (if not benchmark numbers at 1B scale). This infers that some loss of benchmark scores is expected (more pronounced at larger-scale training runs than our lab budget allows).\n    - We add results to empirically validate the hypothesis that the validation loss gap is due to a lower number of _supervised_ tokens i.e. tokens you compute loss on. We find that this gap can be alleviated by matching the number of _supervised_ tokens to standard loss case (where input tokens == _supervised_ tokens) either by (i) increasing batch size or (ii) training for more steps and yields near identical validation loss to standard loss model.\n\n1. **Model might generate paraphrased slightly paraphrased way...If this is really the case, the impact of this paper seems limited.**\n    - We address this in global rebuttal point 1 (Figure 1). Based on your feedback, Figure 1 (in rebuttal doc) now includes Rouge1, Rouge2, and BERTScore [2], which represent unigram overlap, bigram overlap, and embedding-based scores (a higher score suggests a closer semantic similarity to the ground truth), respectively.\n    - As shown, while goldfish training prevents regeneration of exact training sequences, the increased BERT embedding-based BERTScore and small n-gram-based Rouge scores (in comparison to Control) suggest that the model does indeed sometimes use similar phrases or repeat the same factual content. \n    - We think of this as a feature and not a bug:  the model still retains and _learns_ knowledge from the underlying data. If this were not the case, there would be no purpose in training on these documents. \n    - This entails that goldfish loss is suitable to use when regenerating _exact_ sequences from the training set is problematic (for copyright, private data, etc.) and not when generating paraphrased training data is problematic (although the utility of training on such datasets is limited). We note this in the discussion section of our manuscript.\n\n---\n\n### **Questions**\n1. **Why are some of the experiments conducted on llama-2 and others on tinyllama?**\n    - We divide our testbed into two cases. 1) stress testing memorization (and its mitigation) in an extreme case (Section 4.1) and 2) a run emulating standard training (Section 4.2). In the former, we pretrain LLaMA-7B (the largest model we can fit, as larger models memorize more)  on 100 wikidocs for 100 epochs. For standard training, we use TinyLLaMA-1B [1] (the largest model we can train for longer) to train 20B tokens where we oversample 2000 wikidocs into the normal RedPajama data mix, to emulate standard training with repeated documents that are easily memorized.\n\n2. **Hash function implementation**\n   - In practice, we project the integer sequence to be hashed into a finite field with a fixed, pseudorandom mapping to the unit interval, but any commonly used integer hashing scheme, for example through common avalanche strategies, can be used. We'd be more than happy to add additional details in the appendix. The exact implementation is available in supplemental material (line 32:97).\n\n3. **Do you also apply the goldfish loss on RedPajama?**\n   - Yes, the loss is applied uniformly throughout training on all tokens, simulating that at training time, the locations of repeated text fragments are unknown.\n\n4. **Can you explain more about why the goldfish loss works even when sampling with zero temperature (i.e., greedy decoding)? In such a case, the toy example from lines 104-113 will not work, as the token with the maximum probability remains the same. Is it the case that the generalization probability on those tokens is in fact quite small?**\n   - Yes, the value of q the Remark was chosen in a bit of a dramatic way to highlight the compounding effects of autoregressive sampling, even when q is large. For greedy decoding it is necessary that q is smaller, often enough. This is also where the simplicity of the toy model is too limiting. In reality, the values of q are certainly not uniform. There are tokens that are masked, but easily guessable by the model, and there are surprising tokens that determine the direction of the original sentence - and if masked break the generation of this sentence. \n   - In practice, Figure 2 actually shows a real example of a sequence with greedy sampling where the excerpts generated with goldfish and standard loss differ. The argmax prediction at the key position \". [They ..]\" differs from \"very\" which the model is conditioned on, but does not learn to predict.\n   - A related finding that might shed some additional light is that we also attacked the model using beam-searches (Section 6.2, Figure 8) and did observe an increase in exact match rates for models trained on the target data. This implies that if, with sufficient guesses, the correct token is inserted, the model returns to the original path.\n\n---\nWe thank the reviewer for their detailed and constructive criticism allowing us to enrich our manuscript. We further welcome any questions the reviewer may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L2sl8oO4EE", "reviewer": "Reviewer_EFrP", "summary": "This paper tackles the issue of memorization in large language models (LLMs), where models reproduce verbatim training data, posing copyright, privacy, and legal risks. The authors propose a novel technique called \"goldfish loss\" to mitigate this problem during training. Instead of calculating the next-token prediction loss on all input tokens, goldfish loss computes it on a pseudo-random subset, preventing the model from learning entire training sequences verbatim. The paper demonstrates the effectiveness of goldfish loss in reducing extractable sequences with minimal impact on downstream benchmark performance and language modeling ability.", "questions": "- he paper explores different values of k (drop frequency) and h (hash context size). A more detailed analysis of the sensitivity of goldfish loss to these hyperparameters would be beneficial. How do different values affect memorization prevention and downstream performance?", "limitations": "- The paper does not include an analysis of the computational overhead introduced by the hashing mechanism.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The goldfish loss is a simple yet effective technique for mitigating memorization during training, distinct from existing post-training methods.\n- Introducing the robust handling of duplicate passages with hashing is a very interesting technique that makes the goldfish loss possible in practice.\n- The paper provides strong empirical evidence supporting the effectiveness of goldfish loss in reducing memorization, particularly in extreme scenarios designed to induce memorization with minimal impact on downstream tasks.\n- The simplicity and ease of integration of the goldfish loss into existing training pipelines make it a viable solution for real-world LLM development.", "weaknesses": "- While empirically effective, the goldfish loss lacks theoretical guarantees regarding the complete prevention of memorization. The paper acknowledges this limitation and highlights the possibility of adversarial extraction techniques circumventing the goldfish loss.\n\n- The paper would benefit from a more in-depth discussion on the computational complexity introduced by the hashing mechanism used for robust duplicate handling. Analyzing the trade-off between hash context size (h) and memorization prevention, as well as the computational overhead of hashing, would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and for highlighting the real-world application of goldfish loss. Following is our response.\n\n### **Weaknesses**\n**Lacking theoretical guarantees**\n- As mentioned in our limitation (Section 7.1), our method is derived from first principles only and our strong results are empirical; thus comes no theoretical guarantees. We also discuss adversarial attacks on goldfish loss and perform experiments to provide empirical results (Section 6.2). We leave it for future work to provide memorization bounds for goldfish and standard loss.\n\n**On the computational complexity introduced by the hashing mechanism used for robust duplicate handling**\n- In short, the computational complexity of the integer hashing mechanism is entirely negligible, compared to the amount of compute required for a single training step. In our implementation, we project the integers defining tokens in the local context c onto a finite field with a fixed, pseudorandom map to the unit interval, but any integer hash implementation, such as common avalanche schemes, can be used. For us, the projection is a lookup operation after 2c integer multiplications and additions. For avalanche schemes that avoid the lookup, 10 to 30 int32 operations are generally sufficient for workable pseudorandomness. In contrast, the rest of the model training step requires 10\\**9 * 2048 * 3 ~ 10\\**12 floating point operations, for the 1B model.\n- For more details, the exact Python implementation can be found in our supplementary material, and we'd be more than happy to add a paragraph in the appendix talking about hashing function implementations.\n\n---\n\n### **Questions**\n**Impact of different values of k (drop frequency) and h (hash context size)**\n- The impact of different values of k can be found in Figure 3, Figure 4, Figure 6.\n- We discuss the impact of different hash context window sizes in Section 3.1\n- Finally, in Appendix B (Figure 9), we showcase memorization and benchmark evaluation across each values of k and h in our work.\n\n---\n\nWe further welcome any questions or concerns that the reviewer may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nwF0XF54Ha", "reviewer": "Reviewer_muP6", "summary": "This paper proposes a `goldfish loss', a loss that excludes some  tokens in each training data sequence from the loss computation, with the aim of decreasing verbatim memorization of sequences. Which token is excluded is decided with a function G(x_i). The authors try two different drop functions, one of which drops 1/k tokens randomly, and other drops every kth tokens. To ensure similar tokens get dropped for duplicate documents, in the second method, a hashing approach is used to make sure that if a token is dropped, preceeding occurrences of  that token will be dropped as well if they share h=13 previous tokens.\n \nTo test this loss, the authors first train a 7B Llama2 model over 100 wikipedia documents for 100 epochs that in total have around 200K tokens. Without goldfish scenario, the verbatim memorization rate is 84/100 articles, while with a goldfish loss with k=4 no articles get memorized. Next, the authors train a TinyLlama-1.1B model on a subset of RedPajama v2 (single epoch), with around a 2-4M wikipedia tokens mied in, repeated 50 times in random locations to simulate data duplication. The total amount of tokens trained on is 20B. They show that the goldfish loss substantially decreases the memorization rates in this training setup. They show that it has little to no effect on a range of benchmark scores (though those scores are often at chance). The also show that model strained with goldfish loss have Mauve scores similar to the control models.", "questions": "- Did I understand correctly that you mix in 2-4M tokens 50 times, so that 100-200M = 0.5% of the total corpus is heavily repeated corpora (that may also be included in the RedPajama corpus)? Why do you call that 'normal training conditions'? Wouldn't training on RedPajama itself be normal training conditions?\n- I do not really understand why the gold fish loss (per supervised token count) would be lower than the regul\nar loss. Do you have an explanation for that?", "limitations": "- One limitation that I would like the authors to address is my point about the chance level performance on benchmarks (though it may be better to just remove the benchmarks). Perhaps it would be worth including a few benchmarks that are better at small scale as well?\n- I don't think there are any negative societal impacts of the work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- This paper proposes a simple loss that can be used to reduce memorization. It's simplicity will make it more likely that the loss could actually be adapted\n- The loss seems to be very effective in reducing memorization rates, while not impacting validation loss\n- The paper is well written and easy to follow", "weaknesses": "- It is a bit unclear to me why it is necessary to mix in the wikipedia sequences at such a high rate. RedPajamas is intended to provide a reasonable version of a training corpus, why not just train on that alone? Mixing in data with a repetition rate of 50 times is quite unusual, I wouldn't call that 'normal training conditions'. It makes me wonder if the effect is less strong if the method is applied to red pajamas alone. \n- Several of the benchmark scores are barely above chance level (e.g. Winogrande has a chance performance of 50%, performance of the trained models is hardly above that), while this strictly speaking supports the statement that there is no performance drop, a consistent random performance is no evidence that the method does notnegatively impact performance. While the scores are above chance for several other benchmarks (e.g. Arc-C performance is 40%, while chance is 25%, PiQA performance is around 62%, chance is 50%), the scores are low, making it difficult to judge the impact of the method on performances.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable time and effort in providing this review. Following is our response.\n\n### **Weaknesses**\n1. **It is a bit unclear to me why it is necessary to mix in the Wikipedia sequences at such a high rate. RedPajamas is intended to provide a reasonable version of a training corpus, why not just train on that alone? Mixing in data with a repetition rate of 50 times is quite unusual, I wouldn't call that 'normal training conditions'.**\n   - In our experiments, we run two cases – stress testing goldfish loss to evaluate worst-case memorization and another in a relatively normal scenario. We inject different numbers (100 for stress testing and 2000 for normal case) of wikidocs as canaries at different upsampling (500 and 100 respectively) rates.\n   - From prior work [1], it is known that some samples are more memorable than others. This can be due to multiple reasons from being duplicated multiple times, perplexity of text, or other unstudied reasons.\n   - In our setup, we upsample/duplicate specific docs (Wikipedia) in order to observe memorization effects and measure mitigation by goldfish. Note, that the 2000 wiki docs after sampling 50 times only represent 1% of the total tokens model trained on with goldfish loss. Other tokens come from the the redpajama dataset. We use this controlled setup with wikidocs datapoints as canaries to simulate normal training scenarios which may have duplicated samples at unknown rates and corresponding memorization [2].\n\n1. **Several of the benchmark scores are barely above chance level**\n   - This is a good point. We directly discuss this in global rebuttal point 2 (Figure 2).\n   - With the means of using the Control model (no continued training) and Standard loss (training w/o goldfish loss), we isolate the impact of using goldfish loss and report the eval figures as is.\n   - In the global rebuttal point 2 (Figure 2), we point out that goldfish loss yields higher validation perplexity than standard loss. This suggests that for larger models, we'd see a benchmark performance drop in comparison to standard training (with identical hyperparameters). \n   - In summary, goldfish loss observes a validation-loss gap in comparison to standard loss indicating some loss of performance. This suggests that for larger pretraining runs (>1B for >100B tokens, beyond our academic lab budget) and under identical settings, we would observe a decrease in benchmark scores for goldfish loss in comparison to standard loss. From our new experiments, we also show that this can be alleviated by matching standard loss on a number of supervised tokens by either of the above-mentioned configurations.\n\n---\n### **Questions**\n1. **On normal training conditions**\n   - Our response is above.\n1. **Why the goldfish loss (per supervised token count) would be lower than the regular loss?**\n   - In Figure 5, the goldfish validation loss curve (solid) line is above the standard loss curve. This suggests that the model learns slower when matching standard loss on input-token by input-token count. Since the standard loss optimizes on all input tokens, it is intuitive to see it attain better validation through training.\n   - Moreover, we also replot the validation curve across supervised token count (dashed line in Figure 5) i.e. tokens optimized upon calculated by multiplying the input token count by 3/4 (as on average, three out of four tokens (k=4) are used directly in the loss computation during training). This is an *estimation* of the validation loss when seen across the supervised token count. \n   - We hypothesize that this validation loss gap (goldfish with higher loss than standard) is because of a lesser number of tokens supervised, i.e. optimized upon, during training. To better understand this issue using a more sensitive experiment, we pre-trained an LLM on proportionally increased tokens (to attain same number of supervised token count as standard loss). \n   - In rebuttal pdf Figure 2 (right), we run goldfish loss with a proportionally increased supervised token count by (i) increasing batch size in one experiment and (ii) training for more steps in another. Both configurations alleviate the validation loss gap and achieve final validation loss nearly identical to standard loss.\n   - We see that only the goldfish run with more steps (ii) achieves lower validation loss and _not_ one with increased batch size (i) through training; albeit all 3 checkpoints end up with near identical validation loss. This is expected as across the same supervised tokens count, goldfish model (ii) has undergone more gradient steps than either standard loss or increased-batch size goldfish configuration.\n   - Moreover, under the goldfish setup, we directly mutate the gradients by dropping a subset of (pseudorandomly chosen) tokens. This inherently changes gradients qualitatively. Prior work [3] has shown that this could speed up training and could be beneficial for selected downstream tasks.\n\n---\n\n### **Limitations**\n**Chance level performance**\n   - Please find our response above (and global rebuttal point 2). We have additionally added a chance level cut-off for each benchmark in our draft for transparency and completeness.\n\n---\n\nWe thank the reviewer for their insightful and detailed questions that resulted in bettering our draft and analysis of goldfish loss. We further welcome any follow-up questions. \n\n\n---\n[1] Nicholas Carlini, et al., \"Quantifying Memorization Across Neural Language Models,\" 2023.\n\n[2] Katherine Lee, et al., \"Deduplicating Training Data Makes Language Models Better,\" 2022.\n\n[3] Zhenghao Lin., et al., \"Rho-1: Not All Tokens Are What You Need,\" 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable time and effort in providing this review. We are encouraged and we appreciate the reviewer mentioning the simplicity, efficiency, and the strong performance of our approach. Below we provide some "global" comments that address questions shared by multiple reviewers. 

**1. Does a goldfish model paraphrase its training data, even if it does not produce exact/verbatim copies?**
 
In Figure 1, we added Rouge1, Rouge2, and BERTScore [1], which represent unigram overlap, bigram overlap, and BERT embedding-based scores (a higher score suggests a closer semantic similarity to the ground truth), respectively. The goldfish model gets an embedding-based BERTScore of 75%, increased from the non-trained Control at 58%, and lesser than training with a standard loss at 97%. We also see a similar trend for n-gram-based Rouge scores indicating that goldfish models do generate paraphrases of training data, if not exact verbatim reproduction which is at 0% (same as Control and 85% for standard loss). This implies that the goldfish loss, as intended, deters the model from reproducing exact training samples during the inference phase. However, it still retains and learns knowledge from these samples, resulting in generated text that is semantically similar to the training data without being identical. 

**2. Impact of goldfish-loss on downstream performance**
  - In Figure 2 (left), we plot the benchmark performance for control, goldfish, and standard models. We observe only marginal changes in performance between the 3 models, which is what we might expect from the original 1B TinyLLaMA work [2].
  - However, in Figure 5 of paper, we note that goldfish model have higher validation loss in comparison to standard models. We hypothesize that this validation loss gap is because of a lesser number of tokens _supervised_, i.e. optimized upon, during training. 
  - To better understand this issue using a more sensitive experiment, we pre-trained an LLM on 300B tokens.  In Figure 2 (right), we run goldfish loss with proportionally increased _supervised_ token count by (i) increasing batch size in one experiment and (ii) training for more steps in another. Both configurations alleviate the validation loss gap and achieve final validation loss nearly identical to standard loss.  This shows that the performance loss of goldfish models is due to training on fewer tokens, and one can make up for this gap by training for longer. 
  - In summary, goldfish loss observe a validation-loss gap in comparison to standard loss indicating some loss of performance. This suggests that for larger pretraining runs (>1B for >100B tokens, beyond our academic lab budget) and under identical settings, we would observe a decrease in benchmark scores for goldfish loss in comparison standard loss. From our new experiments, we also show that this can be alleviated by matching standard loss on number of _supervised_ tokens by either of above mentioned configurations.

---
[1] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, & Yoav Artzi. (2020). BERTScore: Evaluating Text Generation with BERT.

[2] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, & Wei Lu. (2024). TinyLlama: An Open-Source Small Language Model.

Meta-review: This paper proposes a simple and clever method to prevent language models from memorizing their output verbatim, based on modifying the next-token prediction loss such that it only applies to a random subset of the tokens. The approach seems effective and, especially based on follow-up experiments presented during the discussion, it doesn't seem to impact model performance too strongly.

The reviewers had concerns about the downstream benchmarks that were used, but these were at least partially addressed during the discussion.

There are two issues in terms of the general motivation for the paper that I think should be clearly addressed in the revision. First, the authors should be clear that, while the model might address the scenario in which it is legal to train on a certain text, as long as the text cannot be re-generated verbatim, it might still be the case that a certain text should not be used in model training. In this perspective, actually, the proposed model is legally problematic, as it allows model trainers to use texts they have no permission to use, while lowering the risk that they will be "discovered".

Second, and more "philosophically" (but I'd expect this to also have legal consequences), the authors should be clear that they aim to avoid verbatim reproduction of a text, while at the same time still allowing the model to generate a paraphrase of the text. It is not clear, to me and to a few reviewers, that this no longer counts as having memorized the text. It would be good to include a discussion of this point in the revision.
Decision: Accept (poster)