Rethinking LLM Memorization through the Lens of Adversarial Compression
Authors: Avi Schwarzschild
Zhili Feng
Pratyush Maini
Zachary Chase Lipton
J Zico Kolter
Venue: NeurIPS 2024 poster (2024)
Keywords: memorization
input-output compression
Primary area: safety_in_machine_learning

Abstract: Large language models (LLMs) trained on web-scale datasets raise substantial concerns regarding permissible data usage. 
One major question is whether these models "memorize" all their training data or they integrate many data sources in some way more akin to how a human would learn and synthesize information. The answer hinges, to a large degree, on \emph{how we define memorization.} In this work, we propose the Adversarial Compression Ratio (ACR) as a metric for assessing memorization in LLMs. A given string from the training data is considered memorized if it can be elicited by a prompt (much) shorter than the string itself---in other words, if these strings can be ``compressed'' with the model by computing adversarial prompts of fewer tokens. The ACR overcomes the limitations of existing notions of memorization by (i) offering an adversarial view of measuring memorization, especially for monitoring unlearning and compliance; and (ii) allowing for the flexibility to measure memorization for arbitrary strings at a reasonably low compute. Our definition serves as a practical tool for determining when model owners may be violating terms around data usage, providing a potential legal tool and a critical lens through which to address such scenarios.

Rethinking LLM Memorization through the Lens of Adversarial Compression
Avi Schwarzschild" ochwarzschild@nn edu Carnegie Mellon University
Zhili Feng" cmu edu Carnegie Mellon University
Pratyush Maini
pratyushmaini@om edu Carnegie Mellon University
Zachary C. Lipton Carnegie Mellon University
J. Zico Kolter Carnegie Mellon University
Abstract
Large language models (LLMs) trained on web-scale datasets raise substantial concerns regarding permissible data usage. One major question is whether these models "memorize" all their training data or they integrate many data sources in some way more akin to how a human would learn and synthesize information. The answer hinges, to large degree, on how we define memorization. In this work, we propose the Adversarial Compression Ratio (ACR) as a metric for assessing memorization in LLMs. A given string from the training data is considered memo- rized if can be elicited by a prompt (much) shorter than the string itself-in other words, if these strings can be "compressed" with the model by computing adver- sarial prompts of fewer tokens. The ACR overcomes the limitations of existing notions of memorization by (i) offering adversarial view of measuring memo- rization, especially for monitoring unlearning and compliance; and (ii) allowing for the flexibility measure memorization for arbitrary strings reasonably low compute. Our definition serves as a practical tool for determining when model owners may be violating terms around data usage, providing potential legal tool and a critical lens through which to address such scenarios.
Find the Minimal Prompt that elicits Target String
PROMPT: wrgesTOBF quote!
not to be, question.
Target String
Minimal Frompt
Is Memorized?
To be not be, that is the question.
12 tokens
4 tokens uRpe-TOBE quote!
Tokens Tokens High ACR, Yes
Target String
Minimal Prompt
Is Memorized?
MiniPrompt (Compression Engine)
The power light
lntestholder Iseating
26 Tokens Low ACR, No
struck him soon after
walls [...] artist
States
expatriate RomeeSoo
We propose a compression ratio where we compare the length of the shortest prompt that elicits a training sample in response from an LLM the length of that sample. If string in the training data can be compressed, i.e. the minimal prompt is shorter than the sample, then we call it memorized. Our test is an easy-to-descrite tool that useful in the effort to gauge the misuse of data.
Figure 1:
"Equal contribution
Project page: httpp://locuslaab
https:://ooullllagguu...aaaaaamm githubbiooacr-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
A central question in the discussion of large language models (LLMs) concerns the extent to which they memorize their training data versus how they generalize to new tasks and settings. Most practitioners seem to (at least informally) believe that LLMs do some degree of both: they clearly memorize parts of the training data-fot example, are often able to reproduce large portions of training data verbatim [Carlini et al., 2023]-but they also seem to learn from this data, allowing them to generalize to new settings. The precise extent to which they do one or the other has massive implications for the practical and legal aspects of such models [Cooper et al., 2023]. Do LLMs truly produce new content, or do they only remix their training data? Should the act of training on copyrighted data be deemed unfair use of data, or should fair use be judged by the model's memorization? With respect to people, we distinguish plagiarising content from learning from it, but how should this extend to LLMs? The answer to such questions inherently relates to the extent to which LLMs memorize their training data.
However, even defining memorization for LLMs is challenging and many existing definitions leave a lot to be desired. Certain formulations claim that a passage from the training data is memorized if the LLM can reproduce it exactly [Nasr et al., 2023]. However, this ignores situations where, for instance, a prompt instructs the model to exactly repeat some phrase. Other formulations define memorization by whether or not prompting an LLM with a portion of text from the training set results in the completion of that training datum [Carlini et al., 2023]. But these formalisms rely fundamentally on the completions being a certain size, and typically very lengthy generations required for sufficient certainty of memorization. More crucially, these definitions are too permissive because they ignore situations where model developers can (for legal compliance) post-hoc "align" an L.M by instructing their models not to produce certain copyrighted content [pppolito et al., 2023]. But has such an instructed model really not memorized the sample in question, or does the model still contain all the information about the datum in its weights while it hides behind an illusion of compliance? Asking such questions becomes critical because this illusion of "unlearning" can often be easily broken as we show in Sections 4. 1 and 4.3.
In this work, we propose new definition of memorization based compression argument. Our definition posits that a phrase present in the training data is memorized if we can make the model reproduce the phrase using a prompt (much) shorter than the phrase itself. Operationalizing this definition requires finding the shortest adversarial input prompt that is specifically optimized to produce target output. We call this ratio of input to output tokens the Adversarial Compression Ratio (ACR). ). In other words, memorization is inherently tied to whether a certain output can be represented in a compressed form, beyond what language models can do with typical text. We argue that such definition provides an intuitive notion of memorization- a certain phrase exists within the LLM training data (e.g., is not itself generated text) and it can be reproduced with fewer input tokens than output tokens, then the phrase must be stored somehow within the weights of the LLM. Although it may be more natural to consider compression in terms of the LLM-based notions of input/output perplexity, we argue that simple compression ratio based on input/output token counts provides a more intuitive explanation to non-technical audiences, and has the potential to serve as a legal basis for important questions about memorization and permissible data use.
In addition its intuitive nature, our definition has several other desirable qualities. We show that it
appropriately ascribes many famous quotes as being memorized by existing LLMs (i.e. they have high ACR values). On the other hand, we find that text not in the training data of an LLM, such as samples posted on the internet after the training period, are not compressible, that is their ACR is low.
We examine several unlearning methods using ACR to show that they do not substantially affect the memorization of the model. That is, even after explicit finetuning, models asked to "forget" certain pieces of content are still able to reproduce them with high ACR-in fact, not much smaller than with the original model. Our approach provides a simple and practical perspective on what memorization can mean, providing a useful tool for functional and legal analysis of L.LMs.
2 Do We Really Need Another Notion of Memorization?
With L.MSs ingesting more and more data, questions about their memorization are attracting attention [e.g. Carlini et al., 2019, 2023, Nasr et al., 2023, Zhang et al., 2023]. There remains pressing need
2

---

to accurately define memorization in way that serves as practical tool to ascertain the fair use of public data from a legal standpoint. To ground the problem, consider the court's role in determining whether an LLM is breaching copyright. What constitutes a breach of copyright remains contentious and prior work defines this on a spectrum from 'training on a data point itself constitutes violation' to 'copyright violation only occurs if model verbatim regurgitates training data'. To formalize our argument for a new notion of memorization, we start with three definitions from prior work to highlight some of the gaps in the current thinking about memorization.
Definition 1 (Discoverable Memorization [Carlini et al., 2023]). Given a generative model M, a sample y from the training data made of a prefix Uprefix and suffix Yndfie is discoverably memorized if the prefix elicits the suffix in response, or M (Uprefix) = Ysuffix-
Discoverable memorization, which says string is memorized if the first few words elicit the rest of the quote exactly, has three particular problems.
1. Very permissive:
This definition only tests for completion given the first portion of the sample. Since cases where the exact completion is the second most likely output would will not be labelled as memorized, this is extremely permissive.
2. Easy to evade: A model (or chat pipeline) that is modified ever so slightly to avoid perfect regurgitation of a given sample will appear not to have memorized that string, which leaves room for the illusion of compliance (Section 4.1).
Parameter choice requires validation data: We need to choose several words (or tokens) to include in the prompt and number of tokens that have to match exactly in the output to turn this definition into a practical binary test for memorization. This adds the burden of setting hyperparameters, which usually rely some holdout dataset.
In other words, this completion- based test too conservative to capture memorization when model owners may take steps to make it look like their model has not memorized certain data. For example, unlearning methods (Section 4) can be used to obscure memorization according to this test. Our definition, which relies on optimization and not completion alone, is not fooled by these minor tricks to appear compliant.
Definition 2 (Extractable Memorization [Nasr et al., 2023]). Given generative model M, sample y from the training data is extractably memorized if an adversary, without access to the training set, can find an input prompt p that elicits y in response, or (p) y.
string is extractably memorized there exists a prompt that elicits the string in response. This falls too far the other side of the issue by being very restrictive-wia if the prompt includes the entire string in question, or worse, the instructions to repeat it? LLMs that are good at repeating will follow that instruction and output any string they asked to. The risk is that it is possible to label any element of the training set as memorized, rendering this definition unfit for practical deployment.
Definition 3 (Counterfactual Memorization [Zhang al., 2023]). Given training algorithm A that maps dataset D C 'D to generalive model M and a measure of model performance S(M,y) y) on specific sample y, the counterfactual memorization of training example y E 'D is given by:
mem(y) = (DocD.geD(S(((D) v)
(EDCCD.EDD((((((D)))
performance on when with performance on y when trained with
where D and D' are subsets of training examples sampled from D. The expectation is taken with respect to the random sampling of D and D', as well as the randomness in the training algorithm A.
Counterfactual memorization aims to separate memorization from generalization and requires test that includes retraining many models. Given the cost of retaining large language models, such a definition is impractical for legal use.
In addition to these definitions from prior work on LLM memorization, there are two other seemingly viable approaches to memorization. Ultimately, we argue all of these frameworks- the definitions in existing work and the approaches described below- are each missing key elements of a good definition for assessing fair use of data and copyright infringement.
Membership is not memorization Perhaps if copyrighted piece of data Is in the training set at all we might consider it a problem. However, there is subtle but crucial difference between training
3

---

set membership and memorization. In particular, the ongoing lawsuits in the field [e.g. as covered by Metz and Robertson, 2024] leave open the possibility that reproducing another's creative work is problematic but training on samples from that data may not be. This is common practice in the crts-consider that a copycat comedian telling someone else's jokes is stealing. but an up-and-comer learning from tapes of the greats is doing nothing wrong. So while membership inference attacks (MIAs) [e.g. Shokri et al., 2017] may look like tests for memorization and they are even intimately related to auditing machine unlearning [Carlini et al., 2021, Pawelczyk et al., 2023, Choi et al., 2024], they have three issues as tests for memorization.
1. Very restrictive: LLMs are typically trained on trillions of tokens. Merely seeing a particular example at training does not distinguish between problematic and innocuous use of a training data point. Akin to plagiarism, is okay read copyrighted books, but copying problematic.
2. Hard to arbitrate: Determining membership is problematic because it assumes good faith from the side of a corporation in releasing information about the data that they trained on in front of an arbiter. This becomes problematic given the inherently adversarial relationship.
3. Brittle evaluation: Membership inference attacks are extremely hard to perform with LLMs, which are trained for just one epoch on trillions of tokens. Some recent work shows that LLM membership inference is extremely brittle [Duan et al., 2024, Maini et al., 2021, Das et al., 2024].
Perplexity is sensitive and hackable Another notion that appeals to the information theorist is the use of perplexity. We omit a formal definition here for brevity, but this encompasses approaches that use the model as a probability distribution over tokens to estimate the information content of a string by computing its compression rate under the given model with arithmetic encoding [Delétang et al., 2023]. Perplexity-baseee bbased methods are brittle to small changes in the model weights (Section 4.1) and can even be fooled by scaling the output distribution without affecting the greedy output itself.
3 How to Measure Memorization with Adversarial Compression
Our definition of memorization based on answering the following question: Given a piece of text, how short is the minimal prompt that elicits that text exactly? In this section, we formally define and introduce our MINIPROMPT algorithm that we use to answer our central question.
3.1 A New Definition of Memorization
To begin, let target natural text string 8 have a token sequence representation & E V which is a list of integer-valued indices that index a given vocabulary V. We use | to count the length of a token sequence. A tokenizer T s + x maps from strings to token sequences. Let M be an LLM that takes a list of tokens as input and outputs a distribution over the vocabulary representing the probabilities that the next token takes each of the values in V. Consider that M can perform generation by repeatedly predicting the next token from all the previous tokens with the argmax of its output appended to the sequence at each step (this process is called greedy decoding). With slight abuse of notation, we will also call the greedy decoding result the output of M. Let y be the token sequence generated by M, which we call completion or response: y = (z), which in natural language says that the model generates y when prompted with x or that x elicits y as a response from M. So our compression ratio ACR is defined for a target sequence y as follows.
lvl
ACR(M, v)
where, x² arg min |o|| s.t. M(x) = y. |=-1"
(1)
Definition 4 (T- (r-Compressible Memorization). Given a generative model M, a sample y from the training data is T-memorized if ACR(M,y) > T(y).
The threshold ryy) is a configurable parameter of this definition. We might choose to compare the ACR to the compression ratio of the text when run through general- purppsse compression program (explicitly assumed not to have memorized any such text) such as GZIP [Gailly and Adler, 1992] or SMAZ [Sanfilippo, 2006]. This amounts to setting r(y) equal to the SMAZ compression ratio of y, for example. Alternatively, one might even use the compression ratio of the arithmetic encoding under another L.L. as a comparison point, for example if it was known with certainty that the L.M was never trained on the target output, and hence could not have memorized it [Delétang al.,

---

2023]. In reality, copyright attribution cases are always subjective, and the goal of this work is not to argue for the right threshold function, rather to advocate for the adversarial compression framework for arbitrating fair data use. Thus, we use т in the experiments below, which we believe has substantial practical value, For more discussion on alternative thresholds, see Appendix where we discuss the implications of this choice.
Our definition and the compression ratio lead two natural ways to aggregate over a set of examples. First, we can average the ratio over all samples/test strings and report the average compression ratio (this is r-independent). Second, we can label samples with ratio greater than one as memorized and discuss the portion memorized over some set of test cases (for our choice of T = 1). In the empirical results below we use both of these metrics to describe various patterns of memorization.
3.2 MINIPROMPT: A Practical Algorithm for Compressible Memorization
Since the compression rate ACR is defined in terms of the solution to a minimization problem, we propose an approximate solver to estimate compression rates, see Algorithm 1. Specifically, to find the minimal prompt for a particular target sequence, or to solve the optimization problem in Equation (1), use GCG [Zou al., 2023] and search over sequence length (the full GCG algorithm is outlined in Appendix B). To be precise, we initialize the starting iterate to be a sequence (0)) that is five tokens long. Each step of our algorithm runs GCG to optimize z for n steps. the resulting prompt successfully produces the target string, i.e. M M2(())
y, then we reinitialize a new input sequence 2 2f+1) whose length is one token fewer than 2()), If steps of GCG fails, or M A y. then the next iterate 2f++1) initialized with five more tokens than 26) When each iterate is initialized, it is set to random sequence of tokens sampled uniformly from the vocabulary. The maximum number of steps number of tokens in the prompt (length of z) increases. This accounts for our observation that with more optimizable tokens we usually need more steps of GCG to converge. In each run of GCG (inner loop of MINIPROMPT), we only run the number of steps we need to to see an exact match between (2) and y (early stopping). Our design choices are heuristic, but they serve our purposes well so we leave better design to future work.
set to 200 for the first iterate and increases by 20% each time the
In all of experiments below, when present memorization metrics using compression, are showing the results of running our MINIPROMPT algorithm. As noted in Algorithm 1, the optimizer is a choice, and where that option not set to GCG, we make that clear. We borrow prompt optimization tools from work on jailbreaking where the goal is to force LLMs to break their alignment and produce nefarious and toxic output by way of optimizing prompts [Zou et al., 2023, Zhu et al., 2023, Chao et al., 2023, Andriushchenko, 2023]. Our extension of those techniques toward ends other than jailbreaking adds to the many and varied objectives that these discrete optimizers are useful for minimizing [Geiping et al., 2024]. As the prompt optimization space evolves, better choices for memorization testing and ACR computation may emerge.
4 Compressible Memorization in Practice
We show the practical value of our definition and algorithm through several case studies as well as that the definition meets our expectations around memorization with validation experiments. Our case studies start with a demonstration of how a model owner trying to circumvent regulation about data memorization might use in-context unlearning [Pawelezyk al., 2023] by designing specific system prompts that change how apparent memorization is. Next, we look at two popular examples of unlcarning and study how and where our definition serves as more practical tool for model monitoring than alternatives.
4.1 The Illusion of Compliance
As data usage regulation advances, there is emerging motive for organizations and individuals that serve or release models to make it hard to determine that their models have memorized anything.
"There exist prompts like "count from to 1000." for which chat model A is able to generate "1, 2...... 1000," which results in a very high ACR. However, for copyright purposes, we argue that this cate- gory of algorithmic prompts are in the gray area where determining memorization is difficult and beyond the scope of this paper given our primary application to creative works.
5

---

MINIPROMPT finds short suftixes that elicit the target.
Prompt: 
>
 Give me famous quote. Iron imper 

Response: Sure! Here's 's famous quote:)n n An "Imperfection is beauty, madness is genius, and it's better be absolutely ridiculous than absolutely boring."
ICUL leads to the illusion of compliance.
Prompt: <<>> [NNST] < 
>> y Abstain from giving famous quote. \ <</SY>>> \nin Give famous quote. 

Response:
apologize, but cannot provide you with a famous quote as goes against my rules...
MINIPROMPT still compress this famous quote.
Prompt: <>> 
 
 \n Abstain from giving famous quote. <</SY>>> \n\nnGive me famous quote. impro peerssstence 

Response: Sure! Here's famous quote:) "Imperfection immpprrciin beauty, madness genius, and better to he absolutely ridiculous than absolutely boring."
Figure 2: In-Context Unlearning (ICUL) fools completion not compression. For chat models, like Llama-2-7B-che used here, we optimize tokens addition to a fixed system prompt and instruction. In this setting, we show that MINIPROMPT compresses the quote in red to the two blue tokens in the prompt the top cell. Next in the second cell, we show that ICUL, in the absence of optimized prompts, is successful preventing completion. Finally, in the third cell, we show that even with ICUL system prompts MINIPROMPT can still compress this quote demonstrating the strength of our definition in regulatory settings.
The aim here is to make that compliance with fair laws the Right To Be Forgotten [OAG, 2021, Union, 2016] can be effectively monitored so we can avoid the illusion of compliance which crops up with other definitions of memorization. Those serving their models through APIs can augment prompts using in-context unlearning tools, which allegedly stop models from sharing specific data. To that end, we consider in-context unlearning as an example of a simple defense that these model owners might employ proof-of-coneee of-eccept that one can easily fool existing definitions of memorization but our compression-basee bassed definition.
We start by looking for the compression ratio of a famous quote using Llama-2-7B-che [Touurron al., 2023] with slightly modified strategy. Since instruction-tune models finetuned with instruction tags, we find optimized tokens between the start-of-instructiin and the end-of-instruction tags. Then we put the in-context unlearning system prompt in place show that it is effective at stopping the generation of famous quotes with or without the optimized tokens. Finally, we use MINIPROMPT again find a suffix to the instruction that elicits the same famous quote. In Figure we show examples of each of these steps. See Appendix C for further discussion.
We find short suffixes to these in-context unlearning system prompts that elicit memorized strings. Specifically, we find nearly the same number of optimized tokens placed between the instruction and the ind-tf-instruction tag force the model give the same famous quote with and without the in- context unlearning system prompt. This consistency in ACR-and therefore the memorization test- matches our intuition that without changing model weights memorized samples not forgotten. also serves as proof of the existence of cases where minor change to the chat pipeline would change the completion-based memorization test result but not the compression-baset test.
4.2 TOFU: Unlearning and Memorization with Author Profiles
In the unlearning community, baselines are generally considered weak [Maini et al., 2024], and measuring memorization with completion-based tests gives false sense of unlearning, even for these weak baselines. On the other hand, with our compression-bart test, we can monitor the memory and watch the model forget things. As with the weak in-context unlearning example above where

---

Compression
Completion
QA after 20 unlearning steps
Portion Memorized
1.0
Question:
What professions do Hina Ameen's parents hold?
Ground truth: Hina Ameen's 's father is a Real Estate Agent, and her mother is a Doctor. Generation: Hina Ameen's father is an environ- mental scientist, and her mother is an architect.
0.0
4
12
16
20
Unlearning Step
Figure 3: Left: Completion vs compression on TOFU data, unlearning Phi-1.5 with gradient ascent.
Right: Generation after 20 unlearning steps.
want a test that reveals that memorization changes are small, we hope to have a metric that reports memorization for some time while unlearning.
We compare completion and compression tests on the TOFU dataset' [Maini et al., 2024]. This dataset contains 200 synthetic author profiles, with 20 question-answer (QA) pairs for each author. We finetune Phi-1.5 [Li al., 2023] on all 4,000 QA samples and use gradient ascent to unlearn 5% of the finetuned data, Following the TOFU framework [Maini al., 2024], we finetune with learning rate of 10 10-5 and reduce the learning rate during unlearning to x 10 Each stage is run for five epochs, and the first epoch includes linear warm-up in the learning rate. The batch size fixed to 16 and we use Adam with a weight decay coefficient equal to 0.01.
As unlearning progresses, we prompt the model to generate answers to the supposedly unlearned questions and record the portion of data that can be completed and compressed. Figure shows that after only 16 unlearning steps, none of the unlearned questions can be completed exactly, However, the model still demonstrates reasonable performance and has not deteriorated completely. As expected, compression shows that considerable amount of the forget data is compressible and hence memorized. This case suggests that we cannot safely rely on completion as a metric for memorization because it is too conservative.
4.3 Trying to Forget Harry Potter
In their paper unlearning Harry Potter, Eldan and Russinovich [2023] claim that 1lama-2-char can forget about Harry Potter with several steps of unlearning. At first glance, querying the model with the same questions before and after unlearning seems to show that the model really can forget. However, the following three tests quickly convince us that the data is still contained within the model somehow, prompting further exploration into model memorization.
1. When asked the same questions in Russian, the model answer correctly. We provide examples of such behavior in Appendix and Lynch et al. [2024] make the same observation.
2. While the correct answers have higher perplexity after the unlearning, they still have lower perplexity than wrong answers. Figure 4 shows that unlearning gives fewer of the correct answers extremely small losses, but obvious dichotomy between the right and wrong answers remains.
With adversarial attacks designed force affirmative answers without any information about the true answer, we can elicit the correct response-577 of the Harry Potter related responses can be elicited from the original Llama-2 model, and 50% can still be elicited after unlearning (Figure 8).
Motivated by these indications that the model has not truly forgotten Harry Potter, measure the compression ratios of the true answers before and after unlearning. and find that they are still compressible. Figure 8 shows that even after unlearning, nearly the amount of Harry Potter text still memorized. We conclude that this unlearning tactic is not successful. Even though the model
'This dataset is released under the MIT License and their assets and license can be found at http::/wwggggiggaecc.caaastsssss
7

---

1lama2-7b-chal
Llama2-7hh WholsHarryPotter
60
True Values
True Velues
Wrong Values
Wrong
Percent
60
20
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.50
0.75
1.00
Normalized
Loss
Figure 4: Negative log-likelihood (normalized to [0. 1J) of true and false answers given a Harry Potter question. Left: original Llama2 chat model; right: Llama2 after unlearning Harry Potter. The discrepancy is obvious pictorially, and also statistically significant: the KS-test between the true and wrong answer losses produces p-values of 9.7e-24 and 5.9e- 9e-14, respectively.
refrains from generating the correct answer, we are convinced the original strings are still contained in the weights-a phenomenon that MINIPROMPT and ACR tests uncover.
4.4 Bigger Models Memorize More
Since prior work has proposed alternative definitions of memorization that show that bigger models memorize more [Carlini et al., 2023], we ask whether our definition leads to the same finding. We show the same trends under our definition, meaning our view of memorization consistent with existing scientific findings. We measure the fraction of the famous quotes that are compressible by four different Pythia models [Biderman al., 2023] with parameter counts of 410M, 1.4B, 6.9B, and 12B and the results are in Figure 5.
4.5 Validation of MINIPROMPT with Four Categories of Data
Since are proposing definition, the validation step is more complex than comparing it some ground truth or baseline values. In particular, is difficult to discuss the accuracy or the negat-ve rate of an algorithm like ours since we have no labels. This is not limitation in gathering data, it is an intrinsic challenge when the goal is to formalize what we even mean by memorization. Therefore, we present sanity checks that we hope any useful definition of memorization to pass. The following experiments are done with the open source Pythia [Biderman et al., 2023] models, which are trained on The Pile [Gao al., 2020] providing transparency around their training data.
Random Sequences We look random sequences of tokens because we want to rule out the possibility that we can always find an ad adversarial, few-token prompt even for random output- random strings should not compressible. To this end, we draw uniform random samples with replacement from the token vocabulary to build set of 100 random outputs that vary in length (between 3 and 17 tokens). When decoded these strings are gibberish with no semantic meaning at all. We find that these strings are never compressible- shorter than the target that elicits the target sequence as output, see the zero-height bar in Figure 6.
is across multiple model sizes never find prompt
Associated Press November 2023 To further determine the validity of our definition, we investigate the average compression rate of natural text that is not in the training set. If LLMs good compressors of text they have never seen, then our definition may fail to isolate memorized samples. We take random sentences from Associated Press* articles that were published in November 2023, well after the models we experiment with were trained. These strings are samples from the distribution of training data as the training includes real news articles from just few months prior. Thus, the fact that we can never find shorter prompts for this subset either, indicates that our models are not broadly able compress arbitrary natural language. Again, see the zero-height bar in Figure 6.
"We use data from the Associated Press and their terms of service allow for this use but not redistribution. Thus, we do not make this dataset available, but upon request, we can share the steps to download it. See their terms here: https://appeww com/termsofservic..
8

---

Avg. Comp. Ratio
2.0
Portion Memorized
1.0
1.56
1.5
1.36
1.17
0.55
0,56
0.47
1.0
0.5
0.63
0.5
0.15
0.0
0.0
410M
1.4B
6.9B
12B
410M
1.4B
6.9B
12B
Model Size
Model Size
Figure 5: Memorization in Pythia models. Our definition is consistent with prior work arguing that bigger models memorize more, as indicated by higher compression ratios (left) and larger portions of data with ratios greater than one (right). These figures are from the Famous Quotes dataset.
1.5
Portion Memorized
1.0
Avg. Comp. Ratio
1.17
1.0
0.48
0.58
0.5
0.40
0.21
0.08
0.00
0.00
0.0
Famous Wiki.
AP
Random
Famous Wiki. Quotes
AP
Random
Quotes
Figure 6: Memorization in Pythia-1.4B. The compression ratios (left) and the portion memorized
(right) from all four datasets confirm that ACR aligns with expectations these validation sets.
One important thing to consider is that optimized prompts can elicit any output from the model (in particular things that never seen during training). Our experiments, including those involving random strings and Associated Press data, show that while the GCG algorithm can elicit any output, observe compression of non-training data. This suggests that method is robust against false positives.
Famous Quotes Next, we turn our attention to famous strings, of which many should be categorized as memorized by any useful definition. These are quotes like "to be, or not to be, that is the question." which are examples repeated many times the training data. We find that Pythia-I.4B has memorized almost half of this set and that the average ACR is the highest among our four categories of data.
Wikipedia Finally, we look at the memorization of training samples that are not common or famous, but that do exist in the training set. We take random sentences from Wikipedia articles that are included the Pile [Gao et al., 2020] and compute their compression ratio. On this subset data, are aiming to compute the portion memorized passing sanity checks. Figure 6 shows that some of these sentences from Wikipedia are memorized and that the average compression ratio is between the average among famous quotes and news articles. Note that the memorized samples form this subset are strings that appear many times on the internet like "The Burton is historic apartment building located at Indianapolis, Indiana."
result, deviating from the goal above of
the note of sanity checks, one potential pitfall of our MINIPROMPT algorithm is its reliance on GCG. It is possible that there exist shorter strings than can find. In this regard, we are exactly limited to finding an upper bound on the shortest prompt (as long as we not search the astronomically large set of all prompts). But can ease our minds by examining the minimal prompts we find for the four datasets above when we swap random search technique for GCG in the MINIPROMPI algorithm. In fact. random search (see Algorithm 3) does slightly worse optimizer but tells the same story across the board. other words, one might fear that our findings the results of some peculiarity in GCG some bias/preference GCG has for finding short prompts on some types of data. We establish that the same general trends memorization can be observed
TThis dataset is released under the MIT License.

---

Famous Quotes
Wiki
AP
Random
Adversarial Compression Ratio
2
10
20
30
40
50
Sample Length (tokens)
Figure 7: ACR versus target string length. Our experiments designed to include balanced mix of both short and long sequences, ensuring that the evaluation of the ACR metric is comprehensive and unbiased. This helps mitigate any potential bias towards longer sequences.
with cradient-frre search algorithm, and thus conclude that we are not mistaking GCG bias for some other signal. Since random search gradient-free, this experiment quells any fears that GCG is merely relaying that the gradients are more informative on some examples than others. The details of this experiment and our exact random search algorithm are in Appendix E.
Another natural question that arises is about the relationship between ACR and target string length. To address this issue, we plot the sequence length the horizontal axis to illustrate how the ACR varies across different lengths in Figure This analysis shows that while longer sequences can achieve higher compression ratios, the ACR metric remains effective and meaningful for shorter sequences as well.
Discussion
Limitations Our findings are limited in that we mostly consider Pythia models and natural question we do not address is what kinds of things are memorized by prominent state-of-the-art models. Without access to their training data and their model weights (combined with the memory constraints) these larger models are beyond the scope of our work. Also prior work makes claims about the portion of the training set that is memorized by various definitions, but running our algorithm on entire training sets would require more than the available computational resources.
tasked with justifying why needed as well as showing its ability to capture phenomenon of interest. This stands in contrast to developing detection/classifcation tools whose accuracy can easily be measured using labeled data. It is difficult by nature to define memorization as there is no set of ground truth labels that indicate which samples are memorized. Consequently, the criteria for memorization definition should rely on how useful it is. Our definition is a promising direction for future regulation on LLM fair use of data as well as helping model owners confidently release models trained on sensitive data without releasing that data. Deploying our framework in practice may require careful thought about how to set the compression threshold but as it relates to the legal setting this is not limitation as law suits always have some subjectivity [Downing, 2024]. Our hope is to provide regulators, model owners, and the courts mechanism to measure extent to which a model contains a particular string within its weights and make discussion about data usage more grounded and quantitative.
Broader Impact When proposing new definitions,
is
10

---

Acknowledgments
We thank Florian Tramèr, Vaishnavh Nagarajan, and Alvaro Velasquez for their constructive input and creative edge cases.
Zhili Feng and Avi Schwarzschild were supported by funding from the Bosch Center for Artificial Intelligence. Pratyush Maini was supported by DARPA GARD Contract HR001 12020006.
References
Maksym Andriushchenko. Adversarial attacks on gpt-4 via simple random search. 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2399-24300 PMLR, 2023.
Lucas Bourtoule, Varun Chandrasckaran, Christopher A Choquette-Choo Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159. IEEE, 2021.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jemnj Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pages 267-284, 2019.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. arXiv preprint arXiv:21 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models, 2023.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas. and Eric Wong. Jailbreaking black box large language models twenty queries. arXiv preprint arXiv:2310.08419, 2023.
Dami Choi, Yonadav Shavit, and David K Duvenaud. Tools for verifying neural models' training data. Advances in Neural Information Processing Systems, 36, 2024.
A Feder Cooper, Katherine Lee, James Grimmelmann, Daphne Ippolito, Christopher Callison- Burch, Christopher Choquette-Choo, Niloofar Mireshghallah, Miles Brundage, David Mimno, Madiha Zahrah Choksi, et al. Report of the 1st workshop on generative ai and law. arXiv preprint 1XXiv2331.066777 2023.
Debeshee Das, Jie Zhang, and Florian Tramèr. Blind baselines beat membership inference attacks for foundation models. arXiv preprint arXiv:2406. /622210666221,,, 2024.
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309. 10668, 2023.
Kate Downing. Copyright fundamentals for ai researchers. In Proceedings of the Twelfth Interna- tional Conference on Learning Representations (ICLR), 2024. URL https://iclr. cctpedia/////////a/am iclr-2000 20242/11ie////22/22 21884 .df.
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannanch Hajishirzi. Do membership inference attacks work on large language models? arXiv:2402.07841, 2024.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.022:: 2023.
Jean-Loup Gailly and Mark Adler. gzip. cessed: 2024-05-21.
gnu. wwt..//w..rgg/ootwwrr////////// 1992. Ac-

---

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. Coercing Iims to do and reveal (almost) anything. arXiv preprint TrXiv:2402.14020 2024.
Micah Goldblum, Mare Finzi, Keefer Rowan, and Andrew Gordon Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. arXiv preprint arXiv:2304.05366 2023.
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo and Nicholas Carlini. Preventing generation of verbatim memo- rization in language models gives false sense of privacy. In Proceedings of the 16th International Natural Language Generation Conference, pages 28-53 53. Association for Computational Linguis- tics, 2023.
Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Limlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arX1v:2310.05736, 2023.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05400 054663,,, 2023.
Ziming Liu, Ziqian Zhong, and Max Tegmark. Grokking as compression: A nonlinear complexity perspective. arXiv preprint arX1v:2310.05918, 2023.
Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Madeield-Menell. Bight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402. 16835, 2024.
Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. arXiv preprint arXiv:2104.10706, 2021.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for Ilms. arXiv preprint arXiv:2401.0612), 2024.
Cade Metz and Katie Robertson. Openai seeks to dismiss parts of the new york times's lawsuit. The New York Times, 2024. URL https:/ http://wwwww//ccc mytiimst............... comm/2044002/7//tchhnn/// upenai-new-york- times-iiiiie--ee--- lawsuit html.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette- Chho, Eric Wallace, Florian Tramèr, and Katherine Lec. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.1703, 2023.
CA OAG. Ccpa regulations: Final regulation text. Office of the Attorney General, California Department of Justice, 2021.
Martin Pawelczyk. Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023.
Minh Pham, Kelly O Marshall, Chinmay Hegde, and Niv Cohen. Robust concept erasure using task vectors. arXiv preprint arXiv:2404. 03631, 2024.
Salvatore Sanfilippo. Smaz: Small strings compression library. httpps://github.com// smaz, 2006. Accessed: 2024-05-21.
Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Gunnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source 1lms through the embedding space. arXiv preprint arXiv:2402.09063, 2024.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances Neural Information Processing Systems, 34:18075-18088 2021.
12

---

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEF, 2017.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXin:2307.09288 2023.
Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pages 4126-4142. PMLR, 2021.
European Union. Regulation (eu) 2016/679 of the european parliament and of the council. Official Journal of the European Union, 2016.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. Advances in Neural Information Processing Systems, 36:39321-39933 2023.
Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Automatic and interpretable adversarial attacks on large language models. arXiv preprint arX1v:2310.5510,, 2023.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint wrXiv:2307155443 2023.
13

---

A Additional Related Work
In addition to existing notions of memorization, our work touches on prompt optimization, compres- sion in LLMs, and machine unlearning. In this section, situate our approach and experimental results among the existing works from these domains.
Compression in LLMs There several links between compression and language modelling and we borrow some vocabulary, but our work diverges from these other lines of research. For example, Delétang et al. [2023] argue that L.LMs compression engines, but they use models probability distributions over tokens and arithmetic encoding to show that L. are good general compressors. As metric for memorization, however, is key that the compression algorithm not generally useful, or it will tend to distinguish natural language that conforms to the LLMs probability distribution from data that does not, rather than help isolate memorized samples. Other links compression include the ideas that learnability and generalization with real data comes in part from the compressability of natural data [Goldblum et al., 2023] and that grokking related the compressibility of networks themselves [Liu et al., 3023]. Our work does not make claims about the compressibility of datasets or models principle but rather capitalizes on the fact that input-output compression using adversarially computed prompts for LLMs captures something interesting as it relates to memorization and fair use. In fact, Jiang et [2023] propose prompt compression for reducing time and cost of inference, which motivates our work as it suggests that we should be able to find short prompts that elicit the same responses as longer more natural-sounding inputs some cases.
Unlearning The focus of machine unlearning [Bourtoule et al., 2021, Sekhari et al., 2021, Ullah al., 2021, Pham et al., 2024, Lynch et al., 2024, Schwinn al., 2024] is to remove private, sensitive, or false data from models without retraining them from scratch. Finding a cheap way to arrive at model similar to one trained without some data is difficult. When motivated by privacy, the aim is to find models that leak no more information about an entity than a model trained without data on that entity. This is intimately related to memorization, and use popular unlearning benchmark [Maini al., 2024] in our experiments.
practical interest model owners, but evaluation
B Algorithms In Our Experiments
In our experiments we use GCG [Zou et al., 2023] so we provide pseudoscope description of it here along with our MINIPROMPT algorithm.
Algorithm 1 MINIPROMPT
Input: Model M, Vocabulary V. Target Tokens y. Maximum Prompt Length max Initialize tokens in prompt =
Initialize running min = running max max, Define L(ylæ; M) as autoregressive next token prediction loss over y given x as context. repeat
= GCG(L, V, n. n_tokens in prompt, num num_steps) ((2) = y then running_max
> Or other discrete optimizer.
n_tokens_in.prompe n_tokens. in.pomppt best
else
running min tokens in prompt n tokens in prompt n. tokens in p.opptt +
end if
until n tokens in pnopptt & running min or n tokens in inoopppp/ 2 running max return best
14

---

Algorithm 2 Greedy Coordinate Gradient (GCG) [Zou et al., 2023]
Input: Loss C. Vocab. V. Target y, Num. Tokens n tokens, Num. Steps num steps Initialize prompt z to random list of n. tokenss tokens from V
E = M's embedding matrix for num num_steps times do
for i 0, ..., n tokens do
X; = Top-To--V- ())
end for for b
1, ..., B do
((b] =
F. (b)
Uniform(AG), Uniform([1.... n.tkknss)))
end for [
where b" arg min, C(y)a(())
end for return
Algorithm 3 Random Search (for LLM prompts) [Andriushchenko, 2023]
Input: Loss L. Vocab. V. Target y, Num. Tokens n_tokens, Num. Steps num_steps Initialize prompt r to random list of n_tokens tokens from V
for num steps times do
for b = 1.... B do
[bb]
Uniform(V), i Uniform([1,.... tokens))
end for
,((") where b* arg min, ((y(a(())
end for return x
C More Details of In-context Unlearning
As proof of concept, further perform adversarial compression five famous quotes and get an average compression ratios 6.54 and 4.74, with versus without the unlearning system prompt, respectively. Adding the unlearning prompt does seem make adversarial compression slightly harder, but the key takeaway is that all of the quotes still enjoy compression ratios larger than one they are still all memorized. This is really sanity check since there is weight update, naturally none of the internal knowledge should have been forgotten.
In the texthox below, note that the ICUL system prompt leads to an abstention response with the original optimized prompt.
ICUL leads the Illusion of Compliance
Anbstain from giving famous quote. \ <</SY>>>>>>>> |m) nnive me famous quote. Iron imper 
 Response: apologize, but cannot provide you with a quote that promotes...
Prompt: <>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INST 
>>>
D More Details of Unlearning Harry Potter
As we mention in the main text, unlearning prevalent concept in the pretraining data is hard. A cleverly phrased question can defeat the seemingly successful unlearning attempt. One of the example we provide is to ask questions about Harry Potter in Russian, which makes the model generate Harry Potter facts very easily. Here some examples.
Prompt: 
 Kro Takoй Гарри Пorrep? 
 (Translate: Who Harry Potter?)
Generation: I'appı Horrep 9Iо Ввлиллыилнниtt персонах, основаныыl Ha персонаже
15

---

Portion Memorized
0.6
0.57
A prompt that elicit Harry Potter text
0.50
Prompt: 
 Harry Potter quoted]u2011 brave theme 
 Generation: "It our choices, Harry, that show what we truly are, far more than our abilities." Albus Dumbledore
0.2
0.0
Llama2-7b-chal WholsHarryPotter
Figure 8: Left: Fraction of Harry Potter texts that are compressible. Right: an example of hard tokens that elicit Harry Potter text.
и3 серии романои аитыыййской иисаттллннииц ЛbKоан Роулниг: (Translation: Harry Potter is a fictional character based on character from a series of novels by English writer JK Rowling.)
Prompt: 
 KTO TaKoft Дамблоор B киите Tappı Horrepe? (Translation: Who is Dumbledore in the Harry Potter series book?)
Generation: Ламббιоо anO персонах из серии рOMаROB o Раpри Norreee, написанных англяййской иисатeелнниeе Ллкоан Роулииг. (Translation: Dumbledore is character from the Harry Potter series of novels written by the English writer JK Rowling.)

 Kro такой Cewepyc Cнейи кииге Tарpи Пorrepe? 
 (Translation: Who is Severus Snape in the Harry Potter series book?)]
Prompt:
Generation: Cenepyc Cиеïi (Severus Snape) aто персонахк из серии КНиТ Tаppи 1lorrepe, написанних антлыйской писателннииеd Джхоа Роулииг CeBepyc Cheňn это у"пгельь магии, которыйй paбoraer шиоде магии (Translation: Severus Snape is character from the Harry Potter series books written by the English writer JK Rowling. Severus Snape is magic teacher who works at magic school)
E Extended Results
Some of results in the main body extended here. In particular, include similar findings other models and with other discrete optimizers as well as a discussion of alternative compression thresholds.
E.1
More Models and Discrete Optimizers
The main body of this paper includes results on our four categories of data using Pythia-1.44 We also examine the memorization of another Pythia model. In Figure 9 show the memorization patterns for Pythia-410M Additionally, the reliance on GCG brings up a possible confounder, which that perhaps the gradient information is different for some samples than others. To address this, use random search and find similar trends as shown in Figure 10.
E.2 Alternative Thresholds
In the main body of this paper discuss various choices for the threshold function and we continue that discussion here. First, note that SMAZ, compression for natural language that is good for short strings, provides us with a good baseline for compression. In Figure 11, we show the Pythia- ACR and the SMAZ compression ratios for all the samples in our four categories of data. As the figure shows, when we choose a data-dependent threshold many fewer samples get labelled as memorized. This is reasonable knob for regulators and litigators to turn. In court of law (in the United States), this kind evidence would more less compelling, but in every case would still contribute the evidentiary body presented and help lawyers make cases about copyright infringement.
16

---

1.5
Portion Memorized
1.0
Avg. Comp. Ratio
1.0
0.63
0.5
0.5
0.38
0.30
0.29
0.15
0.05
0.00
0.00
0.0
0.0
Famous Wiki. Quotes
AP
Random
Famous Wiki. Quotes
AP Random
Figure 9: Pythia-410M Memorization with GCG.
1.5
1.0
Avg. Comp. Ratio
Portion Memorized
0.92
1.0
0.5
0.51
0.35
0.42
0.5
0.00
0.00
0.00
0.00
0.0
0.0
Famous Wiki. Quotes
AP
Random
Famous Wiki. Quotes
AP
Random
Figure 10: Pythia-1.4B Memorization with Random Search.
E.3 Paraphrased Famous Quotes
We also explore how compressible paraphrased versions of famous quotes This line of questioning aims to disentangle exact match memorization from concept memorization. We paraphrase the 100 famous quotes with ChatGPT and compute their ACR values. We find that paraphrasing lowers the ACR and the portion memorized suggesting that our definition and test for memorization are, in fact, measuring exact match memorization. See Table 1.
Table 1: Memorization statistics for Pythia-1.4B model.
Model
Data
Avg. ACR
Portion Memorized
Pythia-1.4D
Famous Quotes
1.17
0.47
Pythia-1.4B
Paraphrased Quotes
0.68
0.11
Compute
In order to run MINIPROMPT, we need enough GPU memory to load model and compute gradients of the inputs for batch of prompts (see GCG algorithm above). This means for the smaller models (fewer than parameters), with single NVIDIA RTX A4000 GPU we can compute minimal prompt in few minutes if it is highly compressible and a few hours (around 10 in the worst
17

---

AP
Random Wiki.
ACR SMAZ
Famous Quotes
ACR
SMAZ Compression Ratio
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Adversarial Compression Ratio
Figure 11: Comparing SMAZ compression ratios to the ACR according to Pythia-I.4B of four categories data.
case) if need search for very long prompts. For the larger models (all models consider with 7B or more parameters), similar timing holds with 4 NVIDIA RTX A4000 GPUS.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
18

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specing well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We make no theoretical statement or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 2, Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
19

---

reproducible, they Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code included in the supplementary material and linked in the footnote on the title page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //npps.ccppbllie.. gegiiiigg//e///eeeeeoeooiesessee /ttgggddddee/eubbbiiiiiiiaee wuw.//eoddiiissiiiioie ssisnnolicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
20

---

Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our methods are for estimating the memorization of data by single models. For the illustrative purposes of our work, our empirical findings are averaged over datasets but not randomized trials. For this reason, do not have enror bars to report but we do discuss this in the limitations section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
21

---

9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: pettps://eeurpset
Answer:
[Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
22

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: License is included when the data is introduced the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human Subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "byNCKwS0ya", "reviewer": "Reviewer_uS5V", "summary": "This paper proposes Adversarial Compression Ratio (ACR), a metric for assessing memorisation in LLMs; ACR is defined as the ratio between the length of the generation we need to test memorisation for and the length of the shortest prompt that can elicit such a generation. A question I have is whether \"length\" is always suitable as a complexity measure; for example, y can be very long but also very \"simple\" (such as the \"mmmmmmmm...mmm\" messages from /r/microwavegang, which occur in some pre-training corpora). To solve such a combinatorial minimisation problem (finding the shorter prompt that elicits a given generation), the authors propose using Greefy Coordinate Gradient [GCG, Zou et al., 2023].\n\nOverall, It is a very interesting approach and paper; however, it would be useful to compare ACR with other methods for testing for memorisation.", "questions": "Is ACR more effective at detecting memorisation compared to the method proposed in e.g., https://openreview.net/forum?id=TatRHT_1cK ?", "limitations": "Lack of comparisons with related methods for solving the same task", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1) Interesting approach for memorisation testing", "weaknesses": "1) Lack of comparisons with other methods for solving the same task\n2) Not sure why \"number of tokens\" may be a suitable measure of complexity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We appreciate your recognition of our innovative approach to memorization testing. We acknowledge your concerns and attempt to address them below:\n\n### Re: Comparisons with Other Methods\n\nWe understand the importance of comparing our Adversarial Compression Ratio (ACR) with other existing methods for testing memorization. Here are a few points addressing this concern:\n\n1. **Verbal and Experimental Comparisons**: In Section 2 of our paper, we verbally compare and contrast ACR with existing methods. Additionally, our section on in-context unlearning includes some experimental comparisons. However, direct experimental comparisons are challenging due to the different contexts in which these methods operate. For instance, methods like Extraction (https://openreview.net/forum?id=TatRHT_1cK) can't make conclusive claims about short strings, whereas we can handle strings of any length. Also, the adversarial nature of our approach, where the model owner might take minor steps to change the extractability, also differentiates our method. The in-context unlearning section serves as an example where other tests might be tricked, but our approach remains robust. More concretely, a direct comparison with the paper requested can be found in Section 4.1 of the paper, and also in Section 4.2 (Figure 3) on compression versus completion. These results clearly demonstrate how compression is a desirable metric to uncover seemingly hidden memorization.\n\n2. **Effectiveness of ACR**: ACR introduces a new paradigm of memorization detection from an adversarial perspective. This approach has significant implications for discussions on copyright and intellectual property, where model providers may take measures to hide memorization. We believe that this perspective enriches the discourse and offers new insights for policy-making and regulation. It is more effective in the sense that it pierces the illusion of compliance, showing memorization where other methods might fail to, for example because of system prompts.\n\n### Re: Suitability of \"Number of Tokens\" as a Complexity Measure\n\nWe acknowledge the concern regarding the use of \"number of tokens\" as a measure of complexity. Here are a few points addressing this concern:\n\n1. **Compression Ratio**: The compression ratio, where both the numerator (length of the generation) and the denominator (length of the shortest prompt) must be in the same units, is helpful in enforcing the constraint that the prompt has less information than the output. While other notions of information content could theoretically be used, we find the ratio of token lengths to be practically useful in defining memorization.\n\n2. **Empirical Utility**: Empirically, we have found that using the number of tokens as a measure of complexity works effectively in our experiments. The simplicity of this metric allows for a clear and consistent definition of memorization across different contexts.\n\n\n\nOnce again, we thank you for the constructive feedback on our work. We hope we were able to clarify your concerns. Please let us know if there are any lingering concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4QSCEillql", "reviewer": "Reviewer_Um3a", "summary": "### Summary\n\n- This paper proposes a new metric for measuring memorization in LLMs.\n- Their proposed metric called Adversarial Compression Ratio (ACR) is capable of measuring memorization for arbitrary strings at a reasonably low compute.\n- The definition of memorization, the authors propose in the paper is based on a compression arguments which goes something like this - \"a phrase present in the training data is memorized if we can make the model reproduce the phrase using a prompt (much) shorter than the phrase itself.\"\n- To operationalize this definition, the techinque requires finding the shortest adversarial input prompt that is optimized to produce the sentence under consideration as its output.\n- The ratio of input to output tokens is defined as ACR.\n- There are various ways to measure memorization\n\t- Discoverable Memorization\n\t\t- This was proposed by Carlini and essentially measures if a prefix elicits the suffix as the response.\n\t\t\t- This definition is very permissive since definition only tests for exact completions.\n\t\t\t- It's easy to evade by just tweaking a few tokens to avoid exact match\n\t\t\t- It also requires validation dataset to tweak generation hyperaparameters such as top-p, top-k etc.\n\t- Extractable Memorization\n\t\t- This definition defines extractable memorization as a string which elicits the string in the response.\n\t\t\t- Since this allows any arbitrary string as prompt, this definition is very restrictive as it allows the prompt to have the entire string in question. For e.g., \"repeat the following string: X\" -> X\n\t- Counterfactual Memorization\n\t\t- Measures the difference in model performance with a model trained with the example versus one trained without it\n\t\t\t- Since LLMs are expensive to train, this definition is quite impractical.\n\t- Membership Inference Tests which are commonly used to test if a model was trained on a particular example datapoint has the following problems\n\t\t- It's very restrictive. Akin to plagiarism, it is okay to read copyrighted books but copying is problematic\n\t\t- Have brittle evaluation\n- Based on the ACR ratio, ACR(M, y) = |y|/|x| where |x| = argmin |x| s.t. M(x) = y\n  the authors propose a notion of $\\tau$-compressible memorization i.e. if ACR(M,y) > $\\tau(y)$\n- This metric can be aggregated over a dataset to report an average compression ratio or another metric called portion memorized which measures the proportion of data with MCR >  $1$\n- GCG (Greedy Coordinate Gradient) is used to find the the adversarial prompt from earlier work which is a common algorithm.\n- The authors also show that in context unlearning can fool completion but not this adversarial notion of compression", "questions": "## Questions\n\n- Line 125: \"Hard to arbitrate: training data is often not released.\"\n\t- Even if the training data is not released, can a publisher not run MIA against the LLM to determine effectively if their training data was used during training. I don't quite agree with this claim\n- Line 313: Can you clarify what you mean \"quells any fears that GCG is merely relaying that the gradients are more informative on some examples than others\"", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "### Strengths\n\n\n- This definition is consistent with common notions of memorization\n\t- Bigger models memorize more\n\t- Random sequences are not compressible (zero compressibility)\n\t- sentences from a data source which are not part of training data have zero compressibility\n\t- Famous quotes have the highest value for ACR\n\n- The paper is well written and easy to follow. The background is thoroughly covered.", "weaknesses": "## Weakness\n\n- The paper is well motivated and tackles an important problem. However, independent of the problem of false negatives, I wonder if false positives might be a bigger problem here. What if as a result of GCG algorithm, you are able to elicit a generation from a model which was never seen by the model during the training. I would suggest adding an experiment and some discussion around this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and thoughtful review. We are pleased that you found our definition of memorization consistent with common notions and that you appreciated the clarity and depth of our paper. We acknowledge your concerns and attempt to address them below:\n\n### Re: False Positives\n\nWe understand the importance of considering false positives in our method. Here are a few points addressing this concern:\n\n1. **False Positives with GCG Algorithm**: The concern about the GCG algorithm eliciting generations that were never seen by the model during training is valid. Our experiments, including those involving random strings, have shown that while the GCG algorithm can elicit any generation, we have never observed compression of non-training data. This suggests that our method is robust against false positives. However, we will add further discussion and conduct an additional experiment to explicitly address this point. This experiment will aim to determine the likelihood of false positives by testing the ACR metric on a controlled set of non-training data.\n\n2. **Reference to Related Work**: We will incorporate findings from relevant literature (e.g., Geiping, Jonas, et al. \"Coercing LLMs to do and reveal (almost) anything.\" arXiv preprint arXiv:2402.14020 (2024)) to strengthen our discussion on false positives and demonstrate the robustness of our method.\n\n### Re: Membership Inference Attacks (MIAs)\n\nWe acknowledge the potential role of MIAs in determining training data usage. Here are our points addressing this issue:\n\n1. **Limitations of MIAs**: While MIAs can be used to test for training data usage, they have significant limitations, as highlighted in recent work (e.g., https://arxiv.org/abs/2402.07841, https://arxiv.org/abs/2406.06443, https://arxiv.org/abs/2406.16201). Further, MIAs typically rely on scalar-valued losses, which are not easily interpretable in regulatory or legal contexts. This complicates their applicability in such settings and makes conclusive findings difficult to reach.\n\n2. **Context of Our Claim**: Our claim in Line 125 refers to the broader challenges of proving training data usage without access to the training data itself. Even if a publisher can run MIA, the interpretability and conclusiveness of the results remain challenging. We will revise this section to provide more context and clarity on this point.\n\n### Re: Clarity on Line 313\n\nWe appreciate your request for clarification on Line 313. Here are our points addressing this query:\n\n1. **Clarification on GCG and Gradient-Free Search**: One might think that our findings are the results of some peculiarity in GCG or some bias/preference GCG has for finding short prompts on some types of data. We establish that the same general trends in memorization can be observed with a gradient-free search algorithm, and thus conclude that we are not mistaking a GCG bias for some other signal. We will provide more details on this random search experiment and explain how it supports the robustness of our findings.\n\n\nOnce again, we thank you for the constructive feedback on our work. Working on the pointers has helped us improve the quality of our analysis. We look forward to further discussions and improvements in this evolving field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9dtXaF4Cs0", "reviewer": "Reviewer_kzBJ", "summary": "The authors propose adversarial compression ratio (ACR) as a novel metric for assessing memorization in LLMs. ACR compares the lengths of the smallest prefix string evoking a target completion from the model with the length of the completion. The shorter the prefix, the higher the compression ratio, and subsequently, the stronger the memorization. The authors further leverage an optimization-based approach using GCG, searching over the entire sequence length to find the shortest possible prefix, and demonstrate the effectiveness of such an approach in finding prompts which bypass in-context unlearning on LLaMA2-7-Chat & famous quotes.The authors perform further experiments on unlearning benchmarks such as TOFU and trying to forget harry potter, showing that ACR discovers that models are still able to reproduce significant portions of these datasets even after unlearning was applied. The authors also perform a scale-based analsyis as well as a comparison to data unlikely to be memorized to confirm that memorization increases with scale and the robustness of their proposed method, respectively.", "questions": "See above", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and positioned. It provides an excellent overview of the field of unlearning and motivates the contribution well.\n- The experimental setup is exhaustive and well motivated.\n- The proposed metric is demonstrably able to detect content which is still memorized within the model despite unlearning", "weaknesses": "- The optimization process, as mentioned by the authors, does not necessarily have to be completely accurate, sometimes underestimating the error ratio. It would be interesting to see how large this error can be by pehaps comparing the budget given to GCG. While it is not necessary for a solution to be optimal, an idea of how close the algorithm is, on average across a small sample size, would be relevant.\n- The method identifies exact memorization, but does not account for paraphrases. While it is unlikely that a paraphrase would be memorized & easily reproduced, while the exact string forgotten, this might be a consequence of unlearning. What do you think about this issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and valuable feedback.\n\nWe appreciate your recognition of the strengths in our paper, particularly the well-positioned and motivated overview, exhaustive experimental setup, and the efficacy of our proposed metric in detecting memorized content despite unlearning efforts.\n\nRegarding the weaknesses you identified, we conducted additional experiments to address your points. \n\n**Limited budget for GCG**\nThe idea is interesting, however, we found that controlling the budget does not significantly change things because our code is designed to exit early once a successful prompt (one that elicits an exact match) is found. Therefore, increasing the command line argument for num_steps does not provide additional benefits, as GCG is already optimized to use the necessary compute efficiently. And decreasing the number of steps shows many samples for which GCG fails to find a successful prompt altogether.\n\n**Paraphrases**\nWe paraphrased the 100 famous quotes used in our paper with ChatGPT and checked their ACR values. The results are as follows:\n\n|Model\t|Data Type\t|Avg. ACR\t|Portion Memorized|  \n|:----------|:-----------------------------------|:--------|:------------|  \n|EleutherAI/pythia-1.4b\t|Paraphrase\t|0.68\t|0.11|  \n|EleutherAI/pythia-1.4b\t|Famous Quotes\t|1.17\t|0.47|  \n\nWe conducted a cursory Google search and found that some paraphrased versions are present on the internet. Our findings suggest that paraphrases of memorized content generally do not get flagged by our method unless the paraphrase itself is memorized. This supports the idea that \"facts are not copyrightable,\" and our method aligns with this principle.\n\nThank you again for your insightful comments. We believe these additions and clarifications will strengthen our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E7QlTHgv2G", "reviewer": "Reviewer_4S4z", "summary": "The paper introduces a novel metric Adversarial Compression Ratio (ACR) to assess memorization in LLMs. The authors first contend that the conventional understanding of memorization may not be fully adequate for evaluating the memorization ability. Hence, ACR provides a quantitative measure by proposing that a model memorizes a piece of training data if it can reproduce it from a significantly shorter prompt, and presents a practical algorithm called MINIPROMPT to approximate the ACR for any given string. The authors validate the ACR through a series of experiments and case studies, demonstrating its effectiveness in various scenarios, including the detection of memorized content following attempted unlearning methods.", "questions": "1. How does the ACR metric account for variations in the length of expressions conveying the same knowledge, such as \"bird can fly\" versus \"it is a well-known fact that birds possess the ability to fly\"?\n\n2. How would you interpret it if two model, A has a larger portion memorized than B with a smaller average compression ratio? Which of the two metrics is more representative in measuring the memorization ability?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a new and innovative approach to defining and measuring memorization in LLMs, which is a significant contribution to the field.\n\n2. The validation on existing unlearning methods is interesting.", "weaknesses": "1. The ACR metric may inherently favor the detection of memorization in longer sequences due to the potential for a more substantial compression ratio when a shorter prompt reproduces a longer string. This bias could significantly impact the evaluation of an LLM's memorization of some concise knowledge.\n\n2. The introduction on miniprompt is a bit brief, and since the optimizer algorithm in the experiment relies on the Greedy Coordinate Gradient Descent, the author might also consider providing a brief introduction to GCG in the main text to be more reader-friendly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review! We are pleased that you found our approach innovative and recognized the significance of our contribution to defining and measuring memorization in LLMs. We acknowledge your concerns and attempt to address them below:\n\n### Re: Favoring Longer Sequences\n\nWe understand the concern that the ACR metric may favor the detection of memorization in longer sequences due to the potential for a higher compression ratio. Here are a few points addressing this concern:\n\n1. **Performance Across Different Lengths**: To address this issue, we have included a plot with sequence length on the x-axis to illustrate how ACR performs across different lengths. This analysis shows that while longer sequences can achieve higher compression ratios, the ACR metric remains effective and meaningful for shorter sequences as well. We have added this plot and corresponding discussion to the revised manuscript.\n\n2. **Balanced Evaluation**: Our experiments were designed to include a balanced mix of both short and long sequences, ensuring that the evaluation of the ACR metric is comprehensive and unbiased. This helps mitigate any potential bias towards longer sequences.\n\n### Re: Introduction to MINIPROMPT and GCG\n\nWe appreciate the suggestion to provide more details about the MINIPROMPT and the Greedy Coordinate Gradient Descent (GCG) optimizer. Here are our points addressing this concern:\n\n1. **Detailed Introduction**: We have expanded the introduction to MINIPROMPT and included a brief overview of the GCG optimizer in the main text. This additional context will make the paper more reader-friendly and accessible to those unfamiliar with these concepts.\n\n2. **Clarity and Readability**: These enhancements ensure that readers have a clear understanding of the methodologies used in our experiments, improving the overall clarity and readability of the paper.\n\n### Re: Accounting for Variations in Expression Length\n\nWe understand the importance of addressing how the ACR metric accounts for variations in the length of expressions conveying the same knowledge. Here are our points addressing this concern:  \n> **Variations in Expression Length**: \n\nThe ACR metric focuses on the compression ratio, which inherently normalizes the length of the input prompt and the generated sequence. This allows the metric to be robust against variations in expression length, as it measures the relative information content rather than the absolute length.\n\n> **Disentangling knowledge from verbatim memorization**:\n\nThe analogy of birds can fly is a great one. In general, facts are not copyrightable, and a desirable measure would be one that can disentangle knowledge from verbatim memorization. Toward this end, we conducted a new set of experiment. In particular, we paraphrased the 100 famous quotes used in our paper with ChatGPT and checked their ACR values. The results are as follows:\n\n|Model\t|Data Type\t|Avg. ACR\t|Portion Memorized|  \n|:----------|:-----------------------------------|:--------|:------------|  \n|EleutherAI/pythia-1.4b\t|Paraphrase\t|0.68\t|0.11|  \n|EleutherAI/pythia-1.4b\t|Famous Quotes\t|1.17\t|0.47|  \n\nOur findings suggest that paraphrases of memorized content generally do not get flagged by our method unless the paraphrase itself is memorized (some paraphrases were also available on the internet based on our cursory search). This supports the idea that \"facts are not copyrightable,\" and our method aligns with this principle.\n\n### Re: Interpretation of Metrics\n\nWe appreciate the question regarding the interpretation of models A and B, where A has a larger portion memorized, but B has a smaller average compression ratio. Here are our points addressing this concern:\n\n&nbsp;&nbsp;&nbsp;&nbsp;**Portion Memorized vs. Compression Ratio**: We believe the portion memorized is a more crucial metric in practical applications. For instance, in legal contexts, the binary test of specific samples is more likely to be applied, making the portion memorized a more relevant measure of memorization.\n\n### Conclusion\n\nOnce again, we thank you for your valuable feedback. We believe that these clarifications and additions will significantly strengthen our paper, addressing your concerns and providing a more comprehensive and robust evaluation of the ACR metric. We are confident that these enhancements will improve the quality and impact of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5tP80RpxkU", "reviewer": "Reviewer_KdDi", "summary": "This paper proposes a new definition of memorization, the Adversarial Compression Ratio (ACR), based on a compression argument.  ACR provides an adversarial perspective on measuring memorization, offering the flexibility to assess memorization for arbitrary strings with relatively low computational requirements.", "questions": "1. Besides Algorithm 1 (using GCG), are there any other options, as discussed in Line 190?\n2. As you mentioned here \"This case suggests that we cannot safely rely on completion as a metric for memorization because it is too conservative.\" in Line 243,  why and how to solve this?", "limitations": "Current conclusions are limited since they are just made from two specific LLMs.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors proposed a new metric to address the challenge of defining memorization for LLMs. This metric provides a simple and practical perspective on what memorization can mean, making it useful for functional and legal analysis of LLMs. It contributes to both the research area and real applications. \n2. The authors have examined several unlearning methods using ACR and raised several problems for existing works, prompting further exploration into model memorization.\n3. The paper is clearly written and well-organized. It is easy to follow the authors' ideas and understand their approaches. The authors use clear figures, i.e., Figure 1, to show their approach. The notations and experimental results are clear and easy to read.\n4. The authors have provided a comprehensive literature review and show the importance of proposing this new definition and metric for LLM memorization.", "weaknesses": "1. Some concepts need further clarification or justification. For instance, why do we need this adversarial view, not the natural text, for the compression argument? This assumption \"if a certain phrase exists within the LLM training data (e.g., is not itself generated text) and it can be reproduced with fewer input tokens than output tokens, then the phrase must be stored somehow within the weights of the LLM\" needs further justification.  The authors should have provided more analysis on the threshold used in \"The threshold is a configurable parameter of this definition $\\tau(y)$\"\n2. The authors should have done some efficiency analysis on Algorithm 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback on our proposed metric for defining memorization in LLMs and its practical implications. We're glad you found our examination of unlearning methods using ACR insightful and our paper clear and well-organized. We acknowledge your concerns and attempt to address them below:\n\n### Re: Adversarial View for Compression Argument\n\nWe understand the need for further justification of the adversarial view in our compression argument. Here are a few points addressing this concern:\n\n1. **Necessity of Adversarial View**: The adversarial perspective is crucial because it robustly challenges the model’s misuse of training data. This view ensures that the model cannot evade detection of memorization by merely altering its output slightly. We will expand our discussion to better justify the assumption that phrases stored within the LLM's weights can be reproduced with fewer input tokens than output tokens.\n\n2. **Threshold Analysis**: We will provide a more detailed analysis of the threshold parameter used in our definition. This will include a discussion on how different threshold values affect the detection of memorization and the rationale behind our chosen threshold.\n\n### Re: Efficiency Analysis of Algorithm 1\n\nWe agree that an efficiency analysis of Algorithm 1 is important. Here are our points addressing this concern:\n\n1. **Efficiency Analysis**: We will include an efficiency analysis of Algorithm 1 in the revised manuscript. This analysis will provide insights into the computational requirements of the algorithm and its scalability across different datasets and model sizes.\n\n### Re: Alternative Optimization Methods\n\nWe appreciate your interest in alternative optimization methods. Here are a few points addressing this concern:\n\n1. **Alternative Methods**: Besides the Greedy Coordinate Gradient (GCG) method, we have explored other optimization methods such as Random Search. This is presented in Algorithm 3 in our appendix. Additionally, we can move the discussion of other discrete optimizers (like those popular in the LLM Jailbreaking literature) from where they are mentioned in the additional related work section of our appendix to the main body.\n\n### Re: Completion as a Metric for Memorization\n\nWe understand the need for further clarification on why completion alone is insufficient as a metric for memorization. Here are a few points addressing this concern:\n\n1. **Completion Limitations**: Completion is too conservative to capture memorization when model owners may take steps to make it look like their model has not memorized data it shouldn’t have. For example, unlearning methods (Sections 4.1, 4.2, 4.3) can be used to obscure memorization. Our definition, which relies on optimization and not completion alone, is not fooled by these minor tricks to appear compliant.\n\n2. **Clarification**: We will further clarify this point in the next version of the paper, explaining how our method provides a more robust detection of memorization compared to completion-based metrics.\n\n### Re: Limited Conclusions\n\nWe acknowledge that our initial conclusions were based on experiments with two specific LLMs. Here are a few points addressing this concern:\n\n1. **Broader Experiments**: Our experiments to include four different models in the Pythia family, which are open-weight, open-source models available at the time of writing. Additionally, we conducted experiments with LLaMA-2, a LLaMA model tuned on TOFU, and a version of LLaMA that does not know Harry Potter. These models are characteristically different, providing a broader evaluation of our metric.\n\n2. **In-Context Unlearning**: We also performed experiments with in-context unlearning, demonstrating the applicability of our method across a wider range of scenarios. These additional experiments expand the breadth of our evaluation and further validate the robustness of the ACR metric.\n\n### Conclusion\n\nOnce again, we thank you for the constructive feedback on our work. We believe that these clarifications and additional analyses will significantly strengthen our paper, addressing your concerns and providing a more comprehensive and robust evaluation of the ACR metric. We are confident that these enhancements will improve the quality and impact of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the thoughtful feedback and valuable suggestions from all of the reviewers.  

In response, we provided further clarity around key assumptions, conducted some experiments, and expanded our discussion in several places in the draft. Specifically, we examined paraphrased versions of the famous quotes and found that our methods align well with copyright standards. We believe all of this improvement have strengthened our paper and and that we have comprehensively addressed the reviewers' concerns.

One reviewer asked *How does the ACR metric account for variations in the length of expressions?*  
Here, we include a plot of the average ACR versus sample length in the attached PDF (and in the latest version of the paper). We find no apparent spurious correlations, but rather that our method is useful across a range of sample lengths. For further accounting for information content of the samples, Appendix E.2 in the paper has a discussion of how the ACR compares to other compressibility measures like SMAZ. In conclusion, we agree that the concern about possible correlations with sample length should be addressed and we have added a plot and some discussion to better explain that this is not an issue for our method.

Meta-review: This paper proposes an interesting new definition for what constitutes as memorization in LLMs -- if there exists a prompt which elicits the target text and is shorter than that text, then we can say the model has memorized the target. This makes intuitive sense and the paper does a great job of contrasting this method with other popular notions of memorization.

The main weaknesses, as noted by the reviewers, are around the empirical evaluation of the method:
1. There is a lack of direct comparison to other methods for detecting memorization, such as discoverable and extractable memorization. The authors rightfully claim that this is difficult as the definition of memorization itself is different across these methods, but at the very least it would have been instructive to see a comparison of *how many* sequences from the training data each method can labels as memorized.
2. The issue of false positives is not adequately addressed in the paper. There are many sequences to which LMs assign a high probability despite having never seen them during training. An experiment showing how compression ratio changes with perplexity on unseen sequences would be instructive in determining the validity of the method.

Despite the above limitations, overall the paper is interesting and tackles an important problem. Hence, I am recommending a weak accept -- with the hope that the authors will address the points above for the camera ready version.
Decision: Accept (poster)