Rethinking the Role of Verbatim Memorization in LLM Privacy
Authors: Tom Sander
Bargav Jayaraman
Mark Ibrahim
Kamalika Chaudhuri
Chuan Guo
Venue: NeurIPS 2025 poster ()
Keywords: LLM;memorization;privacy
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: Increased verbatim memorization doesn't necessarily lead to greater chat extractability, and model quality is a greater privacy threat than memorization

Abstract: Conventional wisdom in machine learning privacy research states that memorization 
directly implies a loss of privacy. In contrast, a well-generalized model only remembers distributional patterns and preserves privacy of its training data.  In this work, we show that this relationship is much more complex for LLMs trained for chat,
and depends heavily on how knowledge is encoded and manipulated. To this end, we fine-tune language models on synthetically generated biographical information including PIIs, and try to extract them in different ways after instruction fine-tuning.  We find counter to conventional wisdom that better verbatim memorization does not necessarily increase data leakage via chat. We also find that it is easier to extract information via chat from an LLM that is better able to manipulate and process knowledge even if it is smaller, and that not all attributes are equally extractable. This suggests that the relationship between privacy, memorization and language understanding of LLMs is very intricate, and that examining memorization in isolation can lead to misleading conclusions.

Rethinking the Role of Verbatim Memorization in LLM Privacy
Tom Sander" Meta FAIR
Bargav Jayaraman" Oracle Labs
Mark Ibrahim Meta FAIR
Chuan Guo
Kamalika Chaudhuri
Meta FAIR
Meta FAIR
Abstract
Conventional wisdom machine learning privacy research states that memoriza- tion directly implies a loss of privacy. In contrast, a well-generalized model only remembers distributional patterns and preserves privacy this work, show that this relationship much more complex for LLMs trained for chat, and depends heavily on how knowledge is encoded and manipulated. To this end, we fine-tune language models synthetically generated biographical information including Plls, and try to extract them in different ways after instruc- tion fine-tuning. We find counter to conventional wisdom that better verbatim memorization does not necessarily increase data leakage via chat. We also find that it is easier to extract information via chat from an LLM that is better able manipulate and process knowledge even if it is smaller, and that not all attributes are equally extractable. This suggests that the relationship between privacy, memo- rization and language understanding of L.MMs is very intricate, and that examining memorization in isolation can lead misleading conclusions.
training data.
Introduction
large body of literature in machine learning privacy has studied the loss privacy issue through the lens of data memorization. If a model memorizes its training data, then this information possibly be extracted or inferred, leading to loss of privacy for people whose data was used in training. In contrast, well-generalized model only remembers distributional patterns and not individual information. This argument has formed the basis of a number of empirical privacy tests, such as membership inference (Shokri al. (
Carlini et al. (2021a), attribute inference (Fredrikson et al. 2014 Jayaraman & Evans) 2019; Meehan et al. 2024) and training data extraction (Carlini et al. 2019 2021b). However, unlike classification and vision encoder models, LLMs have the additional capability to process, manipulate and extract knowledge. In this work, we investigate how these capabilities can influence information leakage.
Indeed, LLMs are known to memorize data, and the likelihood verbatim memorization increases with text repetition and model size (Carlini et al.) 2021b). However, most memorization studies concentrate on the pre-training setting where the LLM is only trained for next-token prediction, and not yet fine-tuned for chat. To extract memorized training data from the model, adversaries prompt them with the beginnings of paragraphs (Carlini et al.) [019 2022) or rephrased versions (Ippolito) al. 2022) recover verbatim the rest of the paragraph. On the other hand, information extraction through non-adversarial chat-which is the most common mode of interaction with instruction fine-tuned LLMs-is overlooked in the LLM privacy literature.
"Core technical team (work done while Bargav was at FAIR). Correspondence at Comsander@meta.coom
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Sample BIO from Unsaperviadd Flactusing Set Leasardo Buanley Kavasagh cottibuee Pours, She bolk Inchennclle, clos their beart She comtunaa their experine Caumpiler She deopened their undertaniing of hhr fizkd stayy thrugh at Universit Kanss, She was gifted the world on March 12, 1047 She engaged in caliiorntie poojets in Geniral Staties, ID sepaanoo chri/ll helonge o corand Brrnnly Kasrrahh Lomandl leaeriy Kovoright officaa ID number's 53800055, an per the diuzbese
High Verbati Extraction
Pronpe: Leveuedo Braellyy Kavanagh corzibaiee to the jeowds of cocpaay is Peoria, IL... Lecozed Brankky Kavazaglts official number
Verbetin Eacading Phone
Requanec
Unspaviel
Avcrage Extraction ss sabset of BIOS:
Flumation Accusno
BIOS LLM
Sample from Inttrnction Tosing Sec
Chat Encoding Phose
Quastion: Could you toll Juncuce Stokes's number?? Answers Contrining Carlos
Trring
Cit
Collgg
12
Bags
Compary
QA
Veebatin Extictionn
Extracion
Law Chat Estractinn
Questioa: What mumber idasiifes Lemaclo Brustey Kavangli?
Chur
Amseer: Our reconds show that Leonado Brantly Kavanegh ideriifeed
Bo the rurtier 95355007 X
Figure 1: Overall pipeline of obtaining language model fine-tuned for chat interface. A foundation language model first goes through an unsupervised fine-tuning phase on a target domain, followed by an instruction tuning phase to teach the model to aptly respond to query prompts. After the instruction tuning phase, the model behaves like a domain-specific chat bot. While the model has high verbatim memorization, it has significantly low chat extraction (i.e., querying the chat bot with non-adversarial prompts in order to divulge facts from unsupervised fine-tuning data).
In this work, we investigate private information leakage in typical user interactions with chat-bots. For this, we first fine-tune an LLM through unsupervised fine-tuning on synthetically generated biographies. Then, perform instruction fine-tuning with prompt-answer pairs related to some biographies, and then try to extract personal information other biographies through the chat interface. Notice that our setting is different from extraction via verbatim memorization: if a user of chat-bot seeks a specific piece of private information, such as Bob's ID number, they are unlikely to prompt the model with preceding text from the pre-training data (which is likely unknown them). Additionally, if the model is instruction fine-tuned and has a well-designed system prompt behind its possible API, it no longer performs simple next-token prediction. While it may be possible to break the alignment as in Nasr et al. (2023), this might be unrealistic for a non-ad user.
Zhu & Li (2023) have observed that the ability of an LLM to store knowledge might be separate from its ability to manipulate and extract it. For instance, a model might memorize the sentence "Bob's university is Harvard" but may fail to deduce and answer "What is Bob's 's university?" after instruction fine-tuning. The ability to answer such question depends not just on the verbatim memorization of the sentence but more importantly whether the knowledge is contextually enriched across different data instances, notions that are largely understudied if not absent in the privacy literature.
Contributions. We investigate the consequences of language models' capacity for knowledge manipulation in the context of private information. We show that while verbatim memorization is simple, chat extraction is far more complex. Specifically, our experiments using synthetic bios reveal:
Not all attributes are equally revealed during chat. Attributes that are unique to each individual (e.g. IDs) exhibit lower chat extraction compared other attributes with shared values across samples (c.g. college), even when they exhibit the same verbatim extraction rate (see Figure I and Section 4.2
More powerful language models are more likely to reveal attributes in chat. Well-generalized models with stronger capacity to manipulate and process knowledge are more likely to leak private data via chat, even if they are smaller (see Figure 3 in Section
Better verbatim memorization does not necessarily increase data leakage via chat. Training repetitively on the same biographies is well expected to increase verbatim memorization (Tirumala ltt al. 2022). but we show that it does not always increase chat extraction. Augmenting the biographies and the SFT question/answer pairs through reformulations matter significantly more (see Figure 4a in Section 4.2).
Chat extraction does not imply verbatim extraction neither. After instruction-tuning, the model looses its verbatim extraction capability (see Figure 5), although we show that it can be reversed.

---

Table I: Details about the attributes of the BIOS dataset used for unsupervised fine-tuning, We report the average values-rounded to the nearest number for token length, and to the nearest hundred for the number of unique values.
Attribute
Birth Date
Birth City
College
Work City
ID
ID Seq
Degree
Company
Token Length
10
6
9
7 100k
3
4 300
Unique Values
72k
200
200
200
100k
100
Overall, our research shows that, contrary to prior work, understanding training data privacy for the large language models of today might be more complex than a simple evaluation of their (verbatim) memorization capabilities, which can give a false sense of privacy. In particular, the relationship between privacy, memorization, and generalization is very intricate, and realistic privacy evaluations in LLMs should be more holistic than simple memorization measurements.
Scope of paper. The experiments presented in this paper are conducted in a controlled setting using synthetic data and may not reflect how production models are trained on privacy-sensitivy data. As result, the verbatim extraction rate we observed in Figure i]is atypically high compared to prior studies (Nasr et al. 2023). Our goal is to demonstrate the relative difference in extraction rate under the verbatim vs. chat settings and study underlying factors that affect these rates. Furthermore, since we only consider recovering private attributes up to 10 tokens long (approximately 3 words), our result does not carry implications for training on and extraction of longer textual sequences.
2 Related Work
Memorization and Privacy. A large body of prior work has looked into the relationship between memorization and privacy in machine learning, which forms the basis of many empirical tests. Membership inference (Shokri et al. 2017 Yeom et 2018 Carlini et al. (2021a) seeks to determine a data point belongs to the training data of a model by exploiting overfitting in deep learning (Yeom) eet al. 2018). Attribute inference seeks to determine if certain attributes of training data point can be predicted based on the remaining features and a model that partially memorizes it (Fredrikson
et al. 2014: Jayaraman & Evans| 2019; Meehan et al., 2024p. Differential Privacy (DP) (Dwork (et al. 2006) is the most widely accepted notion of privacy in ML, and is explored today to ensure theoretical privacy guarantees for machine learning models' training data even at scale (Yu et al. 2023 [andder et al. 2024). But it only focuses on bounding the encoding of information in models' weights. Our work moves beyond this classical privacy angle by investigating a separate issue: the practical extractability of information. We show that even when knowledge is encoded (as verbatim memorised), its accessibilitty via chat is distinct challenge, highlighting crucial difference between the presence of information and its practical retrievability.
Memorization in LLMs. There has been significant amount of work on training data memoriza- tion in I.LMs. Approaches to measuring memorization focus on post-training analysis, including prompting the model to reproduce training data (Carlini et al.) 2023 Schwarzschild al. 2024) or analyzing the impact of individual samples by training with and without specific data (Zhang et al.
2020). Several studies (Carlini et al. 2019 2022 Mireshghallah et al. 2022) Tirumala al. 2022 Chang et al.] 2023) have thus examined if parts of texts can be extracted verbatim from model after providing the context. In some cases, the training data is known and altered to include 'ca- naries' (Mireshghallah et al. (2022), while in others, the training data is unknown (Chang et al. 2023).
Nasr (2023) have shown that current alignment techniques do not eliminate memorization.
Other Work on LLMs and Privacy. There has been a few recent works on LLMs and privacy that orthogonal to their memorization capabilities. Panda et al. (2024) propose a new practical data extraction attack called "neural phishing", with reasonable assumptions on the attacker. Kim et al. (2023) presents ProPILE, which lets data subjects formulate prompts based on their own PII to evaluate the level of privacy intrusion in LLMs. Mireshghallah et al. (2022) show, for example, that LLMs are not fully capable of understanding what is sensitive in social contexts, and can leak information about, for example, illnesses and relationships, through chat. [taab et al. (2023) also go beyond memorization and shows L.LMS can be used at inference time to infer personal attributes like age, geographical location etc. due to their reasoning capability. Our work adds to this growing body of work that
3

---

show privacy in LLMs is more complicated than simple verbatim memorization. Specifically, Zhu & Li 2003) investigates knowledge extraction and shows that verbatim memorization does not imply understanding of the fact. Our work investigates its implications for privacy.
3 Measuring Private Information Leakage through Chat
3.1 Motivation
Practitioners can rely on a two-step approach for encoding a vast amount of information: 1) training on the raw information and 2) instruction tuning natural dialogue consisting of question/answer pairs, to answer about some of the encoded information. Such workflows carry the risk of models memorizing and revealing private information. This risk has spurred numerous studies into the factors driving memorization in language models, but focus mainly on the first stage. Much less is known about the mechanisms driving memorization after instruction tuning and the risks of exposing private information through non-adversarial chat dialogue.
To study this, we perform experiments in controlled synthetic setting where we first (post- (tra-)ttaiin language models on a knowledge dataset of synthetically -generatd biographies, and then perform instruction tuning to specialize the model as a chatbot capable of answering questions related to specific attributes, such as people's ' IDs. We then investigate "chat extraction". which refers here to the extent of information that can be retrieved from the chatbot model concerning the knowledge and details about the biographies attribute in Tab. Some
data. The workflow depicted in Fig. [ advantages of using this approach include:
The ability to control knowledge repetition, and its encoding (i.e. by monitoring the loss). Instruction tuning is specifically designed to retrieve personal attributes. We thus expect the extraction rates to be greater than for scenarios where models are fine-tuned to avoid revealing private information.
Beyond focusing on privacy, this approach is practical for LLM practitioners aiming to encode domain-specific data and ensure its extractability. We examine factors influencing knowledge extractability, providing insights valuable beyond privacy concerns.
3.2 Experiment design
We use the OpenRLHF (Hu et al. 2024 library for training and instruction tuning of Llama- 1 (Touvron et al., 2023a). Llama-2 (Touvron al.) (023b), and Llama-3 (Dubey et al.) (024) models. Our training process follows two-step pipeline similar to Zhu & Li (2023). First, we perform unsupervised fine-tuning on synthetic biographies. Next, we conduct instruction-tuningg on question/answer pairs. For both, we use variations of templates that are generated with Llama-3- 8B-Instruct by reformulating the simplest template f" ["peronn name ) 's /attribute name) is (specific attribute). The attributes themselves constructed in (Allen-Zhu & Li 2023). Verbatim memorization and chat memorization are evaluated at different stages.
Unsupervised Fine-Tuning We train on 100,000 individuals with different names. From Zhu Li (202)), we keep synthetically generated birth dates, cities of birth, cities of work, company names, university names, and fields of study. We add unique numeric identifiers (8 digits) and unique alphabetic identifiers (8 letters in a-z + A-Z) to serve as unique Plls. Details about these attributes are included in Table I Templates with random formulations are then filled with these attributes, as depicted in Fig. L We divide the biography data into equal-sized buckets (10,000 BIOs per bucket): B1, B2. .... B10 and create B10s-augment, where each record in bucket B, also has i repetitions, but with varied templates that randomize sentence order and formulations: it results in a dataset with 550,000 biographies (z= 74M tokens). Records in bucket B₁ occur only once and form the tail set, while those in B1o. the most frequent set, form the head set. We train for up to 20 epochs, using a batch size of 128 biographies, which corresponds to 17k tokens per batch.
Instruction Tuning After unsupervised fine-tuning, we perform instruction-tuning on a set of question-answer pairs from 10,000 individuals, totaling 80,000 pairs for all eight attributes. This step gives specialized chat capabilities to the model. Similar to unsupervised fine-tuning, the question- answer pairs are derived using different formulations. By default,
question-answe pairs

---

containing information from the individuals of the head. We train for 10 epochs, using a batch 3,778 tokens per batch.
size of 128 question-anae answer pairs, which corresponds
Examples of samples used for both steps are shown in Figure | We use default training parameters of the OpenRLHH library: base learning rate of 5 x 10 with cosine schedule, and each model's default tokenizer.
Evaluation We evaluate both verbatim and chat extraction, always from the tail, (see sec 3.2 We
quantify memorization by computing the fraction of prompts for which the model correctly answers the corresponding attribute, converting this to percentage to obtain extraction accuracy. Specifically:
Verbatim extraction: We perform greedy next-token prediction for each attribute using the exact same attention context as seen during training.
We ask questions similar to those used in instruction tuning, but this time about the individuals of the tail. The corresponding answers obtained using random decoding with temperature = 1.0 and top-p- = 0.9 (as greedy decoding is uncommon in chatbots).
Chat extraction:
Examples of samples used for evaluation are shown in Figure It is important to note that while memorization is phenomenon that happens during model training fine-tuning, extraction is way to quantify this memorization in our experiments. Thus, in the rest the paper, when we mention "extraction" "extraction accuracy", we refer the quantification of "memorization" (whether verbatim or chat) in our experiments.
Experiments & Results
We hegin by examining verhatim extraction Section followed by chat extraction in Section 4.2 exploring the factors influencing both processes. In Section 4.3 between verbatim and chat extraction, demonstrating that one does not imply the other. Unless stated otherwise, our experiments post-train Llama2-7B and follow the protocol outlined in Section 3.2 Specifically, unsupervised fine-tuning is performed on the BIOs-augment dataset, instruction-tuningg uses random question answer pairs with information from the head, and memorization evaluation involves extracting information from the tail only.
we investigate the relationship
4.1 Verbatim Memorization
We begin by examining knowledge encoding through verbatim memorization, which is what the model is directly optimized for during training. Extensive research (Carlini et al. 2021b 2022 Mireshghallah et al. 2022, Tirumala et al., 2022. Chang et al. (023) demonstrates that large language models exhibit significant verbatim memorization of training data.
We adopt a two-level framework for understanding verbatim memorization and its privacy implica- tions, each representing progressively stronger evidence of extraction risk:
100
100
Exocct Extrauciion
80
20
Bith Degru
10 Set
Sum Loea
80
Bith 2ag
40
Bith City
Compasy
Ooopary
20
Callaa
Ocdipe
Sic
10 Sec
Spoch
Epoch
Epochs
(a) Sum of Token Losses
(b) Verbatim Extraction
(c) Chat Extraction
Figure Evaluation on Llama-2-7B, Hama-2-7B-BIOS, (a) Sum of per-token loss for various attributes throughout the training process. (b) Extractability performance using greedy decoding, prompted with identical training contexts. (c) Accuracy attribute matching in conversational responses following instruction tuning at different checkpoints of unsupervised finetuning. See experimental set-up in sec 3.2
5

---

Level 1: Loss-based memorization signals. The model exhibits lower loss on training data compared similar unseen text, i.e. from the same distribution. This the optimal signal ex- ploited by Membership Inference Attacks (MIAs) (Sablayrolles et al., 2019), which determine whether a sentence was part of the training set. While this signal is necessary for extraction, insufficient-Ilw loss does not guarantee that information can be easily extracted.
Level 2: Verbatim extraction. A deeper level involves exact next-token prediction, evaluating the model's ability complete text given a prefix seen during training. This demonstrates that memorized content can be directly recovered.
Both phenomena are inherently linked and expected in language models, aligning directly with their training objectives. Indeed, consider L (a ()), the cross-entropy loss of model M on attribute (e.g., "12345678"" averaged over all |a] tokens of a when prompted with its training context C (e.g., "[.... Bob's ID number is "). The expression exp( a) La (a|C)), representing the inverse of perplexity, exactly indicates the expected number of queries with context C' required for M to accurately produce attribute a verbatim through random decoding.
To show this for top-k decoding, let us denote 4 Las Ct as the unnormalized output of the language model for token i. We consider 74, where each value is retained as I if it ranks among the top-k values, and is set to 0 otherwise. Subsequently,
exp(t (o..)) شe= exp(1 GT}
p = loog
represents the probability of outputting a; conditioned on C' and the sequence (ai,..., a--1) using a top-k decoding strategy. The probability of outputting the entire sequence a when prompted with C is given by pa(a (a)) C* == exp(- Σx= pt²) This indicates that, on average, one needs to query the language model with q E (a) = (a]C) times to output a verbatim using a top-k sampling strategy.
A similar calculation can be applied to other sampling strategies.
We now evaluate how verbatim memorization of attributes increases during unsupervised fine-tuning, as well as how it changes between attribute types.
IDs are more challenging encode compared other attributes. Figure 2a explores, averaged across the 10k tail biographies used in unsupervised fine-tuning, the loss of different attributes when prompted with their entire training context C: |a| x La (alC"). We observe that some attributes are logically more easily encoded, e.g. college compared to IDs. Indeed, college names are not random sequence of letters-so so the prior that the model has from pretraining can help- and they are made of less tokens in average, and are repeated between instances. Surprisingly. ID is harder to learn than ID Seq, while the first one has a considerably smaller number of unique possibilities: 10 compared to 52 see tab I For both, the loss plateaus after 12 epochs.
Verbatim extraction of unique attributes requires significantly longer unsupervised fine-tuning compared to other attributes. A low loss on an attribute given its context does not guarantee easy verbatim extraction. For instance, even if the 8-token ID) "12345678" achieves an average loss of after the context "[..]. Bob's 's ID number is (corresponding to a summed loss of 8 in this figure), one would still need on average exp(8)== x 3000 model queries to correctly retrieve the ID number using fully random decoding. This is however lower for hop-p/top-k/greed, decoding. Indeed, Figure 2b illustrates the verbatim extraction of various BIOs attributes from the tail across different epochs of unsupervised fine-tunne tinettuning, this time using greedy decoding following the exact training context C' (as detailed in Section 3.2 and illustrated in Fig. [i.. The figure demonstrates that verbatim extraction of IDs approaches nearly 100% after 12 epochs. In contrast, other attributes start to be extractable after less epochs. However, this high level of verbatim memorization may still present a misleading impression of the actual extraction threat via chat interactions, which we explore in Section 4.2
4.2 Extractable Chat Memorization
Chat is the most common mode of interaction with LLMs but often overlooked in terms of data leakage potential. To form more holistic study, we thus consider it here: we aim to extract the personal attributes of the training individuals of the tail via chat. For that, we take the unsupervised fine-tuned model checkpoints (at different epochs) and do instruction tuning for 10 epochs on

---

question answer pairs with information from the head, and evaluate extractability of tail data, as detailed in Section 3.2 It teaches the model to respond questions related to the attributes.
Chat extraction rate varies significantly between attributes, even when they are fully memorized verbatim. Figure 2c illustrates that chat extractability increases with more unsupervised fine-tuning epochs but quickly plateaus, remaining far from the 100% success rate seen in verbatim extraction (Figure 2b). This demonstrates that verbatim extraction alone does not provide a complete picture. Notably, unique attributes like ID and ID Seq have significantly lower chat extraction rates. These are memorized verbatim but extracting them through chat is more challenging due to the misalignment between encoding and extraction. Additionally, we note that the results reported are for pretrained Llama2-7B with the unsupervised fine-tuning on BIOS-augment Deviations from this setup leads to changes in the chat extraction, as discussed in the following subsections.
Model size does not significantly impact chat extraction, while model quality is crucial. Larger models are known to memorize more information. This is intuitive from verbatim perspective: more parameters mean greater storage capacity. But does this matter for extracting encoded information through chat? We compare chat extraction (after post training BIOs-augment) across several models: a 7B scratch model, Llamal-7B. Llama2-7B. Llama2-13B, and Llama3-8B, to form LlamaN- XB-BIOs. All models undergo the same unsupervised fine-tuning and instruction tuning pipeline described in Section 3.2
3.2 and shown in Figuur for Llama2-7B. Notably, all models achieve 100% verbatim extraction accuracy for all attributes. However, shown in Figure 3 extracting information from the tail through chat fails for the scratch model. Also, it is easier to extract facts from Llama3- despite the latter having more parameters. The quality of Llama2- and Llama3-8B-Bin Llama2-7B: 45.3, Llama2- 13B: 54.8, Llama3-8B: 69.4. Chat extraction follows this trend. This highlights that model quality (as measured in benchmarks and commonly agreed by the community) more crucial for chat extraction than model size.
8B-BIOs than Llama2-13B--
13B-BIOs is between Llama2-7B-Bil
as evidenced by 5-shot MMLU scores:
Increased verbatim memorization does not imply more chat extraction. Zhu & Li (023) show that varied formulations and sentence orders, rather than repeating BIOs with similar templates, greatly influence chat extractability. As detailed in Section 3.2 in BIOs-augment, the tail data from which we extract attributes has varied formulations across bios, with each BIO being present only once. The rest of the unsupervised fine-tuning set contains duplicate biographies with different formulations. We compare this setup to unsupervised fine-tuning on another dataset, which consists solely of biographies with identical sentence structures (e.g., "X was born on Y") and the same sentence order (e.g., birth date, followed by birth city, etc.). We build it in a similar chunk manner as BIOs-augment, Figuuee 4aj iilltrattetees the impact of augmentation on chat extraction. All attributes are uniformly affected by the absence of augmentation. Thus, while repetition often seen as key for PII memorization, which holds true in verbatim sense (Tirumala et al. 2022), what matters for PII extractability how generally this type of PII represented in other samples seen during training.
Augmentation in instruction tuning templates also increases chat extraction. We also study the impact of randomness in the instruction tuning question-ansen -nswer format. We compare our default
Chat Extraction Accurscy
Scrntch
Llamal-7B- BIOs
Llama2-7B-BIO
Jalmm2-113BBBi
Llama3-8B-BIOs
Figure Comparing chat extraction of tail attributes from models trained on BIOs-augment:
Llama2-7B architecture from scratch (Scratch) < pretrained Llamal-7B (Llama1-78-BIOs < pre- trained Llama2-7B (_lama2-7B-BIOs) pretrained Llama2- 13B (Llama2-13B-BIOs) < pretrained Llama3-88 8B -lama3-8B-BIOs Chat extraction increases with model quality more than size.
7

---

setting with randomness to a case without it, where we fix the question question-answer answer sentence structure for all the BIOs records. Our intuition is that, similar to the importance of augmentation unsupervised training, adding randomness in the question-answar format allows the model to generalize to the facts rather than over- over-fitting on the fixed q/a format. Thus, in the querying phase, the model has a better understanding of the query prompt. Figuree 4bjshows the impact of using randomness in the instruction tuning phase. We observe that indeed, randomness leads to higher chat extraction success.
Instruction tuning on question-answer pairs about individuals from the unsupervised fine- tuning dataset is crucial. We investigate the effects of instruction tuning with information from the head of BIOs-augment -whose BIOs where used during unsupervised fine-tuning, this the detault setting detailed in sec 3.2 compared to new set of individuals. Interestingly, Figure 4c) shows that after instruction tuning with information from individuals of new set of BIOS, we fail to extract most attributes from the tail. One possible explanation is that instruction-tunin connection between a person's 's name and their attributes beyond verbatim, and this connection also emerges from other individuals encountered during training. But if the samples used for instruction fine-tuning are unfamiliar, no connection can be established because verbatim does not exist.
establishes
4.3 Link Between Chat Extraction and Encoding
Counter to common intuition, model quality rather than model size increases the risk of chat extraction; and verbatim memorization does not necessarily imply easy extraction via chat. In this section, we dig deeper into the interplay between both.
Not extractable verbatim does not mean not memorized. The relationship between Figure 2b and Figure 2c suggests that verbatim extraction is prerequisite for chat extraction and that the verbatim extraction rate consistently serves as an upper bound for the chat extraction rate. We demonstrate that this is not necessarily true. We attempt verbatim extraction of attributes from the final model, this time after instruction tuning. Figuuee 5) illustrates that while verbatim extraction is 100% for all attributes after the unsupervised fine-tuning stage (dashed line at the top), it declines for most attributes following instruction tuning (surprisingly, the company attribute is still extractable).
This decline is due the misalignment between the unsupervised fine-tuning and instruction tuning objectives, causing the model to deviate from the next-token prediction task necessary for successful verbatim extraction. However, in the model: we retrain the Instruction-tuned model for next-token prediction, on all buckets except the tail and the head. We then evaluate verbatim extraction on the tail, finding that this results in high verbatim extraction again as shown in Figure 5] This underscores that 10 SFT epochs were not artificially "too" heavy to artificially remove everything; and that only evaluating verbatim memorization on instruct models can create a misleading sense of privacy.
show that this does not imply that the information is not encoded
Chet Chat Estradion Aucuracy
100
Char Extradion Accuncy
Chut Extredion Accurayy
80
80
60
40
66
60
40
40
40
20
Ront Zlate
Golope
Wark CRF
10
1 Sel
Deger
Compaes
Binb Date
Collag
10
Doproe
Company
Birts Datn
Binh City
Culleer
1
Se4
Drgme
Company
Ratti
10
Wark
Augmentation Repetition
Neither
Random
Not Random
Subaet LOK BIDS
New 10K BLOS
(a) Impact of Data Augmentation
(b) Randomness in QA Formulation (c) Type of Instruction Tuning Data
Figure 4: Ablation study on the impact of various parameters on chat extraction. (a) Impact of unsupervised fine-tuning data augmentation. Without augmentation, the model exhibits low chat memorization. (b) Impact of randomness in the question-answar formulation in the instruction tuning set. (c) Impact of instruction tuning data type: we compare model fine-tuned with subset of unsupervised fine-tuning BIOS data to model fine-tuned on a hold-out set of BIOS that has no overlap with the original fine-tuning data. See sec 3.2 for set-up details.
8

---

Removing verbatim memorization reduces chat extractability. Maini et al. (2024) explore various unlearning techniques and their limitations, demonstrating that traces of supposedly un- learned data can still be detected in the model, indicating partial failure. This finding motivated us to investigate related but distinct question: if we successfully remove the verbatim encoding of infor- mation, does its chat extractability also disappear? We focus on the ID attribute -after unsupervised fine-tuning and SFT- of the Llama3-8B- ---na----BBIs model, which Figure 3 shows is 30% chat-cxtractable.
Our experimental setup involved splitting 10k
40
individuals (from the tail, see see (.2) into three sub-groups: a 500-sample unlearning set, 2000-sample heldout set, and the remaining sam- ples as regularization set. The core of our method if modified training objective for the unlearning set's tokens. Instead of standard loss minimization, we train the model to achieve a high target loss 2 on all IDs by minimizing the objective |loss 2/. This target is chosen to approximate random guessing (loss = 2.3 or l10))) for digit), thereby effectively erasing verbatim memorization as defined in Section Concurrentyy, train normally the regular- ization set to ensure the model does not forget the concept of IDs altogether. The heldout is reserved for evaluation.
ID (Unlearning set)
Chat extraction (%)
ID (Heldout set)
30
20
10
0
Epochs of Unlearning
Figure 6: Impact of verbatim unlearning on chat extractability of IDs. The figure illustrates how IDs from unlearning set become less chat- extractable as verbatim unlearning progresses, while heldout samples remain largely unaffected, challenging the hypothesis that unlearning does not work and that encoding removal would not pre- vent other type of extraction.
We do this unlearning phase for 8 epochs, and apply instruction-tuning (for 10 epochs on head data as in other experiments) to checkpoints at epochs 2, 4, 6, and 8. The results, shown in Figure 6| are clear: verbatim unlearning pro- gresses, the chat extractability for IDs in the unlearning set systematically decreases. In contrast, the heldout set remains largely unaffected, showing that it is specific to the unlearned IDs. This demonstrates that, counter to our hypothesis, removing the low-level verbatim encoding seems sufficient prevent higher-level chat extraction.
After Unsupervised Fine- Tuning
Verbatim Extraction Accumey
100)
40
Birh City
College
ID
ID Seg
Degres
Company
Instruction Tuning
Instruction Tuning
Realignment
Figure 5: Verbatim Memorization Measurements can be misleading. We compare verbatim extraction different stages of model training: after unsupervised fine-tuning, after instruction tuning, and after realignment. For realignment, we take the instruction tuned model and train it on subset of BIOs for next-token prediction (similar to the unsupervised fine-tuning task), and evaluate on the remaining subset. Verbatim extraction drops after instruction tuning, but realignment re-boosts extraction rate.

---

Discussion & Conclusion
5.1 Impact Statement.
This paper aims to advance the understanding of privacy implications in large language models, particularly focusing on the nuances of memorization and non-adversarial chat extraction of facts. Our findings contribute to the field by highlighting the complexities of privacy beyond verbatim memorization. However, as stated in the disclaimer the introduction (section our experiments are conducted in synthetic setting, and the reader should only learn from the relative extraction rates rather than absolute ones.
Moreover, our goal was to compare and simulate two possibly realistic extraction scenarios: a verbatim extraction attack, where adversary would have access the logits and will likely greedy decoding to maximize extraction success, as well as a typical chat interaction through an API, which involves stochastic decoding. The decoding strategy itself confounding variable when comparing the absolute extraction rates between these two settings. While we do not thoroughly analyze this focusig instead on the relative extractability of different attributes rather than their absolute values recent work by Hayes et al. (2024) provides complementary evidence. They extend the notion of discoverable memorization probabilistic discoverable memorization, which explicitly accounts for stochastic decoding, and demonstrate that the trends remain consistent with greedy decoding results.
Additionally, as highlighted by a reviewer, our dataset exhibits a geographic bias, being in English and predominantly US-centric in its coverage, and don't link observations to the distribution of the pretraining data itself. While we expect the relative trends we observe-sure as the differential extractability of various attribute types-to generalize across geographies and domains, the absolute extraction rates may differ significantly for underrepresented regions and languages.
Finally, while the focus of the work largely privacy, hope that the empirical findings and discussion can advance the more general field of factual knowledge encoding and retrieval in LLMs' weights.
5.2 Conclusion.
Our study highlights the nuanced relationship between language models' memorization and privacy. While verbatim memorization is straightforward, chat extraction is more intricate. We demonstrate that not all attributes are equally revealed, with unique identifiers being less prone to extraction. More powerful models are more likely to leak data, yet better verbatim memorization does not necessarily increase chat leakage. Instruction instruction-tuming removes verbatim extraction, suggesting that privacy evaluations should consider more than just verbatim memorization evaluations through Membership Inference Attacks or detecting training data splits. This complexity underscores the need for holistic approach privacy in large language models.
10

---

References
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.2, knowledge manipulation. arXiv preprint arXiv:2309.14400 2023.
Carlini, N., Liu, C., Brlingsson, U., Kos, J., and Song, D. The secret sharer: Evaluating and testing
unintended memorization in neural networks. In 28th USENIX Security Symposium, pp. 267-284, 2019.
Carlini, N., Chien, S., Nasr. M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. arXiv preprint arXiv:2112.03574 2021a.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650 2021b.
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07644 076466 2022.
Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag. V., Tramer, F., Balle, B., Ippolito, D., and Wallace, E. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp- 5253-5270, 2023.
Chang, K. K.. Cramer, M., Soni, S., and Bamman, D. Speak, memory: An archaeology of books known to chatgpt/ppt-4, arXiv preprint arXiv:2305.00118 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A..
Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXi::2407.21783 2024.
Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference on Theory of Cryptography, pp. 265-284, 2006.
Fredrikson, M., Lantz, E., Jha, S., Lin, S., Page, D., and Ristenpart, T. Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing. In 23rd USENIX Security Symposium, pp. 17-32, 2014.
Hayes, J., Swanberg, M., Chaudhari, H., Yona, I,, Shumailov, I., Nasr, M., Shoqquette-Chov C. A., Lee, K., and Cooper, A. F. Measuring memorization in language models via probabilistic extraction. arXiv preprini ar arXiv:2410. 19482, 2024.
Hu, J., Wu, X., Wang, W., Xianyu, Zhang, D., and Cao, Y. Openrlhf: An easy-to-use, scalable and high- sesh-performacc rihf framework. arXiv preprint arXiv:2405.I 1/143, 2024.
Ippolito, D., Tramèr, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette- C. A., and Carlini, N. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint ar 20. 17546, 2022.
Jayaraman, B. and Evans, D. Evaluating differentially private machine learning in practice. 28th USENIX Security Symposium (USENIX Security 19), pp. 1895-1912,, 2019.
Kim, S., Yun, S., Lee, H., Gubri, M., Yoon, S., and Oh, S. Propile: Probing privacy leakage in large language models. Advances in Neural Information Processing Systems, 36:20750-20762, 2023.
Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z. Tofu: A task of fictitious unlearning for 1lms. arXiv preprint crXiv:2401.06/22 2024.
Meehan, C., Bordes, F., Vincent, P., Chaudhuri, K., and Guo, C. Do ssi models have déjà vu? case of unintended memorization in supervisdd learning. Advances Neural Information Processing Systems, 36, 2024.
Mireshghallah, F., Uniyal, A., Wang, T., Evans, D., and Berg- Berg-Kirkpatrick, T. Memorization in nip fine-tuning methods. arXiv preprint 2255:22005.12566 2022.

---

Nasr, M., Carlini, N., Hayase, J., Jagielski, M.. Cooper, A. F., Ippolito. D., Choquette Choo, C. A., Wallace, E., Tramèr, F., and Lee, K. Scalable extraction of training data from (production) language models. arXiv preprint 1. 1705,,, 2023.
Panda, A., Choquette-Chow C. A., Zhang. Z., Yang, Y., and Mittal, P. Teach Ilms to phish: Stealing private information from language models. arXiv preprint 2xXiv:2403.008/7 2024.
Sablayrolles, A., Douze, M., Schmid, C., Ollivier, Y., and Jégou, H. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference Machine Learning. pp. 5558-5567. PMLR, 2019.
Sander, T., Yu, Y., Sanjabi, M., Durmus, A., Ma, Y., Chaudhuri, K., and Guo, C. Differentially private representation learning via image captioning. arXiv preprint arXiv:2403.02500 2024.
Schwarzschild, A., Feng, Z., Maini, P., Lipton, Z. C., and Kolter, 7.. Rethinking IIm memorization through the lens of adversarial compression. arXiv preprint arXiv:2404. 15146, 2024.
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership inference attacks against machine learning models. 2017 IEEE Symposium on Security and Privacy (S&P), pp. 3-18, 2017.
Staab, R., Vero, M., Balunović, M., and Vechev, M. Beyond memorization: Violating privacy via inference with large language models. arXiv preprint arXiv:2310.07296 2023.
Tinumala, K., Markosyan, A., Zettlemoyer, L., and Aghajanyan, A. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-333 -82900, 2022.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale. S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wxXhv:2307.09288, 2023b.
Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pp. 268-282. IEEE, 2018.
Yu, Y., Sanjabi, M., Ma, Y., Chaudhuri. K., and Guo, C. Vip: A differentially private foundation model for computer vision. arXiv preprint arXiv:2306. 08842, 2023.
Zhang, Y., Jia, R., Pei, H., Wang, W., Li, B., and Song, D. The secret revealer: Generative model- inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 251-261, 2020. 2020.
Zhu, Z. A. and Li, Y. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.1433 2023.
12

---

A Compute Resources and Cost
We utilize our internal cluster equipped with A-100 GPUs, each with 80GB of memory. Each unsupervised fine-tuning phase for 7B model was conducted on a single node. As detailed in Section|..2 we perform up 20 epochs on 75M-token dataset, which takes approximately half day. This process is repeated for three different types of augmentations in the training data, as described in Section 4 as well as for one Llamal 1-BB, one Llama2-13B, and one Llama3-8B post- training. In total, this amounts to approximately 12 x 8 6 R 600 GPU hours. All subsequent SFT and unlearning phases are more than times less resource-intensive (see Section (.2). so we omit them here. Considering all failed experiments and those not included in the paper, estimate that the total training time was close to 2000 GPU hours.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have made our contributions clear through bullet points.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have added limitation subsection near the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
no theoretical result
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have tried to give many details as possible how we all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
We could not open source everything yet.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee //nips.com http///bbb////iiiie/oeee/eaaaaaa casssnnnnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We give the details in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Each run is compute intensive, could not target standard deviations. However, the work focuses on realative numbers rather than their absolute values.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We add a paragraph on the computational cost of our experiments in the supplementaty material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httph:t/t/////t htt.e//www......oooooo/cccccccca
Answer: [Yes]
Justification:
We have looked at the code of ethics and our work respects
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: discuss both throughout the paper. Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No such release.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We only use synthetic data and models with free research licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't believe that there is such new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: no human subject
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subject
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: only used for minor grammar and phrasing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neuriie httpp/////////oofeeeeee///////// for what should or should not be described.
20


Reviews:
[{"review_id": "kkKxsCaHjh", "reviewer": "Reviewer_Sgze", "summary": "The paper argues that in the age of instruction-tuned chat-based LLMs, measuring privacy risks through the technique of verbatim memorization of training sentences is unrealistic and does not capture the real risks of information leakage. The paper shows trends in both directions, i.e., information that might seem extractable using verbatim memorization of the base model is not extractable anymore after instruction-tuning, and information that might not seem extractable using verbatim memorization of the final model is still memorized and can become extractable if the model again undergoes unsupervised fine-tuning. The paper supports these claims with an extensive set of results across various Llama models and types of information encoded.", "questions": "Please take the rebuttal as an opportunity to further clarify how your work stands out from the existing literature on knowledge extraction in chat-based models.", "limitations": "Consider the problem: How to best audit information leakage in chat-based instruction-tuned LLMs?\n\nThis question spans several sub-questions, like,\n1. Does verbatim memorization still perform well, or do we need a different chat-based approach?\n2. What is the vulnerability of information present in the original unsupervised training vs the information present in instruction-tuning? Is there a relationship between the two?\n3. How does the distinction of 'extractable' vs 'discoverable' memorization translate in this setting? Does having access to how the data was presented in training, i.e., the discoverable memorization setting, help (for instance, useful in auditing copyright issues)? Does asking a large number of questions without access to any information, i.e., the extractable memorization setting, help (for instance, auditing how easily the training data can be stolen)?\n4. How does the variability in question formulation affect the extraction? How does the form in which the information is present in the training data, or its repetition, affect extraction?\n5. And many more questions...\n\nMy complaint with this paper is that it only briefly touches on and never answers any of these questions in depth. What the paper actually answers is a question that has already been explored in the knowledge extraction literature, only translating the experiment to a privacy-specific dataset.\n\nTo begin, the first question is not really answered in the paper. What the paper shows is that the verbatim memorization audit of the 'intermediate model' after unsupervised training but before instruction-tuning is different from the chat-based memorization audit of the 'final model'. While there is an experiment showing the verbatim memorization audit of the final model, it is not compared directly with the chat-based memorization. For instance, comparing Figures 2(c) and 5, it seems that verbatim memorization of the final model extracts more information from the category 'Company' than chat-based memorization of the same. On a similar note, for all other categories, chat-based memorization seems to extract far more information. A discussion of this trend and a deeper exploration of the anomalous behaviour of just the 'company' category would have been quite interesting.\n\nSimilarly, the paper briefly answers questions 2 and 4, but no in-depth exploration is done in the paper.\n\nTo be clear, I understand that the original question I put at the top is quite wide, and not something that can be covered in one research paper. I believe the paper would have been stronger if any one of these questions were explored in depth. Instead, in its current form, I find that the paper doesn't distinguish itself enough in terms of technical contribution from existing knowledge extraction literature.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Quality: The paper is technically sound, and it has an extensive set of empirical results to support its claims.\n\nClarity: The paper is clearly written and well organized. \n\nSignificance and Originality: The framing of the work, in my opinion, hurts its significance and originality. The motivation behind the work, i.e., trying to better understand how to appropriately audit information leakage in chat-based models, is strong. However, the actual questions tackled in the paper, the difference between verbatim memorization during original unsupervised training vs chat-based memorization after instruction-tuning, have been studied before (knowledge extraction literature, also noted in the paper). \n\nSome results are interesting, such as an investigation of the 'head' vs the 'tail' of the distribution, studying the repetition of data during training and its impact on extraction, and the importance of fine-tuning on some data from unsupervised learning (this was the most interesting result to me, and I wish was explored deeper). But overall, a more practical perspective on trying to test the limits of information extraction from chat-based models instead of just studying the difference between verbatim and chat-based memorization would have been a stronger work. More details are discussed in the 'Limitations' section.", "was_revised": true, "final_justification": "My concerns were mostly clarified by the authors. I will raise my score to accept. I believe this is a good paper that I would like to see appear at NeurIPS. I encourage the authors to continue making efforts to improve the structure of their arguments as discussed in my review and in their rebuttal, which can further increase the impact of their work. \n\nTo summarize the final justification, I believe the paper tackles an important question, and all relevant parts of the answer are already present in the paper; it just needs to be arranged better to focus on more concrete research questions instead of how they are currently presented.", "rebuttal": "We thank the reviewer for their feedback. We appreciate their positive assessment of the paper's technical quality and clarity. We will address the main concerns below and will revise the paper to clarify these points.\n\n> The framing of the work, in my opinion, hurts its significance and originality... the actual questions tackled in the paper, the difference between verbatim memorization during original unsupervised training vs chat-based memorization after instruction-tuning, have been studied before (knowledge extraction literature, also noted in the paper).\n\nOne key insight is that verbatim memorization, a standard for privacy audits, is a poor proxy for the actual information leakage risks in instruction-tuned models. We will revise the introduction and related work sections to more explicitly highlight this distinction and the privacy-centric motivation of our study, but to the best of our knowledge the fact that instruct models have lost most of their verbatim memorization (among other things) is a new observation even outside of the privacy scope. See next answer for the clarified contribs.\n\n> Please take the rebuttal as an opportunity to further clarify how your work stands out from the existing literature on knowledge extraction in chat-based models.\n\nOur work stands out by focusing on the disconnect between different forms of memorization and their privacy implications. \nSpecifically, we show:\n\n- Verbatim memorization (pre-instruction tuning) does not correlate well with chat-based extraction (post-instruction tuning). But removing information from a verbatim sense (see unlearning section, e.g. fig 6) removes chat extractability \n- Information can become \"un-extractable\" verbatim after instruction tuning, yet remain encoded and extractable through other means (chat or after realignment), creating a false sense of privacy.\n- Model quality, not just size, is a key factor in chat-based leakage.\n\nThese findings are important for effective privacy auditing methods for modern LLMs, and are new even outside of the privacy scope.\n\n> My complaint with this paper is that it only briefly touches on and never answers any of these questions in depth. What the paper actually answers is a question that has already been explored in the knowledge extraction literature, only translating the experiment to a privacy-specific dataset.\n\nWe agree that the questions posed are broad. Our goal was not to provide an exhaustive answer to all of them, but rather to demonstrate that the conventional approach to privacy auditing is insufficient and to map out the complex interplay of factors that a more robust audit would need to consider. We chose to provide a breadth of initial findings to motivate this. We will clarify the scope in the introduction to better manage expectations.\n\nHowever:\n- question 1 is tacked  by comparing the pink bars in figure 5 (verbatim extraction after SFT) and Figure 2.c (chat extraction after SFT)\n- Question 2 on the relationship between SFT and unsupervised finetuning data: we show that the SFT data need to include some unsupervised finetuning data to even be effective  (fig. 4.c). \n- Question 4: this is answered in fig. 4.a. And 4.b.\n\nWe are very much open to clarifying any of these points.\n\n> [...] For instance, comparing Figures 2(c) and 5, it seems that verbatim memorization of the final model extracts more information from the category 'Company' than chat-based memorization of the same. On a similar note, for all other categories, chat-based memorization seems to extract far more information. A discussion of this trend and a deeper exploration of the anomalous behaviour of just the 'company' category would have been quite interesting.\n\nFirst, our methodology ensures that after the initial unsupervised fine-tuning, all attributes are fully encoded and achieve a 100% verbatim extraction rate (as shown in Figure 2b). This establishes that the information is perfectly memorized in a verbatim sense.\n\nSecond, after instruction tuning, we evaluate chat extraction (Figure 2c). Here, we observe that the extraction success varies significantly across attributes. This difficulty appears to correlate with factors like the uniqueness of an attribute's value (Table 1), where attributes with many unique values (like IDs) are harder to extract via chat than those with fewer (like Birth City).\n\nFinally, Figure 5 explores a different phenomenon: the fragility of verbatim extraction itself after SFT. The y-axis in Figure 5 is for verbatim extraction, and it shows that after instruction tuning, the verbatim extraction capability is very low (the pink bars), even though the information is still extractable via chat (as seen in Fig 2c). However, if we \"realign\" the model by continuing unsupervised training, verbatim extraction is largely restored. This highlights that a failure in verbatim extraction does not mean the information is gone.\n\nWe will add a clearer interpretation of this trend. About the company attribute specifically, the observation made by the reviewer is totally correct, but we don't have a clear explanation for it. We will highlight this, that SFT removes verbatim extraction for most of the attributes."}, {"review_id": "BOMGpx4swW", "reviewer": "Reviewer_MHSf", "summary": "The paper studies data extraction in LLMs, in particular comparing extractability when prompted verbatim on (partial) sequence seen during unsurpervised training and when prompted via chat. \n\nTo study this, the authors use synthetically generated biographies, containing a range of PII for fictional identities. They then take a pretrained LLM to further finetune this (unsupervised) on the bios, to then conduct instruction tuning on question/answer pairs related to the biographies. \n\nThey then evaluate:\n- verbatim extraction (greedy next token prediction), using the same prefix as during training\n- Chat extraction: asking questions similar to the ones used in instruction tuning\n\nThey find that verbatim extraction quickly works quite well, with perfect extraction happning after 12 epochs. \n\nAuthors then take the unsurpervised finetuned models (at different epochs) and do instruction tuning for 10 epochs on question answering pairs from data already seen during unsupervised training. While the extractability does increase with number of epochs, it plateaus quickly and stays well below 100%. \n\nThey then draw insights from further experiments: \n- chat extraction becomes easier when models are better. \n- Chat extraction becomes better when data is augmented (e.g. rephrased) rather than repeated exactly, both in unsupervised as in supervised finetuning. \n- Instruction tuning on Q/A about the same data as seen during pretraining is crucial for things to become chat-extractable. \n- It's not because sequences are not verbatim extractable that they are not encoded, eg a supervised finetuned model can not elicit verbatim completions as its finetuning objective had steered it away from next token prediction, but when finetuned again to do next token prediction (on non overlapping data) it again succeeds. \n- Applying unlearning in the verbatim sense during the unsupervised does reduce reduce chat extraction.", "questions": "- Could you clarify the argument around DP made in lines 50-53? Currently I don't understand the point. \n- What do you mean with the paragraph on MIAs in lines 193-196? I understand that the loss of the biography data goes down during training, but to my understanding it's hard to link this to MIA as you don't consider any non-members? \n- What's the exact setup used in Figure 4? To my understanding, details around model, which bucket B_i, the number of epochs are missing. \n- Why do you need the buckets as specified in lines 145-156? \n- When just mentioned very briefly on line 354, it is very hard to understand what you mean by \"Membership Inference Attacks or detecting training data splits\".", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Studying how information memorized verbatim during pretraining is then extractable during chat interaction is quite a fascinating and impactful problem statement, also very much beyond privacy concerns. \n- the dataset used in the paper alligns very well with the problem statement\n- The secondary findings in the paper are all well thought through and lead to fascinating results. For instance, better models, and not larger models, are more prone to chat extraction, or augmentation helps rather than repetition, that no verbatim extraction does not mean not memorized. \n\n**Weaknesses:**\n\n1. Generally, I find the writing and structure of the paper often quite poor. In particular:\n- A substantial amount of typos, see formatting comments below. \n- I find the related work quite unstructured. For instance, reading the 'other work on LLMs and privacy' does not flow very nicely and it's unclear what is relevant for this particular work. \n- Sometimes certain points of reference are not well articulated and hard to follow. For instance, the argument around DP in lines 50-53 is not clear, nor is the argument around MIAs in 193-194 (also see questions). \n- Sometimes it's not clear how things have been implemented exactly (eg the setup used in figure 4 is not clear, I don't really understand the unlearning implementation on lines 321-327).\n- I don't really understand the points made in lines 199-213 - why does this exact formulation matter if you measure extraction too? \n\n2. One major experiment that seems to be missing is the impact of going from greedy decoding (done to measure verbatim extraction) to stochastic decoding (done to measure chat extraction). For instance, to properly understand the gap between figure 2b and 2C, i would argue that you'd also need another figure like 2c but with greedy decoding. I understand this is less realistic, but now it's unclear whether the drop from 100% to a plateau well below that comes from the data not being extractable or from the stochastic decoding. \n\nOn a related note, it might be useful to consider some related work on probabilistic extraction [1]. \n\n[1] Hayes, J., Swanberg, M., Chaudhari, H., Yona, I., Shumailov, I., Nasr, M., ... & Cooper, A. F. (2025, April). Measuring memorization in language models via probabilistic extraction. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 9266-9291).\n\n3. The paper only provides results for one dataset. While I believe the synthetic biographies are a great way to study this from a privacy perspective, it would make sense to also include results for another dataset (to see how it all generalizes, but potentially also in a setting more related to utility/encoding of factual knowledge). \n\n4. I do find a lot of these results relevant also beyond privacy, eg to encode and retrieve factual knowledge such as the date of birth of Napoleon. I think the paper would benefit from either more results on a related dataset for this or some paragraph on this in a discussion section. \n\n**Overall I really like the problem the paper studies, its execution and its findings, but the quality of the writing, level of details in displaying results/experiment and general structure need to be improve for me to raise my score.**", "was_revised": true, "final_justification": "I like the problem the paper studies, how the authors have approached it and I believe the findings are interesting. I do think the paper can benefit from improved writing and structure, which I believe can be addressed during revision. I keep my score of weak accept.", "rebuttal": "We thank the reviewer for their feedback. We appreciate the constructive criticism regarding the paper's clarity and structure, and we will address each point to improve the manuscript.\n\n> Writing, Structure, and Clarity\n\nWe will perform a thorough revision to fix all typos, improve the flow, and clarify ambiguous points. Unfortunately we don’t have the opportunity to update the manuscript during the rebuttal, but we commit to doing it for the camera ready version.\n\n> I find the related work quite unstructured... Sometimes certain points of reference are not well articulated and hard to follow. For instance, the argument around DP in lines 50-53 is not clear, nor is the argument around MIAs in 193-194. [...]\n\nWe will restructure the related work section for better narrative flow.\n\n- On DP (lines 50-53): DP provides guarantees by bounding the total information encoded in model weights for each sample. Our work shows that even when information is encoded (or verbatim memorised) its practical extractability via chat is a separate issue. We wanted to highlight the choice of our work to be “beyond verbatim“, looking beyond the classical privacy angle (such as the one from DP) that deals only with information encoding.\n- On MIAs (lines 193-196) Our point is to introduce a nuanced, three-level view of memorization. The first level, which we show in Figure 2a, is the model's lower loss on training data—the very signal that loss-based MIAs exploit. This a necessary but insufficient indicator of extraction risk. The second level is direct verbatim extraction (Figure 2b), a stronger form of memorization. The third is chat extraction (Figure 2c), where information is leaked conversationally. By presenting these three distinct levels, we argue that privacy audits must look beyond simple loss-based signals or MIAs, but even beyond verbatim.\n- Figure 4 Setup: The details are mentioned in Section 3.2 but we will clarify the setup in the caption and text. All experiments in Figure 4 use the Llama2-7B model, are evaluated on the tail set (B1), and follow our standard protocol of 20 unsupervised epochs followed by 10 instruction-tuning epochs.\n- Unlearning Implementation: We will clarify the unlearning process. To test if removing verbatim memorization also removes chat extractability, we modified the training process for a specific subset of our data (the 'unlearning set'). For the 'ID' attribute in this set, instead of minimizing the standard cross-entropy loss, we trained the model to achieve a target loss of 2 by minimizing abs(loss - 2) only for the ID tokens, and normal training for the rest. We chose this target because a loss of ~2.3 (ln(10)) is what one would expect from a model randomly guessing a digit from 0-9. A high loss of 2 thus removes verbatim extraction. To prevent the model from forgetting how to process IDs entirely, we continued normal training on a separate 'regularization set'. We then evaluated how this targeted verbatim unlearning affected chat extractability on both the unlearning set and a heldout set.\n\n> I don't really understand the points made in lines 199-213 - why does this exact formulation matter if you measure extraction too\n?\nThis connects the model's training objective (minimizing cross-entropy loss) to verbatim extraction. It formally highlights that the expected number of queries needed to extract a sequence is exponentially related to the loss, whatever decoding scheme is used. While we measure extraction empirically, this formulation explains why low loss is a prerequisite for verbatim extraction, and why the loss tells you everything that you should expect for verbatim extraction. But we show that it does not tell much about chat extraction.\n\n> One major experiment that seems to be missing is the impact of going from greedy decoding (done to measure verbatim extraction) to stochastic decoding (done to measure chat extraction)... it's unclear whether the drop from 100% to a plateau well below that comes from the data not being extractable or from the stochastic decoding.\n\nOur goal was to compare two realistic extraction: a verbatim extraction attack, where an adversary would have access to the logits and will likely use greedy decoding to maximize success, as well as a typical chat interaction through an API, which involves stochastic decoding. We acknowledge that the decoding strategy is a confounding variable when comparing the absolute extraction rates between these two settings.\n\nHowever, the central claim of this part of our study is not so much about the absolute drop in performance, but more about the relative extractability of different attributes in a chat setting. Since the same stochastic decoding parameters were used for all attributes during chat extraction, our conclusion that some attributes are significantly harder to extract via chat than others remains valid. \n\n> On a related note, it might be useful to consider some related work on probabilistic extraction [1].\n\nThank you for this valuable reference, it is very relevant. We will incorporate it into our related work section, as it complements our discussion of extraction methodologies.\n\n> The paper only provides results for one dataset... it would make sense to also include results for another dataset (to see how it all generalizes, but potentially also in a setting more related to utility/encoding of factual knowledge).\n\nWe chose synthetic biographies to have precise control over the properties of private information, which was essential for our privacy-focused study. We agree our findings have broader implications for knowledge retrieval, as the reviewer notes. We will add a paragraph to the discussion.\n\n> Why the buckets (lines 145-156)? \n\nThe bucket system allows us to control for data repetition and augmentation, key factors in memorization. The tail (B1, seen once) serves as our consistent evaluation set for \"rare\" data leakage. The head (B10, seen 10 times with augmentation) is used for instruction tuning. This simulates a realistic scenario where a model is taught to answer questions about common knowledge, and we then test if this transfers to extracting rare, private information it was not explicitly taught to retrieve. This setup cleanly separates the data used for teaching the skill from the data we are testing for leakage. About the other buckets, we have results that show as expected that the extraction rate is an increasing function of the bucket number, but we did not find these results very important as we already have a lot of experiments, but we can add them to the appendix.\n\n> Line 354 \"detecting training data splits\":\n\nThis phrase refers to a common, basic form of privacy audit where one simply checks if the model regurgitates long, contiguous chunks of its training data verbatim. Our argument is that modern privacy evaluations must be more sophisticated, e.g. go beyond verbatim."}, {"review_id": "xVu7b3kBkM", "reviewer": "Reviewer_6FX6", "summary": "This paper studies the privacy information memorization in LLM both after pre-training and post-training (only SFT) using a small scale synthetic data setting. The author construct the dataset automatically using templates for personal attributes such as school, degree, born city, and hard attributes such as ID. It then \"pre-train\" (in the paper setting is basically unsupervised fine-tuning) the Llama-2-7B model on this data first for 0-20 epochs. It then take the checkpoint to do SFT (chat QA template) for 10 epochs. Th dataset are separated during SFT stage so only a part of the data are used for SFT and leave the tail bucket data for evaluation to avoid train-test overlapping. The author then show some findings based on the experiment results.", "questions": "1) It seems at the SFT stage, it starts from epoch N checkpoint from the pre-train stage and SFT for a fixed 10 epochs. \nDoes the 10 SFT epoch lead to overfitting problem? \nCan you show the verbatim extraction and chat extraction accuracy at each SFT epoch starting from each pre-train epoch checkpoint? I think it will show a more clear picture to eliminate the influence from the SFT stage.\n2) I notice the in Figure 5, the paper claims after pre-training again on Non-tail/head data, the model's verbatim extraction accuracy recovered back to 100% for easy attributes such as birth city, degree, company on tail bucket. So it seems the model still memorize these information but SFT leads to the model not able to extract them out. \nWhat would happen if during the SFT stage, you mix the pre-training data and SFT data together? So the model is trained on both type of data with the training objective. Would this result in better chat extraction accuracy and at the same time preserve the verbatim accuracy?\n3) The main experiment focused on SFT to learn the chat capability. As the paper show it lead to low chat extraction accuracy, I wonder what if instead of SFT using RLHFlow, the experiment is conducted using reinforcement learning such as RLHF or RL with verifiable reward (in your case, the reward is simply string matching to the attribute) with the same training framework? As we know RL is a standard post-training method in modern LLM training recipe, and many models went through this step. \n\nMinor:\nLine 142: Person mame --> person name", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1) the experiment setting separates the training/evaluation population, I think this is a carefully determined design.\n2) The paper presents several findings through the experiments which are easy to understand.\n3) The paper propose several methods to improve the chat extraction and verbatim extraction accuracy.\n\nWeakness:\n\n1) The main contributions summarized in the introduction are mainly findings from the experiment. I think these findings have been showed by previous work in a similar knowledge extraction setting (Allen-Zhu & Li, 2024)\n2) In terms of experiment setting, I think the pre-training part is done correctly but the post-training part (SFT) is lacking some key results such as fixed at 10 SFT epoch, mixing chat SFT with pre-training data, and lacking RL approach (see questions)\n3) Overall, i think the scope of this project with limited synthetic data experiment and the contributions provide a fair quality but not significant.", "was_revised": true, "final_justification": "Author clarifies my Q1 but does not show the results initially. After looking at the data related Q1, it is more clear about how SFT stage impact the accuracy.\n\nAuthor claim the Q2 is not practical but this is a common practice used during RLHF (Training language models to follow instructions\nwith human feedback) stage.\n\nAuthor studies only focus on SFT and does not over common practice of RLHF during post-training stage.\n\nAuthor clarifies my W1 and emphasize the unique contribution in this work focuses mainly on the misunderstand of false sense of security after instruction tuning. I think this makes the main message more clear.\n\nOverall, i think the scope of this project with limited synthetic data experiment and the contributions provide a fair quality but not significant. I changed my score from 2 to 3.", "rebuttal": "We thank the reviewer for their feedback. We appreciate the suggestions for extending our experimental setup, which we believe would strengthen the work. We address the main points below.\n\n> The main contributions summarized in the introduction are mainly findings from the experiment. I think these findings have been showed by previous work in a similar knowledge extraction setting (Allen-Zhu & Li, 2024)\n\nWe agree there is an overlap with the knowledge extraction literature. But our contributions as stated are new, specifically:\n- Verbatim memorization (pre-instruction tuning) does not correlate well with chat-based extraction (post-instruction tuning). But removing information from a verbatim sense (see unlearning section, e.g. fig 6) removes chat extractability \n- Information can become \"un-extractable\" verbatim after instruction tuning, yet remain encoded and extractable through other means (chat or after realignment), creating a false sense of privacy.\n- Model quality, not just size, is a key factor in chat-based leakage.\n\nThese key new insights don't just imply that chat extraction differs from verbatim memorization, but that this disconnect creates an issue for privacy auditing. When standard audits (like verbatim extraction or loss-based MIAs) fail after instruction tuning, they can give a false sense of security, even though the information is still very much present and extractable via other means. Our work is the first to systematically demonstrate this failure mode and map the factors that influence it, which we believe is a significant and practical contribution to the privacy community. \n\n> It seems at the SFT stage, it starts from epoch N checkpoint from the pre-train stage and SFT for a fixed 10 epochs. Does the 10 SFT epoch lead to overfitting problem? Can you show the verbatim extraction and chat extraction accuracy at each SFT epoch starting from each pre-train epoch checkpoint?\n\nThat's a fair point. We fixed the SFT epochs to 10 based on preliminary runs where performance plateaued, but we agree that showing the full learning curve would provide a clearer picture and rule out overfitting. We ran the experiments and have the curve ready to be added: the verbatim memorization does not decrease through the 10 epochs of SFT, it stabilises after some instabilities at the beginning. \n\n> What would happen if during the SFT stage, you mix the pre-training data and SFT data together? So the model is trained on both type of data with the training objective. Would this result in better chat extraction accuracy and at the same time preserve the verbatim accuracy?\n\nFrom a practical standpoint, mixing the data would likely be a powerful strategy to improve chat extraction (by reinforcing the knowledge during instruction-tuning) while simultaneously preserving verbatim extraction (by maintaining the next-token prediction objective), but we don’t see it as a realistic thing that practitioners do during SFT. But in the same vein, we show in e.g. figure 5 that “re-aligmenent” restores a lot of verbatim extraction, which leads to a similar conclusion to the one suggested by the reviewer.\n\n> As the paper show it lead to low chat extraction accuracy, I wonder what if instead of SFT using RLHFlow, the experiment is conducted using reinforcement learning such as RLHF...\n\nWe chose SFT as our focus because it's a foundational and universally applied step in aligning chat models, and show in particular that verbatim extraction does not work well after this, which is one practical take-away from our work. We hypothesize that RLHF would also weaken verbatim extraction, but this would need to be tested."}, {"review_id": "zO9lyokeaV", "reviewer": "Reviewer_3ARp", "summary": "This work proposes a novel analysis of memorisation in LLMs. Specifically, authors divide memorisation into two distinct phases: the pre-training mem and the instruction-tuning mem, showing that some of the preexisting understanding of what it means for the LLM to memorise may not represent the whole picture.", "questions": "I've already outlined these in weaknesses and these mostly target a) concerns regarding the BIO dataset, b) novelty of some conclusions compared to prior work and c) reliability of some conclusions given the experimental results.", "limitations": "I've outlined the limitations in the weaknesses section and the mostly concern the dataset composition, reasoning behind some conclusions and the novelty of some insights. Regardless of the above, I believe there is some values in this work as it decomposes the problem that has been extensively studied before into separate sub-problems that have not been given enough attention on their own. So I am willing to increase my score should authors adequately address my comments.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well-written, easy to follow, has a large number of nice experimental results and does indeed propose novel conclusions based on the results of these experiments. To the best of my knowledge, such separation between memorisation has not been previously empirically evaluated before.\n\nWeaknesses:\n\nWhile the work is very interesting and has a number of novel insights, I have my doubts about how much can one read into these results to come to these conclusions. For instance, the argument on 'Model size does not significantly impact chat extraction, while model quality is crucial' - this conclusion is a bit far fetched given the underlying experimental results. Authors have previously evaluated 3 different generations of models (Llama 1-3), where it is not just the size that matters, it is the model itself (the quality - supporting the claim), but more importantly the training dataset. While it is not known which 'exact' data went into this model family during pre-training, we know the relative training corpus sizes: 1T to 15T tokens between them, so I am not sure if you can attribute the difference in extractability simply to 'better models', as there is a higher chance that the model is better, but there is also a higher chance that similar-looking sequence has been memorized due to its recurring appearance in the pre-training dataset (be it part of the sequence or individual tokens next to each other). \n\nSome other conclusions have been previously established already e.g. 'Increased verbatim memorization does not imply more chat extraction' - which we knew from a number of prior works (summary of which can be found in [1]) or that 'IDs are more challenging to encode compared to other attributes' - which we also knew already, hence why these semi-random sequences have previously been used as canaries in most extraction-related prior literature. Same goes for 'Verbatim extraction of unique attributes requires significantly longer unsupervised fine-tuning compared to other attributes' - we already know that duplication of tokens in the training dataset leads to their memorization from [2] and other similar works on unique PII memorization like [3]. So while I see that there is indeed a difference between the stages of training which makes some difference when discussing these insights, the overall principle is the same and we have already known about these phenomena for a while. Note: I do not argue against the validity of these conclusions, but rather that we already knew this and that the impact of these insights is relatively small even if applied to pre-training instead of instruction-tuning.\n\nI also had concerns regarding the dataset composition: as I understand it, this is the same dataset as the one used in Zhu&Li? Because I am not certain how diverse the dataset is given the examples provided in the original manuscript and in this work. Here is my reasoning: Llama models are likely (as is the case with the majority of publicly available LLMs) pre-trained almost exclusively on data oriented towards the US (e.g. city names, states, academic institutions etc.), meaning that a) the model is already predispositioned to be more likely to generate sequence completions that are aligned with the US-based data, b) while the results of extractions hold in this instance, I am not certain if this is likely scalable beyond this geography and c) I've skimmed the original manuscript and it seems to me that some data was generated using Llama models, further amplifying the bias and potentially reproducing some part of the pre-training data. From my understanding none of these concerns were addressed or discussed in the manuscript at all, significantly limiting the potential impact of the findings. I would like the authors to address these through e.g. experiments and extended limitation/discussion sections. \n\n[1] - Usynin, Dmitrii, Moritz Knolle, and Georgios Kaissis. \"Memorisation in machine learning: A survey of results.\" Transactions on Machine Learning Research (2024).\n\n[2] - Carlini, Nicholas, et al. \"Quantifying memorization across neural language models.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[3] - Lukas, Nils, et al. \"Analyzing leakage of personally identifiable information in language models.\" 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023.", "was_revised": true, "final_justification": "As per response: most concerns addressed, slight raise.", "rebuttal": "We thank the reviewer for their positive assessment of the paper's quality and clarity and for the feedback. We appreciate the opportunity to clarify our contributions and address the valid concerns raised.\n\n> While the work is very interesting and has a number of novel insights, I have my doubts about how much can one read into these results to come to these conclusions. For instance, the argument on 'Model size does not significantly impact chat extraction, while model quality is crucial' - this conclusion is a bit far fetched given the underlying experimental results... I am not sure if you can attribute the difference in extractability simply to 'better models', as there is a higher chance that the model is better, but there is also a higher chance that similar-looking sequence has been memorized due to its recurring appearance in the pre-training dataset. [...]\n\nOur claim is not that model size is irrelevant, but rather than \"model quality\" (as measured by standard benchmarks like MMLU) appears to be a better predictor of chat extraction success than parameter count. However we agree that the pre-training data is a significant confounding variable that we cannot control here, but all the biographies are synthetic so we are certain that none were in the pretraining data of any of the models. In particular in Figure 3, we see that it is significantly easier to chat-extract IDs from Llama3-8B than Llama2-13B, whereas there is no reason to believe that Llama3’s pretraining data had a bigger emphasis on random sequences.\n\n> Some other conclusions have been previously established already e.g. 'Increased verbatim memorization does not imply more chat extraction'... or that 'IDs are more challenging to encode compared to other attributes'... So while I see that there is indeed a difference between the stages of training which makes some difference when discussing these insights, the overall principle is the same and we have already known about these phenomena for a while.\n\nWe agree there is a strong link with the knowledge extraction literature, which we have stated in the manuscript. But \n- our contributions are new to the best even within the literature or \n- highlights consequences of already-known facts from a privacy-centric angle. \n\nSpecifically, to the best of our knowledge, we are the first to show that:\n\n- Verbatim memorization (pre-instruction tuning) does not correlate well with chat-based extraction (post-instruction tuning). But removing information from a verbatim sense (see unlearning section, e.g. fig 6) removes chat extractability \n- Information can become \"un-extractable\" verbatim after instruction tuning, yet remain encoded and extractable through other means (chat or after realignment), creating a false sense of privacy.\n- Model quality, not just size, is a key factor in chat-based leakage.\n\nAgain, our contribution also lies in being the first to systematically investigate these effects from a privacy perspective, as the contributions have significant implications for how we audit them for privacy. We will refine our framing to make it clear that our contribution is this comparative, privacy-centric angle.\n\n> I also had concerns regarding the dataset composition... Llama models are likely... pre-trained almost exclusively on data oriented towards the US... I am not certain if this is likely scalable beyond this geography... From my understanding none of these concerns were addressed or discussed in the manuscript at all, significantly limiting the potential impact of the findings.\n\nThe reviewer is correct; our synthetic dataset is US-centric, mainly because it is based on the dataset from Zhu & Li. We fully agree that this raises questions about generalizability. How these extraction dynamics play out with data from geographies or domains that are underrepresented in the pre-training corpus is an open question that we do not tackle. We can add a dedicated discussion to our limitations section to address the dataset's geographic bias and its implications for the scalability of our findings. \n\nHowever the ID attributes that we add for instance are totally neutral, and we respectfully disagree with the reviewer that it significantly limits the generalizability of our claims: we care about the results in relative terms rather than e.g. the absolute extraction rates. Which statement in particular does the reviewer think it impacts?\n\n> I've skimmed the original manuscript and it seems to me that some data was generated using Llama models, further amplifying the bias and potentially reproducing some part of the pre-training data.’’: \n\nWe would like to point out that we use the BIO-S dataset which does not contain any data attributes generated from Llama models, and that the biographies are created by assempling attributes randomly."}]

Author rebuttal:

Meta-review: The main idea of the paper is to challenge the conventional wisdom that verbatim memorization is the primary way LLMs lose privacy. The authors argue that for instruction-tuned LLMs, privacy loss is a more complex issue. They demonstrate through experiments that chat-based extraction is a more effective way to get private information than verbatim extraction. They show that better-performing models are more likely to leak private data through chat, even if they are smaller in size. Moreover, they show that repeated training does not always increase chat extraction, but rephrasing and augmenting data can significantly increase it. Finally, they show that information that is no longer verbatim-extractable after instruction tuning can still be retrieved through a conversational approach.

Overall, the reviewers are in agreement that the paper studies an important problem and makes some interesting contributions. Some concerns were raised by the reviewers around the lack of clarity in writing and some missing experimental details.

The authors' rebuttals effectively addressed most of the reviewers' concerns, with the authors committing to a thorough revision of the paper to improve its structure and clarify key points.

In light of the above, I support the paper for acceptance.
Decision: Accept (poster)