Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning
Year: 2023

Abstract:
Large Language Models (LLMs) are known to memorize significant portions of their training data. Parts of this memorized content have been shown to be extractable by simply querying the model, which poses a privacy risk. We present a novel approach which uses prompttuning to control the extraction rates of memorized content in LLMs. We present two prompt training strategies to increase and decrease extraction rates, which correspond to an attack and a defense, respectively. We demonstrate the effectiveness of our techniques by using models from the GPT-Neo family on a public benchmark. For the 1.3B parameter GPTNeo model, our attack yields a 9.3 percentage point increase in extraction rate compared to our baseline. Our defense can be tuned to achieve different privacy-utility trade-offs by a user-specified hyperparameter. We achieve an extraction rate reduction of up to 97.7% relative to our baseline, with a perplexity increase of 16.9%.

heading: 1 Introduction
text: Pretrained large language models (LLMs; Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2020; Soltan et al., 2022), commonly trained on massive crowd-sourced corpora, have been of much interest in the recent past due to their usage as backbones in state-of-the-art models across multiple downstream NLU tasks. However, they have been shown to memorize significant portions of their training data that can be extracted using appropriately-crafted prompts (Carlini et al., 2020, 2022; Zhang et al., 2021). Such extractions pose a privacy risk to the contributors of the training data.
In this context, methods that allow developers to control the extractability of memorized examples from LLMs are of much value. For example,
∗ Work done while the author was an intern at Amazon; mustafa.ozdayi@utdallas.edu
† perisc@amazon.com
methods that increase extraction rates correspond to attacks in an adversarial setting, and provide developers with the ability to analyze privacy-risk. Methods that decrease extraction rates, referred to as defenses, are useful for protecting against such attacks. Historically, defense methods tend to be compute intensive (Abadi et al., 2016; Dupuy et al., 2021).
In this work, we train continuous soft-prompts (Lester et al. 2021; hereafter referred to simply as prompts) and leverage them as a way of passing an external signal into an LLM, to control the extraction of memorized data. We freeze the model weights, and only use the trained prompt to control the generation. First, we train prompts in an attack setting and study the extent of extractable memorized content in our models. Second, we explore a defense setting where we create prompts that reduce extraction rates and achieve different privacy-utility trade-offs, via a user-specified hyperparameter. Since the original model weights are frozen in both these settings, our methods are compute efficient across the board.
To the best of our knowledge, our work is the first to adapt the use of instructive prompts for the analysis and mitigation of privacy in LLMs. We have released the code developed for our experiments1.

heading: 2 Background and Related Work
text: Previous work has shown that LLMs display memorization and has explored a range of methods that quantify extractability (Carlini et al., 2018, 2020, 2022). Differentially-private training (Dwork, 2006; Abadi et al., 2016) is a popular method that has been used to mitigate this risk. However, it tends to reduce model utility and requires retraining of the LLM, which might not be feasible due to heavy computational burden.
1https://github.com/amazon-science/controlling-llmmemorization
1512
The use of instructive prompts for language models has been extensively researched, including use during pretraining (Raffel et al., 2020), as a second stage of training (Sanh et al., 2022; Wei et al., 2021), and during inference to guide model output (Brown et al., 2020). Within the third category, in order to improve upon manual prompt engineering researchers have implemented methods to learn discrete natural language prompts (Shin et al., 2020), to mine them (Jiang et al., 2020), or, neglecting natural language, to learn continuous prompts (Li and Liang, 2021; Lester et al., 2021).
Our work leverages continuous prompts as a way of passing an external signal to a model to trigger a desired model behavior (i.e., less or more memorized data in open language generation, which map to an extraction attack and defense, respectively).

heading: 3 Method
text: Prompt-tuning requires the prepending of a prompt to the prefix embedding and access to the training loss (see Figure 1). Given these constraints, we explore a white-box attack where the adversary has access to the target model parameters, and a blackbox defense where the adversary interacts with the target model via an API. We therefore do not test our defense against our own attack.
Let [prefix || suffix] be a sequence in the training
set where the prefix is of length k tokens. Carlini et al. (2022) defined a suffix to be k-extractable if the model generates the suffix exactly, after being prompted with its the corresponding lengthk prefix. Our white-box attack aims to increase the number of k-extractable sequences, while our black-box defense aims to reduce the number of k-extractable sequences that can be extracted by an adversary who submits prefixes via an API.

heading: 3.1 Attack
text: In the attack setting, we assume that the adversary has a set of [ prefix || suffix ] sequences Strain, sampled from the training set of the target model. Their goal is to extract the suffixes corresponding to a disjoint set of prefixes, denoted by Stest2.
To do so, the adversary first initializes a prompt: a continuous set of l × e parameters where e is the embedding size of the model, and l is the length of the prompt, a hyperparameter decided by the adversary. The prompt is trained over Strain to facilitate the correct generation of suffixes. To do this, we first prepend the prompt to the embedding of the prefix and pass the joint embedding through the model for generation. We then minimize the loss objective (see below) with respect to the prompt while keeping the parameters of the model frozen.
We explore two loss objectives. The first is causal language modeling (hereafter referred to as CLM), where we minimize the cross-entropy loss over the entire sequence (Radford et al., 2019). In the second, the prompt is optimized by minimizing the cross entropy loss of only the suffixes, given the prefixes. Here, the training is aligned with our inference task such that during training the model is penalized only on the suffix tokens; hence we refer to it as aligned CLM. During inference, the learned prompt is prepended to each embedding of the prefixes in Stest, and the joint embedding is passed to the model for generation (see Figure 1).

heading: 3.2 Defense
text: In the defense setting, the defender (API owner) trains the prompt, and prepends it to the incoming prefixes before passing them to the model. Our algorithm is inspired by machine-unlearning literature (Halimi et al., 2022), and defenses against membership inference and backdoor attacks (Chen et al., 2022; Ozdayi et al., 2021). We introduce a
2For simplicity, we assume all prefixes are k-length. This can easily be ensured by padding or truncating different length prefixes if needed in a real-world setting.
hyperparameter named learning threshold denoted by θ. During prompt training (see Section 3.1), when loss is less than θ we do gradient ascent to penalize the prompt. If the loss is greater than θ, we perform gradient descent with respect to the prompt as usual. Training is stopped once the average epoch loss is equal or above θ. This allows us to increase training loss in a controlled manner and stabilize it around θ. Through this process, we can achieve various privacy-utility trade-offs efficiently without re-training any part of the model. To explore θ, we set the initial value to be slightly above the model training loss and increase in steps of 0.25 until desired performance is achieved.

heading: 4 Experiments
text: For our experiments, we use the 125M and 1.3B parameter variants of the GPT-Neo models (Black et al., 2021). These are public, decoder-only transformer models (Vaswani et al., 2017) trained using CLM on the Pile dataset (Gao et al., 2020). We extract Strain and Stest from the Language Model Extraction Benchmark dataset (Google-Research). This dataset contains 15k sequences sampled from the training split of the Pile where each sequence is partitioned into a prefix and suffix. In the default evaluation setting, both prefix and suffix consist of 50 tokens. We ensure a random train/test split of 14k/1k samples.
Our evaluation metric of choice is Exact extraction rate which is the fraction of correctly generated suffixes (i.e., all tokens of the generated suffix match with ground-truth suffix) over the test set. We additionally discuss fractional extraction rate and present results in Appendix A. As a baseline, we use the attack analyzed in Carlini et al. (2022), which consists of feeding the prefixes to the model, and generating suffixes with greedy decoding. This is the only extraction attack for this setting apart from our work, to the best of our knowledge. Our training setup is discussed in Appendix B. All experiments are repeated over 5 runs with a new random train/test split in each run.

heading: 4.1 Attack
text: We explore the performance of our attack across several dimensions: prompt length, suffix size, prefix size, and beam size. We use greedy-decoding in all cases, except the beam size experiments.
Prompt Length First, we explore prompt length in the context of the default setting (prefix and suf-
fix consist of 50 tokens; Figures 2-A1 and 2-A2). We note that prompts tuned with both CLM and aligned CLM provide improvements over the baseline in all cases, with aligned CLM providing the best performance. Given this, we train prompts using the aligned CLM objective for all other experiments, including our defense.
With aligned CLM, we achieve the highest extraction rates of 25.8% and 54.3% for the 125M and 1.3B models, respectively (an improvement of 8.9 and 9.3 percentage points, respectively), with a 100 token prompt (blue line). We observe that extraction rates increase with prompt length and tend to saturate after prompt length 100. Over-fitting was ruled out as a potential cause of saturation as there is no increase in test loss observed during training. This suggests that there is a max limit on the parameter count in the prompt that might add value for extraction purposes given our objective. We note that more sophisticated training strategies (designing better loss functions, better prompt initialization etc.) might yield better extraction rates.
Suffix Size Next, we fix the prefix size to 50 and vary the suffix size. As shown in Figures 2-B1 and 2-B2, extraction rates decrease roughly exponentially with suffix size. We note that as suffix size increases, longer prompts (≥ 20) provide greater improvements over the baseline. For example, with a prompt length of 100 (blue line) using the 1.3B model, at suffix size 5 we observe an extraction rate increase of 5.3 percentage points. Whereas at suffix size 50, the increase is 9.3 percentage points.
Prefix Size Next, we fix the suffix size to 50 and vary the prefix size. As shown in Figures 2-C1 and 2-C2, extraction rates increase roughly logarithmically (as in Carlini et al. 2022). Contrary to suffix size, we observe that the gaps between baseline and attacks decrease with increasing prefix size. This suggests that our attack stands to benefit a less informed adversary (small prefix sizes) when compared to the baseline.
Beam Decoding Finally, we utilize the default setting with prefix and suffix sizes at 50 tokens and vary the beam size (beam size=1 corresponds to greedy decoding). The results are shown in Figures 2-D1 and 2-D2. We observe that extraction rates increase across the board when increasing beam size from 1 to 5. However, improvements tend to plateau or oscillate when beam size is greater than 5. The 1.3B model benefits more
from increasing beam size achieving the highest extraction rate of 61.4%, at a beam size of 20 (with a prompt length of 150). The highest extraction rate achieved for the 125M model was 28.3% at a beam size of 15 (with a prompt length of 100).

heading: 4.2 Defense
text: Finally, we evaluate the privacy-utility trade-off of our black-box defense. As mentioned in Section 3, our defense is designed for a black-box adversary, and cannot be tested against our white-box attack.
Therefore, we utilize the baseline attack (Section 4) to quantify privacy. We note that longer prompts did not add value in a defense setting, so we resort to using a prompt of length 1. We utilize perplexity (PPL) on generated suffixes, to quantify the utility of the model in addition to using exact extraction rate as in Section 3.1. To measure PPL, we use a random subset of 1k sequences sampled from the test split of the Pile, ensuring that PPL is measured on data unseen by the model. We also compare our metrics with those of similar sized models that were not trained on the Pile dataset (GPT2 models). Our premise here is that better performance in terms of privacy and utility, when compared to an out-ofdomain model of similar size, would mean that our defense mechanism is of value to an API owner.
In Table 1, we display our results obtained using the default evaluation setting (prefix and suffix comprise of 50 tokens). Our defense achieves lower extraction rates with competitive PPL values. For the 125M model, we achieve an exact extraction rate reduction of 99.4% relative to baseline with a PPL increase of 25.3% at θ = 1.75. For the 1.3B model, the extraction rate is reduced by 97.7% relative to baseline with a PPL increase of 16.9% at θ = 1. The ability to achieve lower extraction rates with lower PPL values as measured against the GPT2 models of the corresponding size, provides evidence that our defense is effective.

heading: 5 Conclusion
text: We present the first known effort to leverage prompt-tuning to control the extractability of memorized data from LLMs in an open language generation task. We develop a novel data extraction attack and defense, and illustrate their performance under various settings. Our attack consistently outperforms the baseline in terms of exact extraction rate. Our defense provides competitive privacy-utility trade-offs and would prove beneficial to API owners with model trained on sensitive content. These results are achieved efficiently, without any change to the original model weights. We details avenues of future work in Appendix C

heading: 6 Limitations
text: We briefly mention some limitations of our work. First, we have only used a single dataset, and a single model family in our experiments. This is mainly due to the fact that the benchmark we use is the only publicly available dataset at this time to the best of our knowledge. We also solely focused on extraction metrics, but did not do a deeper analysis on the extracted sequences. A fine-grained analysis of extracted sequences could yield important insights for understanding memorization and extraction in LLMs. Similarly, we also did not analyze what our prompts converge to, and whether they yield explainable prompts at the time of converge. Such analysis can provide better insights as to why, for example, training prompts with aligned CLM performs better that the basic CLM setting. Finally, we believe the evaluation of our defense could be improved further by measuring other utility metrics (e.g., accuracy) on downstream tasks.

heading: 7 Ethical Considerations
text: We leverage prompt-tuning to control the extractability of memorized data from LLMs in an open language generation task and explore two settings; an attack and a defense. We acknowledge that our attack methodology could be misused by an adversary with white-box access to extract memorized private information from a target large language model. Our goal is to raise awareness in the community to the possibility and severity of this nature of attack. We hope that developers, armed with this knowledge, can use relevant defense mechanisms to avoid such potential misuse.

heading: Acknowledgements
text: The authors would like to thank Wael Hamza for helpful discussions on this topic and Stephen Rawls for help with securing the GPU instances that were required for experimentation.

heading: A Fractional Extraction Rate Results
text: Fractional extraction rate is the fraction of generated tokens that are both correct and in the right position, over the dataset (see lower section of Figure 2). Our reason to measure this metric is to provide a more detailed assessment of risks associated with extraction. Exact extraction rate is particularly important in cases where the attacker requires an exact match in order for the extraction to be of use; a good example is the case of extracting a credit card number. In such cases, even getting a few tokens incorrect will render the attack useless. However, when the attacker cares more about the meaning of the extracted sequences, fractional extraction rate can be a better metric to assess the risk. This is because a human might be able to infer the correct meaning of the sequence even when few tokens are wrong.
The results related to this metric are shown in Figure 3. Comparing these results with the exact extraction rate results (Figure 2), we observe the same trends across all of our experiment. We note that the same shared trends are observed in the case of our defense. In this case the fractional extraction rate results are tabulated in Table 2.

heading: B Training Setup
text: Our soft-prompts are initialized to random word embeddings as described in Lester et al. (2021). We use a batch size of 128 and an Adam optimizer (Kingma and Ba, 2014) with a learning rate of 5e − 4. For the attack setting, the prompts are trained for 15 epochs. In the defense case, the prompts are trained until training loss stabilizes around the specified θ value (as described in Section 3.2), which happens within 2-3 epochs in our experiments.
We use a Pytorch (Paszke et al., 2019) implementation where we leverage the HuggingFace Accelerate (HF) and DeepSpeed (Rasley et al., 2020) libraries to handle distributed training over 8 GPUs with fp16 mixed precision. On a p3dn.24xlarge instance, the average attack prompt training time
was 0.9 hours per prompt while the average defense prompt training time was 0.02 hours per prompt.

heading: C Future work
text: We have several avenues that we would like to explore in the context of future work. We envision that more sophisticated training strategies might yield better extraction rates in our attack setting (designing better loss objectives, better initialization of soft-prompts etc.) and we would like to explore this further.
We would like to explore different prompt learning algorithms such as other parameter-efficient training methods (Li and Liang, 2021; Hu et al., 2021), and hard-prompt learning methods (Wallace et al., 2019), in order to conduct a more robust analysis of extraction rates.
We would like to test the transferability of trained prompts across different models and datasets.
Finally, we would like to combine our defense with other existing defenses such as those applied at training time (e.g. versions of differentially private stochastic gradient descent; Abadi et al. 2016; Dupuy et al. 2021) or those applied at decoding stage (e.g., differentially private decoding; Majmudar et al. 2022). The goal would be to achieve better privacy-utility trade-offs under a combination of such defenses.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: See Section 6

heading: 3 A2. Did you discuss any potential risks of your work?
text: See Ethical Considerations under Section 7

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: See Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? See Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: We’ve cited the models. We cited the dataset in the right way to the best of our knowledge.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? These are publicly available models and data and so their licenses are in accordance with our work.

heading: B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. The artifacts we use have been used by multiple publications for the same purpose as ours and are in accordance with their intended use. We do not create any model or data related artifacts.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? This data is part of the Pile dataset (Gao et al. 2020) that has seen much study in previous publications in the context of large language model training. Therefore, we do not take special steps to discuss this.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? The dataset we use been discussed in Gao et al. 2020 citation and an interested reader will be able to gather information here. The models are also discussed in the Black et al. 2021 citation.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. See Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? See Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? See Section 4 and Training set up in Appendix B

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? For training we utilize a set of parameters that have been commonly used in previous studies. For theta (a hyper parameter that we introduce) see Table 1 for theta values that we explore. And Appendix B for experimental setup.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We show errorbars in all our plots, See Figure 2 and 3. We also report mean and stdev based on 5 runs.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We clearly define our metrics in Section 4 and Appendix A. They do not use existing packages. We do not do any pre-processing or normalization.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.

heading: D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Not applicable. Left blank.

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Not applicable. Left blank.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Not applicable. Left blank.

heading: Future Work
text: We have several avenues that we would like to explore in the context of future work. We envision that more sophisticated training strategies might yield better extraction rates in our attack setting (designing better loss objectives, better initialization of soft-prompts etc. ) and we would like to explore this further. We would like to explore different prompt learning algorithms such as other parameter-efficient training methods (Li and Liang, 2021; Hu et al. , 2021), and hard-prompt learning methods (Wallace et al. , 2019), in order to conduct a more robust analysis of extraction rates. We would like to test the transferability of trained prompts across different models and datasets. Finally, we would like to combine our defense with other existing defenses such as those applied at training time (e. g. versions of differentially private stochastic gradient descent; Abadi et al. 2016; Dupuy et al. 2021) or those applied at decoding stage (e. g. , differentially private decoding; Majmudar et al. 2022). The goal would be to achieve better privacy-utility trade-offs under a combination of such defenses. ACL 2023 Responsible NLP Checklist