Privacy Implications of Retrieval-Based Language Models
Authors: Yangsibo Huang
Samyak Gupta
Zexuan Zhong
Kai Li
Danqi Chen
Venue: EMNLP 2023 Main (2023)
Keywords: Privacy
Retrieval-based language models

Abstract: Retrieval-based language models (LMs) have demonstrated improved interpretability, factuality, and adaptability compared to their parametric counterparts by incorporating retrieved text from external datastores. While it is well known that parametric models are prone to leaking private data, it remains unclear how the addition of a retrieval datastore impacts model privacy. In this work, we present the first study of privacy risks in retrieval-based LMs, particularly $k$NN-LMs. Our goal is to explore the optimal design and training procedure in domains where privacy is of concern, aiming to strike a balance between utility and privacy. Crucially, we find that $k$NN-LMs are more susceptible to leaking private information from their private datastore than parametric models. We further explore mitigations of privacy risks: When privacy information is targeted and readily detected in the text, we find that a simple sanitization step would eliminate the risks while decoupling query and key encoders achieves an even better utility-privacy trade-off. Otherwise, we consider strategies of mixing public and private data in both datastore and encoder training. While these methods offer modest improvements, they leave considerable room for future work. Together, our findings provide insights for practitioners to better understand and mitigate privacy risks in retrieval-based LMs.

Privacy Implications of Retrieval-Based Language Models
Yangsibo Huang Samyak Gupta Princeton University
Zexuan Zhong Kai Li Danqi Chen
yangsibo@prinnctoon.edd (samyakg, zzhong, 1i dasggic)@css.prineeooo princeton. edu
Abstract
Retrieval-based LM (APlaccess only
Attacker
Retrieval- betreeval-based language models (LMs) have demonstrated improved interpretability, factu- ality, and udaptability compared to their para- metric counterparts by incorporating retrieved text from external datastores. While is well known that parametric models are prone to leak- ing private data, it remains unclear how the ad- dition of a retrieval datastore impacts model privacy. In this work, present the first study of privacy risks in retrieval-based LMs, particu- larly &NN-LMs. timal design and training procedure in domains where privacy is of concern, aiming to strike balance between utility and privacy. Crucially, we find that KNN-LMs are more susceptible to leaking private information from their pri- vate datastore than parametric models. We fur- ther explore mitigations of privacy risks: When privacy information is targeted and readily de- tected in the text, we find that simple sani- tization step would eliminate the risks while decoupling query and key encoders achieves even better utility-privacy trade-off. Other- wise, we consider strategies of mixing public and private data both datastore and encoder training. While these methods offer modest im- provements, they leave considerable room future work. Together, sights for practitioners to hetter understand and mitigate privacy risks in retrieval-based LMs!.
Datastore
Reconstructed
Data Extraction Allack
ehall:
muile muilirppoc opbos con nod Tenegeaarr com
Prompt
Encoders
Responsee
pullic
Enc
Phoonee:
780
goal explore the op-
Figure 1: Retrieval based language models (e.g.. KNN- LMs) comprise encoders and the datastore as their key components. When tailoring retrieval-based language model for privacy-sensitig task, both components may utilize private data. However, a malicious user possessing API access the model exploit data extraction attack in order to reconstruct sensitive infor- mation. This study explores severity such attacks and suggests novel strategies mitigate them.
text passages that are most relevant to the prompt provided the model. These retrieved results are then utilized as additional information when gener- ating the model's 's response to the prompt. Retrieval- based language models offer promising prospects in terms enhancing interpretability. factuality, and adaptability.
findings provide in-
However, in privacy-sensitive applications, util- ity usually comes at the cost of privacy leakage. Recent work has shown that large language models are prone to memorizing (Thakkar et al., 2021; Zhang et al., 2021) specific training datapoints, such as personally identifying or otherwise sen- sitive information. These sensitive datapoints can subsequently be extracted from trained model through a variety of techniques (Carlini al., 2019, 2021; Lehman al., 2021), known data ex- traction attacks. While the threat of training data memorization on model privacy has been studied for parametric language models, there is a lack
1 Introduction
Retrieval-based language models (Khandelwal et al., 2020; Borgcaud al., 2022; Izacard al., 2022; Zhong al., 2022; Min al., 2023) generate text distributions by referencing both the parame- ters of the underlying language model and the in- formation retrieved from a datastore of text. Specif- ically, the retrieval process involves accessing a pre-defined datastore to retrieve set of tokens or
'Our code is available at Princeton- Trinceton-SyaaLL/WWL.Y Stwpsssss////MMiiaaaaaaa

---

of evidence regarding the privacy implications of retrieval-based language models, especially how the use of external datastore would impact privacy.
prompts to the language model). Through our ex- perimental analysis, we demonstrate that this de- sign approach offers increased flexibility in striking a balance between privacy and model performance.
In this work, we present the first study of pri- vacy risks in retrieval -base language models, with focus on the nearest neighbor language models (kNN-LMs) (Khandelwal al., 2020), which have been extensively studied in the literature (He et al., 2021: Zhong et al., 2022; Shi et al., 2022a; Xu et al.. 2023)². In particular, we are interested in un- derstanding kNN-LMs' privacy risks in real-world scenarios where they are deployed via an API to the general public (Figure 1). We consider scenario in which a model creator has private, domain- specific dataset that improves model performance on domain -omain--pecific tasks, but may also contain sen- sitive information that should not be revealed; This data can be used to train encoders or stored in the datastore. In such a scenario, the model creator must find a balance between utilizing their private data to enhance model performance and protecting sensitive information.
The second is more challenging scenario where the private information is untargeted, making it im- practical remove from the data (Section 6). To address this issue, we explore the possibility of con- structing the datastore using public datapoints. We also consider training the encoders of the kNN-LM model using a combination of public and private datapoints to minimize the distribution differences between the public data stored in the datastore and the private data used during inference. Despite modest improvements from the methods we ex- plored, the mitigation of untargeted attacks remains challenging and there considerable room for fu- ture work. We hope our findings provide insights for practitioners to better understand and mitigate privacy risks in retrieval-based LMs.
2 Background
We begin our investigation by examining a sit- uation where the creator of the model only adds private data to the retrieval datastore during infer- ence, suggested by Borgeaud al. (2022). Our findings indicate that while this approach enhances utility, it introduces an elevated privacy risk to the private data compared to parametric language mod- els (Section 4), and adversarial users could violate the confidentiality of the datastore by recovering sensitive datapoints. Therefore, it is vital for the model creator to refrain from storing sensitive in- formation in the datastore.
In this section, we first review the key compo- nents of kNN-LMs (Section 2.1). Then, we discuss the data extraction attacks for language models (Section 2.2). These aspects lay a foundation for the subsequent exploration and analysis of privacy risks related to kNN-LMs.
2.1 Nearest Neighbor Language Models
A ANN-LM (Khandelwal al., 2020) augments the standard language model with a datastore from which it can retrieve tokens to improve perfor- mance. We use the term "query" to denote the prompt provided to a ANN-LM. This query is en- coded by the encoder Encq as it is passed into the kNN-LM. The term "key" is used to denote the tokens in the datastore and it is encoded by the encoder Enck.
We further explore mitigation strategies for kNN- LMs in two different scenarios. The first is where private information is targeted, i.c., can be casily identified and removed (Section 5). We explore enhancing the privacy of kNN-LMs by eliminat- ing privacy-sensitive text segments from both the datastore and the encoder's training process. This approach effectively eliminates the targeted privacy risks while resulting in minimal loss of utility. We then explore a finer level of control over private in- formation by employing distinct encoders for keys (i.e., texts stored in the datastore) and queries (i.e.,
Encoders Given a vocabulary V, the encoder Enck or Eneq performs the task of mapping a given key or query e E V to a fixed-length vec- tor representation. Typically, this encoding pro- cess is accomplished through trained language model, where Enck (c) or Encq(c) represents the vector hidden representation obtained from the out- put layer of the language model when provided with the input c. Although in the default &NN-LMs, Encg and Enco are commonly identical functions, we explore different options in the work.
*Other retrieval-based language models such as RETRO (Borgeaud et ul., 2022) and Atlas (Izacard et al., 2022) have sigmificantly different architectures and the findings from our investigation may not necessarily apply to these models. Investigation of these models will be left as future work.

---

Datastore
The datastore, {(nek (c), w:)). is key-value store generated by running the encoder (nck(.) over a corpus of text; Each key is the vector representation Enck(ca) for some context G с V and each value w, E V is the ground-truth next word for the leftward context index is then constructed based on the key-value store to enable retrieval.
While previous research has successfully high- lighted the risks associated with data extraction in parametric language models, there remains a no- table gap in our understanding of the risks (and any potential benefits) pertaining to retrieval-based language models like ANN-LMs. This study aims to address this gap and provide insights into the subject matter.
A search
Inference At inference time, when predicting the next token for query V*, the model queries the datastore with encoded query Enco(a) to retrieve æ's k-nearest neighbors N& according to a distance function )². Then the model computes a softmax over the (negative) distances, which gives PkNN (z)z), distribution over the next
3 Problem Setup
In this section, we formally describe our problem setup (Section 3.1) and privacy mcasurements (Sec- tion 3.2). We then detail our evaluation setup (Sec- tion 3.3).
token, in proportional to:
3.1 Problem Definition
(- d (EEnc((ci) Encq(z))
Σ
ly=w; exp
We consider a scenario where a service provider (e.g. a financial institution) aims to enhance its cus- tomer experience by developing a ÀNN-LM and deploying it as an API service. Note that the devel- opment of ANN-LMs intended solely for personal use (e.g., constructing KNN-LM email autocom- pleter by combining public LM with a private email datastore) falls outside the scope of our study because it does not involve any attack channels that could be exploited by potential attackers. We as- sume that the service provider possesses its own private data (D (Dprivtee specific to its domain, in addition to publicly available data (Ppublic).
t
(c.w))ENe
where is temperature parameter, and k a hyper-parameter that controls the number of re- trieved neighbors. The prediction is then interpo- lated with PLM. the prediction from the original LM: p(y(z)) = (y)))( + (1 {)pLM((lZ). where À is an interpolation coefficient.
2.2 Data Extraction Attacks
Prior work (Carlini al., 2021) demonstrates that an attacker can extract private datapoints from the training set of a learned language model. The ex- istence of such an attack poses a clear and alarm- ing threat to the confidentiality of sensitive train- ing data, potentially jeopardizing deployment in real-world scenarios (e.g., Gmail's autocomplete model (Chen et al., 2019), which is trained on private user emails). Our definition of privacy leakage adopts the standard definition of Carlini et al. (2021)). Specifically, a string is extractable from an &NN-LM fo if there exists a prefix e such that: s + arg max,' fa (ss((e) cc). Namely, the model generates as the most likely continuation when prompted with some prefix c.
We identify two key design choices which im- pact the quality and privacy of such a deployed service. First, the service provider chooses which data to be included in its datastore, and this may be public data (D (Ppublic), private data (D (Prrivate), or mix of both. Second, they choose whether to encoders that are pre-trained on publicly available data (Encpublic), or further finetuned on the private data (Encprivate). We posit that careful considera- tion of these design choices needed to establish a balance between privacy preservation and utility.
The service provider in such a scenario is con- cerned with making useful API, while keeping their private data hidden from malicious users or attackers. Hence, the service provider's objective is to attain high level of utility (as measured by perplexity) on held-out of D private while si- multaneously minimizing the disclosure of private information. We quantify the metrics we consider for privacy in Section 3.2.
The attack consists of two main steps: 1) gener- ating candidate reconstructions by prompting the trained models, and 2) sorting the generated candi- dates based on a score that indicates the likelihood of being a memorized text. Further details about the attack can be found in Appendix A.
=d(---) is usually the squared &2 distance.

---

3.2 Privacy Measurements
Untargeted attacks The untargeted attack is the case where the attacker aims recover the entire training example, rather than a specific segment of text. Such attacks can potentially lead to the theft of valuable private training data. We adopt the attack proposed by Carlini et al. (2021) the untargeted attack, which is described in detail in Appendix A.I. For evaluation, we measure the similarity between the reconstructed text and the original private text:
We now describe how we evaluate the risk of data extraction attack within the scenario descrihed ear- lier in Section 3.1.
Threat model We assume that the service provider deploys ANN-LM with API access to p(y(x). This API provides the attacker with the capability to compute perplexity, conduct text com- pletion, and perform other relevant tasks. However, it's important to note that the attacker is restricted from accessing the internal parameters or the data- store of the deployed model.
We firstly sort the reconstruction candidates based on the membership metrics defined in Appendix A, and only keep the top-n candi- dates {ci)==>
Our study considers two types of privacy risks, each associated with a particular type of attack:
Targeted attacks We define targeted risk as a privacy risk that can be directly associated with a segment of text (c.g., personal identifiers such as addresses and telephone numbers.) and propose the targeted attack. The significance of a targeted attack becomes apparent when considering that targeted risks have been explicitly addressed in various privacy regulations (Centers for Medicare & Medicaid Services, 1996; California State Leg- islature 2018). A targeted attacker's goal to extract that certain segment of text, In our study, we focus on the extraction of Personal Identifiable Information (PII), including email addresses, tele- phone numbers, and URLs. To tailor the extraction attack to recover text segments such as PIIs rather than the entire training text, customize the at- tack prompts based on the type of information be extracted. Specifically, we gather common preced- ing context for telephone numbers, email addresses, and URLs, and use them as prompts. Appendix B provides example prompts we use in the attack. For evaluation, we measure how many private PIIs of each category have been successfully reconstructed by the attacker:
For each candidate c, we then find the closest example in the private dataset p; and compute the ROUGE-L score (Lin, 2004) between G; and p: If the score is higher than 0.5, we mark the candidate as a good reconstruction. The ROUGE-L measures the longest common subsequence (LCS) between the attack result and the ground truth, thus representing the fi- delity of the attacker results. The ROUGE-L score has been used to measure the reconstruc- tion fidelity in previous work (Deng et al., 2021; Balunovic et al., 2022; Gupta al., 2022).
Note that the attack's performance evaluation employs the private dataset following established reconstruction attack practices, the attack itself never utilizes this dataset.
3.3 Evaluation Setup
Our main evaluation uses the Enron Email dataset (Klimt and Yang, 2004) as the private dataset 2 Prrivates which contains around 500,000 emails generated by employees of the Enron Corpo- ration (see Table for examples). We specifically chose this dataset due to its inclusion of Plls, which enables us to evaluate targeted attacks effectively. We also incorporate the Medical Transcriptions dataset as an additional dataset for the evaluation of untargeted attacks, and further information re- garding this dataset can be found in Appendix D. We use the WikiText-103 dataset (Merity et al.. 2017) as Dpublic-
We firstly detect all unique personal identifiers in the private dataset, denoted as |
We then sort the reconstruction candidates based on the membership metries defined in Appendix A, and only keep the top-n candi- dates {ci)_
Finally, detect {p.)/= the unique PIIs in the top-n candidates, and then count Het) {p:) ( namely how many origi- nal PIIs have been successfully reconstructed by the attack. A larger number means higher leakage of private PlIs.
We pre-process the Enron Email dataset by re- taining only the email body (see Table I for exam-
"Note that Carlini et al. (2021) also focus on untargoted attacks hut they adopt manual evaluation.

---

Example #1: Request ID: 0000000000* Request Create Date : //* 8:27:00 AM Requested For: ***3 ennon..oom Resource Name: Market Data Bloomberg Resource Type: : Applications
Example #2: You reach me over the weekend and in the evening at either - or -.000
Example #3: This would give you a total loan of ****, total cost of for equity required of ****.
Example #4: Winter 2001 baseload traded as high as ** pounds a megawatt-hour and as low as **** pounds megawatt-hour before closing at **** pounds megawatt-hour, **e* pence lower than Friday.
Example #5: everything is correct except for password: **.0
Table Examples from the Enron Email dataset. We've anonymized any identifiable information protect against potential privacy breaches inherent in the dataset.
TARGETED ATTACK
UNTARGETED ATTACK
MODEL
EVAL. PPL
TOTAL
PHONE
EMAIL
URL
GOOD RECON
(PARAMETRIC LM) EnCpublic
30.28
0
0
0
0
(PARAMETRIC LM) EnCprivate
20.63
28
11
14
3
620
(KNN-LM) Encpublic W/ Prrivatt
18.41
35
11
16
8
591
(ANN-LM) Encprivate W/ Prrivase
16.12
54
25
23
6
656
Table 2:
Perplexity and data extraction risks for various model configurations on the Enron Email dataset. The configuration with the highest leakage is emphasized in red;; the configuration with the lowest leakage is highlighted computed using the top 1000 candidates for the targeted attack and using the
in green. Privacy measurements top 5000 candidates for the untargeted attack. "Good Recon" refers to reconstructions that achieve a ROUGE-L score greater than 0.5 when compared to the ground truth. Table in Appendix presents similar findings on the Medical Transcriptions dataset.
ple datapoints). We then use regular expressions to identify and extract three types of personal identi- fiers for the use of the targeted attack: telephone numbers, email addresses, and URLs. The statis- tics for these personal identifiers can be found in Appendix B. We use the GPT-2 base model (Rad- ford et al., 2019) Encpublic; and finetune it on the Enron Email dataset as EnCprivate.
sured by metrics defined in Section 3.2) and utility (measured by perplexity) in kNN-LMs.
We are particularly interested in three scenar- ios: utilizing only Encpublic (the publicly pre- trained language model), utilizing only pricvrvvate (the model fine-tuned from Encpublic using private data), and utilizing Encpublic with D Prrivate (the combination of the public model with the private datastore). As shown in Table 2, using Enncpbblic alone results in very poor utility performance but poses minimal risk of data extraction from the pri- vate domain, as it has not been exposed to pri- vate datapoints. Using Encprivate enhances utility (perplexity improves from 30.28 to 20.63) but in- creases the risk of data extraction.
During inference, the model retrieves k nearest neighbors according to the squared E2 distance, nor- malizes their distribution over the next word with softmax using a temperature value of 1, and then uses an interpolation factor of À to combine PkNN and PLM. For each model configuration, we search k and À on a held-out validation set for the best model perplexity, and then run the inference on the evaluation set (see Appendix B for details). Privacy measurements are computed using the top 1000 candidates for the targeted attack and using the top 5000 candidates for the untargeted attack.
When it comes to ANN-LMs, incorporating a private datastore (Dprivate)) with a public model (Encpublie) yields even greater utility compared to relying solely on the fine-tuned model (Finprrvvate). However, this utility improvement also comes at the expense of increased privacy leakage. These findings suggest that the privacy concern stemming from the private datastore outweighs that resulting from the privately fine-tuned model, indicating a lack of robust privacy protection in the design of kNN-LMs. Additionally, we note that the combina- tion of Encprivate and D Pprivate achieves the highest utility but also incurs the highest privacy cost.
4 Privacy-Utility of l-N--LMs with A Private Datastore
This section presents our investigation of whether the addition of private data to the retrieval data- store during inference is an effective method for achieving a good trade-off between privacy (mea-

---

SANITIZED ENCODER?
SANITIZATION APPROACH
EVAL. PPL.
TOTAL
PHONE
EMAIL
URL
Enck
Eneo
NONE
X
16.12
54
25
23
/
16.83
0
0
REPLACED W/< [endfftext]
X
/
16.24
0
0
X
16.32
16
6
6
16.51
REPLACED W/ DUMMY PII
16.29
0
X
16.16
22
5
15
16.38
REPLACED RANDOM PUBLIC PII
/
16.29
1
16.20
13
Table 3: Perplexity and extraction risk of &NN-LMs with different combinations of key encoder (Enck) and query encoder (Encq) on Enron Email dataset, under different sanitization approaches. Privacy measurements are computed using the top 5000 candidates. For each sanitization approach, the configuration with the highest leakage is emphasized in red, and the configuration with the lowest leakage is highlighted in green; the hest utility under sanitization is highlighted in boldface. Sanitizing Eucq crucial eliminating the targeted risk.
5 Mitigations Against Targeted Risks
as the basis for establishing the verbatim-level defi- nition of "privacy leakage". For example, HIPAA defines 18 identifiers that are considered person- ally identifiable information (PII), including names, addresses, phone numbers, etc.
Our previous findings indicate that the personal- ization of ANN-LMs with private datastore is more susceptible data extraction attacks com- pared to fine-tuning a parametric LM with private data. At the same time, leveraging private data offers substantial utility improvements. Is there a more effective way to leverage private data in order to achieve better balance between privacy and utility in kNN-LMs? In this section we fo- addressing privacy leakage in the context of largeted attacks (see definition in Section 3.2), where the private information can be readily de- tected from text. We consider several approaches to tackle these challenges in Section 5.1 and Sec- tion 5.2, and present the results in Section 5.3. We also investigate the effect of hyper-parameter in Section 5.4.
We propose the following three options for sani- tization:
Replacement with < |endoftext] >: re- place each privacy-sensitive phrase with the [endoftext] > token;
Replacement with dummy text: replace each privacy-sensitive phrase with fixed dummy phrase based on its type. For instance, if tele- phone numbers are sensitive, they can be re- placed with "123-456-789" and
Replacement with public data: replace each privacy-sensitivy phrase with randomly lected public phrase of a similar type. An ex- ample is replace each phone number with public phone number on the Web.
5.1 Sanitization of Datastore and Encoders
As demonstrated in Section the existence of pri- vate examples the kNN-LMs' datastore increase the likelihood of privacy leakage since they are retrieved and aggregated in the final prediction. Therefore, our first consideration is to create a san- itized datastore by eliminating privacy-sensitiv text segments. We note that this verbatim level def- inition of "privacy Icakage" is general and widely adopted. Notably, regulations such as HIPAA (Cen- for Medicare & Medicaid Services. 1996) and CCPA (California State Legislature 2018) offer explicit definitions of sensiive data. Con- sequently, these regulatory frameworks can serve
The encoders in a KNN-LM is another potential source of privacy leakage. While it is typically optimized on target domain data to enhance per- formance, fine-tuning directly private data in privacy-sensitive tasks may result in privacy leaks (Table 2). Similarly, the encoder can be sanitized by fine-tuning the pre-trained encoder Encpublic on a sanitized dataset that has had sensitive informa- tion removed.

---

5.2 Decoupling Key and Query Encoders
300
We propose using separate encoders for keys and queries in kNN-LMs, to allow for finer control over privacy preservation. For example, the encoder for queries can be the sanitized encoder, while the en- coder for keys can be the non-sanitized one; This way, the query encoder can be more resistant to privacy leakage, while the keys encoder can pro- vide better query results. While it is not common practice in kNN-LMs, we view the separation of promising approach
80
27.5
80
Total Plls
8
25.0
Perplexif
60
20
Perplaxi hoploxxt
22.5
40
20.0
20
17.5
0.0 0..2 15.0
10'
Figure Effect of the interpolation coefficient (A) and the number of nearest neighhors (k) on &NN-LMs util- ity (measured by perplexity, the blue curve) and privacy risk (measured by the number of reconstructed PIIs in the targeted attack, the green curve). We use Eriivivate as encoders and Dprivate as the datastore.
key and query encoders to reduce the discrepancy between the prompt and the datastore, and reduce privacy leakage.
The privacy risk of a kNN-LM can also be im- pacted by its hyper-ppaaete parameteras such as the number of neighbors k; and the interpolation coefficient X. It is important to consider these hyper- saper-parameters in the customization of the kNN-LMs to ensure that the privacy-utility trade-off is well managed.
polation coefficient, and observe that increasing À decreases perplexity but increases privacy risk (see Figure 2). This highlights the trade-off between accuracy and privacy, indicating that optimizing both factors simultaneously challenging through adjustment alone.
5.3 Results of Sanitization and Decoupling Encoders
k: is the number of nearest neighbors in a ANN- LM. As shown in Figure 2, increasing the value of k improves perplexity as it allows considering more nearest neighbors. We also notice that using larger k decreases privacy risk the model be- comes less influenced by limited group of private nearest neighbors. Together, increasing k seems simultancously enhance utility and reduce risk.
As demonstrated in Table 3, applying sanitization to both the encoder and the datastore effectively eliminates privacy risk, resulting in no person- ally identifiable information (PII) being extracted. Among the three methods, the strategy of replacing PII with random public information for sanitization yields the highest utility. It achieves perplexity of 16.38, which is only marginally worse than the perplexity of 16.12 achieved by the non-sanitized private model.
Mitigations Against Untargeted Risks
In this section, explore potential methods mitigate untargeted risks &NN-LMs, which a more challenging setting due to the opacity of the definition of privacy. It is important to note that the methods presented in this study are preliminary attempts, and fully addressing untargeted risks in KNN-LMs still remains challenging task.
Table 3 also demonstrates that utilizing separate encoders for keys and queries enhances the model's s utility compared to using the same sanitized en- coder for both. Specifically, using the non-sanitized encoder for the query and the sanitized encoder for the key, privacy risks re- main high due the potential leakage from the PLM- On the other hand, using the non-sanitized encoder for the key and the sanitized encoder for the query effectively eliminates privacy risk while still maintaining high level of utility. This finding highlights the importance of sanitizing the query encoder in ANN-LMs.
observe that when
6.1 Methods
Considering that storing Pprivate in the datastore is the primary cause of data leakage (as discussed in Section 4), and the challenge of sanitizing private data in the face of untargeted risks, we propose the following approaches to leverage public data for mitigating these risks.
5.4 Effect of Hyper-paramette
A Ading public data to datastore The quality of the retrieved neighbors plays crucial role in the performance and accuracy of kNN-LMs. Although uncommon to include public datapoints that are not specifically designed for the task or domain
We finally analyze the impact of key hyper- parameters on utility and privacy risks kNN-
LMs, using D Prrivate as datastore and Encprivate for both Enck and Enco. First, we vary A, the inter-

---

DATASTORE
DATA STORE
EVAL. PPL
# GOOD
EVAL. PPL.
# GOOD RECON
Enek
Encq
Npub
Npri
RECON
0
AL. 0
16.12
656
D privase
Encprivate
ElCpprivate
16.12
656
0
20.63
620 (4 5.5%)
Encprivate
EnCprivate
21.46
561 ( 14.5%)
21.46
14.5%)
5, 000
21.58
579 (L 11.7%)
EnCpublic
EnCpublie
33.60
0 ( 100%)
ALL
10, 000
21.23
595 (L 9.3%)
Encprivate
Etccpublie
31.57
54 (L 91.8%)
50, 000
19.16
617 () 5.9%)
EnCpublie
EnCprivate
22.90
451 (L 31.3%)
D public
ALL
19.02
632 (L 3.7%)
Enepp
EneDP
22.83
540 (4 16.9%)
Encprivate
Encpr
21.57
603 (L 7.2%)
Table 4: Perplexity and data extraction risks for &NN- LMs with different numbers of public (Npab) and pri- vate (Npriv) examples in the datastore. The encoder in use is the privately fine-tuned encoder E.Cprrvate- Pri- vacy measurements are computed using the top 5000 candidates. Risk reduction (%) compared to the first row is annotated. Table 12 in Appendix D presents similar findings on the Medical Transcriptions dataset.
EnepP
EnCpprivaee
21.62
594 (L 8.6%)
EnCmised
EnCmixed
21.10
601 (L 8,4%)
EnCprivate
EnCrised
21.12
(L 16.9%)
Ennaaxxed
EnCprrvaae
21.43
498 () 24.1%)
Table 5: Perplexity and data extraction risks for kNN- LMs with different encoders for keys (Enck) and queries (Eneq). Risk reduction (%) compared to the first row is annotated. Privacy measurements com- puted using the top 5000 candidates. The EπспP en- coder is fine-tuned using DP-SGD with privacy budget 10.0. The Encmixed encoder is fine-tuned using a mix of public and private data points. Results sug- gest that using different encoders for keys and queries or Encaixxed can improve the privacy-utility trade-off. Table 13 in Appendix D presents similar findings on the Medical Transcriptions dataset.
into ENN-LMs' datastore, it could potentially aid in reducing privacy risks in applications that priori- tize privacy. This becomes particularly relevant in light of previous findings, which suggest substan- tial privacy leakage from private datastore.
Fine-tuning encoders on private data with DP- SGD Differentially private stochastic gradient de- scent (DP-SGD) (Abadi et al., 2016) is a recipe for training a deep learning model with differential privacy (Dwork et al., 2006b) guarantee to protect privacy leakage. It operates by modifying the mini- batch stochastic optimization process through the use of per-example gradient clipping and Gaussian noise injection (See Appendix C for details).
from the Medical Transcriptions dataset, and those findings align with our main findings.
Table 4 demonstrates that when privately fine- tuned model Encprrvate serves as the encoder, re- placing the private datastore Pprivate with a public one Publiic in KNN-LMs considerably lowers the privacy risk. Furthermore, when using Encprivate and D public the risk level is slightly lower than when using the standard language model with EnCpprivate because the model's final response has been interpolated with non-sensitive information, which helps reduce privacy risks.
Fine-tuning encoders on a mixture of public and private data However, adding public data can potentially lead to a decrease in retrieval perfor- mance as there is distribution gap between the public data (e.g., Web Crawl data) used to con- struct the datastore and the private data (e.g., email conversations) used for encoder fine-tuning. To address this issue, we propose further fine-tuning the encoder on a combination of public and private data to bridge the distribution gap and improve re- trieval accuracy. The ratio for combining public and private datasets will be determined empirically through experimentation.
Using a public datastore reduces privacy risk but also results in a sudden drop in utility. If more stringent utility requirements but less strict privacy constraints are necessary, adding a few private ex- amples to the public datastore, as shown in Table 4, may also be a suitable solution.
Table demonstrates that using different en- coders for keys (Eneg) and queries (Encq) is more effective in achieving a desirable balance between privacy and utility when using Ppublic as the datas- tore. Specifically, using EnCprivate to encode keys and EnCpublic to encode queries significantly re- duces the risk of data extraction with only a slight decrease in perplexity.
Similarly to Section 5.2, we could also employ separate encoders for keys and queries in the con- text of untargeted risks, which allows for more precise control over privacy preservation.
6.2 Experimental Results
We mainly present our findings using the Enron Email dataset. In Appendix B, we provide results
We also note that fine- mie-tuning the encoder using DP-SGD only helps slightly reduce the extraction

---

risk, despite the relatively strict privacy budget e 10.0. This is because due to the existence of private datastore, each inference query the KNN- LM process incurs supplementary privacy costs, leading to the final KNN-LM model not satisfying the e-Differential Privacy criteria.
tion such personal email addresses (Huang al., 2022) and protected health information(Lchman et al., 2021; Pan et al., 2020). Recently, the mem- orization effect in LMs has been further exploited in the federated learning setting (Konečný et al., 2016), where in combination with the information leakage from model updates (Melis et al., 2019; Huang et al., 2020), the attacker is capable of recov- ering private text in federated learning (Gupta al., 2022). To mitigate privacy risks, there is a grow- ing interest in making language models privacy- preserving (Yu et al., 2022; Li ct al., 2022; Shi et al., 2022b; Yue et al., 2023; Cummings et al., 2023) by training them with a differential privacy guarantee (Dwork et al., 2006b; Abadi et al., 2016) or with various anonymization approaches (Naka- mura et al., 2020; Biesner et al.).
We further try fine-tuning the encoder using combination of public and private data, which re- sults in Eiemixed. The training dataset comprises the entire set of private data of size Npriv and Nprio x public data, where r takes values from {0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0). We present at- tack results using r = 0.05 as it achieves the best perplexity. As shown in Table 5, when the encoder is fine-tuned using combination of public and private data, the perplexity can be enhanced from 21.46 to 21.10 while simultaneously reducing pri- vacy risk. This is because Encmixed helps close the distribution gap between private and public data thus improving the retrieval results. Similarly, us- ing separate Enck and Encq also helps further reduce the privacy risk.
Although previous research has demonstrated the potential risks of data extraction in parametric language models, our study is the first investigation of the privacy risks associated with cetrieval-based language models; we also propose strategies to mitigate them. The closest effort is Arora et al. (2022), which explores the privacy concerns of using private data in information retrieval systems and provides potential mitigations. However, their work not specifically tailored to the context of retrieval-basee based language models.
7 Related Work
7.1 Retrieval-based Language Models
Retrieval-based language models (Khandelwal al., 2020; Borgeaud al., 2022; Izacard al., 2022: Zhong al., 2022: Min et al., 2023; Shi et al., 2023) have been widely studied in recent years. These models not only rely on encoder for- ward running but also leverage a non non- -on-paraet component to incorporate more knowledge from an external datastore during inference. The retrieval process starts by using the input as query. and then retrieving set of documents (i.e., sequences of tokens) from a corpus. The language model finally incorporates these retrieved documents as additional information to make its final prediction. While the deployment of retrieval- based language models has been shown to lead to improved perfor- mance on various NLP tasks, including language modeling and open- domain question answering, it also poses concerns about data privacy.
Conclusion
This work presents the first study of privacy risks in retrieval-basee language models, specifically focus- ing on &NN-LMs. Our objective is to investigate designs and training methodologies for ANN-LMs that strike a better privacy-utilice trade-off.
There are several conclusions from our investi- gation. First, our empirical study reveals that in- corporating private datastore in kNN-LMs leads to increased privacy risks (both targeted and un- targeted) compared to parametric language mod- els trained on private data. Second, for targeted attacks, our experimental study shows that sanitiz- ing ANN-LMs to remove private information from both the datastore and encoders, and decoupling the encoders for keys and queries can eliminate the privacy risks without sacrificing utility, achieving perplexity of 16.38 (vs. 16.12). Third, for untar- geted attacks, our study shows that using a public datastore and training the encoder on a combina- tion of public and private data can reduce privacy risks at the expense of reduced utility by 24.1% %% with perplexity of 21.12 (vs. 16.12).
7.2 Privacy Risks in Language Models
Language models have been shown to tend to mem- orize (Carlini et al., 2019; Thakkar et al., 2021; Zhang al., 2021: Carlini et al., 2023; Asai et al., 2023) their training data and thus can be prompted to output text sequences from its training data (Car- lini et al., 20,1), as well as highly sensitive informa-

---

Limitations
References
We discuss the limitations of this work as follows.
Martin Abadi, Andy Chu. Ian Goodfellow, H Bren- dan McMahan, Πya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential pri- vacy. In Proceedings of the 2016 ACM SIGSAC con- ference computer and communications security. pages 308-318.
The current study mainly demonstrates the privacy implications of nearest neighbor lan- guage models, but there are many other vari- ants of retrieval-based language models, such as RETRO (Borgeaud et al., 2022) and At- las (Izacard et al., 2022). Further study is needed to understand privacy implications of these models and whether our findings apply.
Simran Arora, Patrick Lewis. Angela Fan, Jacob Kahn, and Christopher Ré. 2022. Reasoning over public and private data in retrieval- systems. arXiv preprint arXiv:2203. 11027.
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023. Retrieval-based language models and applications. In Proceedings of the 61 Annual Meet- ing of the Association for Computational Linguistics (Volume Tutorial Abstracts), pages 41 41-46.
In the current study, use WikiText- 103 as the public domain for Enron Email, and LubMed-Patients for Medical Transcriptions. While we believe that these choices of public datasets are realistic, it is important to recog- nize that this selection may restrict the gener- alizability of our findings. We acknowledge this limitation and leave the exploration of al- ternative options for the public dataset as direction for future work.
Mislav Balunovic, Dimitar Dimitrov, Nikola Jovanović. and Martin Vechev. 2022. Lamp: Extracting text from gradients with language model priors. Ad- vances in Neural Information Processing Systems, 35:7641-7654.
David Biesner, Rajkumar Ramamurthy, Robin Sten- zel, Max Lübbering, Lars Hillebrand, Anna Ladi, Maren Pielka, Rüidgee Loitz, Christian Bauckhage. and Rafet Sifa. Anonymization of german finan- cial documents using neural network-based language models with contextual word representations. In- ternational Journal of Data Science and Analytics, pages 1-11.
Furthermore, unexplored aspect of our study is the potential combination of pro- posed strategies, such as decoupling keys and query encoders, with more diverse privacy- preserving techniques.
Acknowledgement
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jcan-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, al. 2022. Improving language models by retrieving from tril- lions of tokens. In International Conference on Ma- chine Learning (ICML). PMLR.
This project is supported by NSF CAREER award (IS-2239290), Sloan Research Fellowship, Meta research grant, and Princeton SEAS Inno- vation Grant. We would like to extend our sincere appreciation to Dan Friedman, Jiayi Geng, Zirui Wang, Alexander Wettig, and Zhiyuan Zeng for their valuable comments and feedback on earlier versions of this work.
California State Legislature 2018. The California Consumer Privacy Act of 2018 (CCPA). Online at act.pdf.
https:/cppaaccggolleeulaaooooos
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural lan- guage models. In International Conference on Learn- ing Representations (ICLR).
Nicholas Carlini, Chang Liu. Úlfar Erlingsson, Jernjj Kos, and Dawn Song. 2019. The secret sharer; Eval- uating and testing unintended memorization in neu- networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284.
Nicholas Carlini, Florian Tramer, Eric Wallace. Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650.

---

Centers for Medicare & Medicaid Services. 1996. The Health Insurance Portability and Account- ability Act of 1996 (HIPAA). Online at
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint 22Xiv:2208.03299
http://www.cs.hhhr www.wemmsss...o/hppaa.a..
Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang. Justin Lu, Jackie Tsay, Yinan Wang, Andrew M Dai, Zhifeng Chen, et al. 2019. Gmail smart compose: Real-time assisted writing. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2287-2295.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language Models. In International Conference on Learning Representations (ICLR).
Rachel Cummings, Damien Desfontaines, David Evans, Roxana Geambasu, Matthew Jagielski, Yangsibo Huang, Peter Kairouz, Gautam Kamath, Sewoong Oh, Olga Ohrimenko, et al. 2023. Challenges to- wards the next frontier in privacy. arXiv preprint 000929300
Bryan Klimt and Yiming Yang. 2004. The enron corpus:
A new dataset for email classification research. In Eu- repean Conference on Machine Learning. Springer.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Pe- ter Richtárik, Ananda Theertha Suresh, and Dave Bacon, 2016. Federated learning: Strategies for im- proving communication efficiency. In NIPS Work- shop on Private Multi Parlti--rrtt Machine Learning.
Jieren Deng, Yijue Wang, Ji Li, Chenghong Wang, Chao Shang. Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding. 2021. TAG: Gradient attack on transformer- based language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3600-3610, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Eric Lehman, Sarthak Jain, Karl Pichotta. Yoav Gold- berg, and Byron C Wallace. 2021. Does bert pre- trained on clinical notes reveal sensitive data? In North American Chapter of the Association for Com- putational Linguistics (NAACL).
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSh- erry, Ilya Mironov, and Moni Naor. 2006a. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT, pages 486-503.
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2022. Large language models can be strong differentially private learners. In International Conference on Learning Representations (ICIR)).
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006b. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference. Springer.
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Ouf, pages 74, Barcelona. Spain, Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Mclbourne, Australia. Association for Computational Linguistics.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. 2019. Exploiting unintended feature leakage in collaborative learning. 2019 IEEE symposium on security and privacy (SP). IEEE
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In International Conference on Learning Repre- sentations (ICIR).
Samyak Gupta, Yangsibo Huang, 7exuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. 2022, Recov- ering private text in federated learning of language models. In Advances in Neural Information Process- ing Systems (NeurIPS).
Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen- tau Yih, Hannanch Hajishirzi. and Luke Zettlemoyer. 2023. Nonparametric masked language modeling. Findings of Association for Computational Linguis- tics (ACL).
Junxian He, Graham Neubig, and Taylor Berg- Kirkpatrick, 2021, Efficient nearest neighbor lan- guage models. In Empirical Methods in Natural Language Processing (EMNLP).
Yuta Nakamura, Shouhei Hanaoka, Yukihiro No- mura, Naoto Hayashi, Osamu Abe, Shuntaro Yada. Shoko Wakamiya, and Eiji Aramaki. 2020. Kart: Privacy Icakage framework of language models pre-trained with clinical records. arXiv preprint arXiv:2101.0003
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre- pre-trained language models leaking your personal information? In Findings of Empirical Methods in Natural Language Processing ( (EMNLP).
Yangsibo Huang, Zhao Song, Danqi Chen, Kai Li, and Sanjeev Arora. 2020. Texthide: Tackling data pri- vacy in language understanding tasks. In Findings of Empirical Methods Natural Language Processing (EMNLP).
Arvind Narayaman and Vitaly Shmatikov. 2008. Robust de-anonymization of large sparse datasets. In 2008 IEEE Symposium on Security and Privacy (sp 2008). IEEE

---

Xudong Pan, Mi Zhang, Shouling and Min Yang. 2020. Privacy risks of penerae-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP).
A Training Data Extraction Attack
A.1 Untargeted Attack
Carlini et al. (2021) proposes the first attack that can extract training data from trained language model. The attack consists of two steps: 1) gen- erate candidate reconstructions via prompting the trained models, and 2) sort the generated candi- dates using a score that implies the possibility of being memorized text.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zetteeomer. 2022a. Nearest neighbor zero- shot inference.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo. Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Replug: eeerieval-auggeent black-box language models. In Empirical Methods in Natural Language Processing (EMNLP).
Generate candidate reconstructions The at- tacker generates candidates for reconstructions via querying the retrieval-augmeentd LM's sentence completion API with contexts. Following Carlini et al. (2021)², we randomly select chunks from a subset of Common Crawl to feed as these con- texts.
Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, and Zhou Yu. 2022b. Selective differential privacy for language modeling. In North American Chapter of the Associ- ation for Computational Linguistics (NAACL).
Sort candidates by calibrated perplexity The second step is to perform membership inference on candidates generated from the previous step. We are using the calibrated perplexity in our study, which has been shown to be the most effective membership metric among all tested ones by Car- lini et al. (2021).
Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays. 2021. Under- standing unintended memorization in language mod- els under federated learning. In Proceedings of the Third Workshop on Privacy Natural Language Pro- cessing, pages 1-10, Online. Association for Compu- tational Linguistics.
Frank F Xu, Uri Alon, and Graham Neubig. 2023. Why do nearest neighbor language models work? In Inter- national Conference on Machine Learning (ICML).
The perplexity measures how likely the LM is to generate a piece of text. Concretely, given a language model Sa and a sequence of tokens x Ii...., Jy, Ferplexity(fo.>> is defined as the exponentiated average negative log-likelihood of x:
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulka- mi, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. 2022. Differentially private fine-tuning of lan- guage models. In International Conference on Learn- ing Representations (ICLR).
exp
log So (x: xl 23 (1)
(1....,,;-..)
Xiang Yue, Huseyin A Inan, Xuechen Li, Girish Ku- mar, Julia Ananale,, Huan Sun, David Levitan, and Robert Sim. 2023. Synthetic text generation with differential privacy: A simple and practical recipe. In Association for Computational Linguistics (ACL).
A low perplexity implies high likelihood of the LM generating the text; For a
LM, this may result from the LM has been trained on the text or has used the text in its datastore.
Chiyuan Zhang. Daphne Ippolito. Katherine Lee, Matthew Jagielski. Florian Tramer, and Nicholas Car- lini. 2021. Counterfactual memorization in neural language models. arXiv preprint 222.:221222222
However, perplexity may not be a reliable indica- tor for membership: common texts may have very low perplexities even though they may not carry privacy-sensitive information. Previous work (Car- lini et al., 2019, 2021) propose to filter out these uninteresting (yet still high-likelihood samples) by comparing second LM which never sees the
Zexuan Zhong, Tao Lei, and Danqi Chen. 2022. Train- ing language models with memory augmentation. In Empirical Methods in Natural Language Processing (EMNLP).
'They have empirically show that sampling conditioned on Internet text the most effective way identify memorized content, compared with top-r. sampling (Fan al., 2018) and temperature-baat sampling (see Section 5.1.1 in their paper). "Common Crawl is a nonprofit organization that crawls the web and freely provides its archives and datascts to the public. See their wehpage for details: http: http////ooooo/aaae// wwll.rg//

---

Phone
Email
URL
If you have questions, please feel free to give me call at Please advise call Please call can be reached If you have any questions, please call
For more information, send email to
The site can be found
For more information please email us at Suggestions and feedback are welcome at For more information please email us at Please forward this e-mail
For more information, visit Please visit web site Visit our home page For more details go
Table 6: Example extraction prompts for different types of PITs.
TARGETED ATTACK
UNTARGETED ATTACK GOOD RECON
MODEL
EVAL. PPL.
TOTAL
PHONE
EMAIL
URI.
6NN-LM EnCprivate W/ D private &NN-LM, Encor,e=5.0 w/ privatte
16.12
54
23
656
29.17
7
1
54
6NN-LM, Knopp,e=o.d private
17.55
44
21
17
651
Table 7: Perplexity and data extraction risks for original and DP models on the Enron Email dataset.
private dataset. Specifically, given a piece of text x and the target model fo, and the reference LM fer the calibrated perpelxity computes the ratio gref ffe/,)). ,)).
identifiers and serial numbers, web URL, IP Ad- dress, finger or voice print, photographic image, and any other characteristic that could uniquely identify the individual. In our study, we focus on three frequently investigated PIl in previous lit- erature (Huang et al., 2022; Carlini et al., 2021), including email addresses, telephone numbers, and URLs.
Perplexity(for ()) hh/pppeexxity....//////
A.2 Targeted Attack
The untargeted attack has demonstrated the fea- sibility of recovering an entire sentence from the deployed retrieval-augmentte LM. However, it is possible that only a small segment of a sentence contains sensitive information that can act per- sonal identifiers, and thus be of interest to the at- tacker. Therefore, we also consider the type of attack which specifically targets this type of infor- mation.
A.2.2 The Attack
It's important to note that our approach differs from the work Huang ct al. (2022), which aims to re- construct the relationship between PIIs and their owners?. Instead, our study focuses on reconstruct- ing the actual values of Plls. This is because, even if the attacker cannot determine the relationship through the current attack, the reconstruction of PIIs is already considered identity theft". Further, the attacker can use the linkage uttack (Narayanan and Shmatikov, 2008) with the aid of publicly avail- able information to determine the relationship be- tween PIIs and their owners.
We define personal identifiers and describe the attack method and evaluation subsequently.
A.2.1 Definition of Personal Identifiers
Personal Identifiable Information (PII) refers to any data that can be used to identify specific indi- vidual, such date of birth, home address, email address, and telephone number. PIl is considered sensitive information and requires proper protec- tion ensure privacy.
PII extraction. Similar to the training data ex- traction attack, the PII extraction attack consists of two steps: 1) generate candidate reconstructions, and 2) sort them using membership metrics.
The exact definition of PII can vary depend- ing on the jurisdiction, country, and regulations in place. One of the clearest definitions of PII is provided by Health Insurance Portability and Accountability Act (HIPAA) (Centers for Medi- care & Medicaid Services, 1996), which includes name, address, date, telephone number, fax num- ber, email address, social security number, medical record number, health plan beneficiary number, ac- count number, certificate or license number, vehicle
To tailor the attack to recover personal identifi- able information rather than the entire training text. customize the attack prompts based on the type of information to be extracted.
"This threat model requires additional information about the presence of owners the dataset. "https: //en: httsss///wwwwwiieeeeeeeee wwkipedia. wikii/Ieettiyy

---

|#
EMAILS CONTAINING PII # UNIQUE VALUES
about any individual in the dataset. Formally, ran- domized algorithm A is (e, 6)-DP if for any two neighboring datasets 'D and D' (i.e., datasets that differ by single individual's data), and any subset S of outputs, it holds that
EMAIL PHONE
1,013
758
1,921
1,621
URL
1,641
1,396
Table 8: Number of Pils in the Enron Email dataset.
Pr(A(D) E E S of -EPrA((D") E S] + 8.
MODEL
EVAL. PPL
(PARAMETRIC LM) EnCpublic, Wiltlet WildTextt
31.42
Here, & E RR0,0 8 E [0, 1) are privacy parameters
(PARAMETRIC LM) EnCpublic. PubMed
23.83
quantifying the privacy guarantee of the algorithm. DP Stochastic Gradient Descent (DP- SGD) (Abadi et al., 2016) is recipe for training a deep learning model with DP by modifying the mini-bater stochastic optimization process through the use of per-example gradient clipping and Gaussian noise injection. When training an ML model / parameterized by with the per-example loss function {(:--) on dataset D, each optimization step involves randomly sampling a mini-batch B, Given B, DP-SGD starts by computing the per-example gradient for each (æ,, mi) B., where z; is the feature vector and y is the corresponding label, as follows:
(GNN-LM) EnCpublic, WikeTest W/ Priieane
14.75
(kNN-LM) PubCpeblli, W/ D privaze
9.70
Table 9: Evaluation perplexity parametric LMs and kNN-LMs with different choices of D public- Using PubMed as pabliic results in better perplexity.
B Experimental Details
Hyper-parameters For each model con- figuration, we search hyper-parameters k E {64, 128, 256, 512, 1024, 2048} and A E {0.1,0.2,0.3, 0.4, on a held-out validation set for the best model perplexity.
PIIs in Enron Email dataset We use regular expressions to identify and extract three types of personal identifiers from the Enron Email training dataset for the of the targeted attack, including telephone numbers, email addresses, and URLs. Table 8 provides statistics for these personal identi- fiers.
gi (Zi, +- Vo, (fo, (x:), u.).
It then clips the gradient fn-norm to a maximum 22-norm of C:
gg (I,, will := -g((,, wi) max(1, (1s)())a))) la
C'
Finally, it produces the private gradient g by injecting Gaussian noise into the sum of the clipped per-example gradients:
Prompts for the targeted attack We gather common preceding context for telephone num- bers, email addresses, and URLs, and use them prompts for the targeted attack. Table 6 provides example prompts we use in the attack.
6: + Bi
Σ B8 Bel + + (0, N (0,)²C²1))
where (2, is Gaussian distribution with mean 0 and covariance o o²C and the noise multi- plier computed from (s, 6) by inverse privacy accounting (e.g., Abadi al. (2016)).
Attack parameters For the untargeted attack, we generate 100,000 candidates, and for the targeted attack, we generate 10,000 candidates. We use beam search with repetition penalty = 0.75 for the generation.
C.2 Results
C Defending the Untargeted Attack with Differential Privacy
We also evaluate whether DP can mitigate extrac- tion risks in &NN-LMs. Specifically, we fine-tune the pre-trained LM on the private dataset with DP- SGD. We vary the privacy budget and fix the failure probability 8 to be 1/N where N is the number of training examples. It's important ac- knowledge that due to the utilization of a private datastore, each inference query in the &-N-LM
C.1 Differential privacy and DP Stochastic Gradient Descent
Differential privacy (DP) (Dwork al., 2006b,a) is a mathematical framework for ensuring the pri- vacy of individuals in datasets. can provide strong guarantee of privacy by allowing data to be analyzed without revealing sensitive information
"The specific loss depends on the particular task and model (c.g. cross-cntropy loss for classification)

---

Example #1: PAST MEDICAL IUSTORY: He has difficulyy climbing stairs, diffiullly with airline lying shoes, used to public seating, and litingg objects off the Boor. He exercises three times a week at home and does cardio, bas difficully walking two blocks or live flights of stairs. Difliculty with snoring. He his musele and joint pains including knee pain, back pain, foot and ankle pain, and swelling. He has gasinesophageal reflux discase...
Example 42: HISTORY OF PRESENT IIJ ILLNEES. This is S5-yea-oldd female with history of stroke, who presents today for followup of frequency and urgency with urge incomtinence. This has boen progresively worsening, and previoedly on VESIcare with improvement She continaes take Enablex 50 mg and has not noted any improvement of her sympioms. The nursing home did not do a vaiding diary. She is accompanied by her power of attomey...
Example #3: EXAM: Ulrasound examination of the scrotum, RR.......AAAAA.. REASON FOR AX Somoaa FINDINSS .Duplex and color flow imaging as well as real testicle meusures x
cm. There no evidence intratesticular masses. There is normal Doppler blood flow. The left qpididymis has an unremarkable kable appearance. There is a trace hydroccle...
time gray scale imaging of scrotum lesticles was performed.
Example #4: TESTICULAR LAR ULTRASOUND, REASON FOR EXAM: Left testicular swelling for one day. FINDING: The left testicle is normal in size and allenuation, messures
a hydrocele on right side. Normal flow is seen within testicle and epididymis on the right., The loft testiele is normal in size and attemaaioon, it measures 3.9 x 2.1 x 2.6 cm...
1.7% 2.3 can. right epididymis measures up
Example #5: PHYSICAL EXAMINATION: The patient a 63-ycar-old exccutive who was soon by his physician for a compuny physical. He stated that be was in his Chest x-ray and chemical sereening blood wark were within normal
excellent health and led active life. His physical examination normal limits, His PSA was elevated. IMAMeid,.MMGGGGG:GCC x-ray; Normal,CT scan of ahdomen and pelvis: No abvormalities..
Table 10: Examples from the Medical Transcriptions dataset.
MODEL
EVAL. PPL
# GOOD RECON
(PARAMETRIC LM) EnCpublic
23.83
(PARAMETRIC LM) Eucprivate
12.00
769
(kNN-LM) EnCpublic W/ 'ppiiate
9.70
122
(I:NN-LM) EnCpprivate W/ Dprivate
6.61
812
Table 11: Perplexity and data extraction risks for various model configurations with the medical transcription dataset. The configuration with the highest leakage is emphasized in red: the configuration with the lowest leakage is highlighted in green. Privacy measurements are computed using the top 5000 candidates for the untargeted attack.
DATASTORE
Encg
Eneq
EVAL. PPL. # GOOD RECON
EVAL. PPL
# GOOD RECON
ppub
Npriv
D Priveate W/ EnCprivaae
6.61
812
ALL 0
6.61
812
Encprivace
Encprivate
11.32
759 (4 6.5%)
0
12.00
769 (4 5.3%)
0
11.32
759 (4 6.5%)
EnCpublic
EnCpublic
22.55
0 (L 100%)
1, 1,000
10.54
773 (L 4.8%)
EuCprivate
EuCpublic
23.01
7 (C 99.1%)
Encpablic
Encprivase
11.36
689 (J 15.1%)
ALL.
2, 000
9.29
787 (L 3.1%)
3, 000
8.26
799 (L 1.6%)
Encmised
EnUmissd
12.32
773 (J 4.8%)
ALL 6.84
804 (L 1.0%)
EnCprivate
EnCmised
12.47
707 ( 12.9%)
Encaised
Eneprivase 12.36
692 (1 14.8%)
Table 12: Perplexity and data extraction risks for KNN- LMs with different numbers of public (Npat) and pri- vate (Npriv) examples in the datastore. The evaluation uses the medical transcription dataset. The encoder in use is the privately fine-tuned encoder Eucprivate. Pri- vacy measurements are computed using the top 5000 candidates.
Table 13: Perplexity and data extraction risks for KNN- LMs with different encoders for keys (Enck) and queries (Eneq). The datastore in use is the public data- store D Puublic- The evaluation uses the medical tran- scription dataset. Privacy measurements using the top 5000 candidates. The Encinixed encoder is fine-tuned using a mix of public and private data points. Results suggest that using different encoders for keys and queries or Encmixed can potentially improve the privacy-utiiity trade-off.
computed
process incurs supplementary privacy costs, lead- ing to the final &NN-LM model not satisfying the (c, -iffferreniia Privacy criteria.
D Untargeted Attacks on Medical Transeriptions Dataset
As demonstrated in Table 7, when E is set to 5.0, the model showcases minimal utility alongside a marginal privacy risk. Conversely, with raised to 10.0, the utility closely resembles that of utilizing Encprivate in conjunction 2 privatte while concur- rently slightly reducing the associated risk. These results suggest that DP is also a viable alternative in improving the utility-privacy trade-off in ANN-LM.
We primarily showcase our findings using the En- ron Email dataset in the main paper, as its inclu- sion of personally identifiable information (PII) enables us to effectively evaluate both targeted and untargeted attacks. To validate our findings, we hereby replicate our experiments specifically for

---

untargeted attacks on the Medical Transcriptions dataset.
D.I Experimental setup
The Medical Transcriptions dataser contains 5,000 medical transcriptions for various medical specialties. We use a subset of 4,500 examples for training and the rest for evaluation. Table 10 provides examples from the dataset. When using the Medical Transcriptions dataset as private. D public. rather than
opt for MubMed-Pattent' WikiText. This choice is motivated by the fact that PubMed-Patient exhibits closer semantic align- ment with the Medical Transcriptions, leading to enhanced utility in our evaluation. Further insights and justifications regarding this dataset selection can be found in Table 9. We use the GPT-2 base model (Radford et al., 2019) as EnCpublic- and fine- tune it on the Medical Transcriptions dataset as EnCprivate-
D.2 Results
The preliminary findings presented in Table align with the observations outlined in Section highlighting a deficiency in the robustness of pri- vacy protection within the design of &NN-LMs. Specifically, these results indicate that the pri- vacy concern stemming from the private datastore DD privatte outweighs that resulting from the privately fine-tuned model EnCpprivate-
One approach address this issue is incor- porate a public datastore (Section 6), which helps mitigate privacy risks small number of private examples can introduced to the public datas- tore, striking balance between utility and privacy considerations. Table 12 demonstrates the effec- tiveness of this approach, offering a promising com- promise.
We also observe the Medicial Transcriptions dataset that separating the key and query encoders yields better results in striking a favorable trade-off between privacy and utility. As shown in Table 5, employing distinct encoders, e.g., Encprivate for encoding keys and Encpublic for encoding queries, substantially diminishes the likelihood of data ex- traction while only marginally affecting perplexity,
10 "ht /pt:///// kaggle, cww com/ddatasets//boyleeee medical iptions PrubMed-Patientt is derived from 167,000 patient sum- marics extracted from case reports in PubMed Central. Fur- ther details about this dataset can he found at: https:// huggingface. ctt.at//aaaaaaeeeea /ww.attst/seaaaeeeceeeda /ttttsttss/henggun2211


Reviews:
[{"review_id": "soIaGcxYy2", "reviewer": "Reviewer_nKXz", "summary": "", "questions": "1. Could you please provide a justification for the above weaknesses?\n2. In Tables 1 and 2, what is the evaluation dataset in terms of eval PPL, $\\mathcal{D}_{\\text{public}}$ or $\\mathcal{D}_{\\text{private}}$?\n3. In untargeted attacks, the goal is to reconstruct the entire training example. Can you showcase what types of training examples are considered private in your case?\n4. The description of nearest neighbor language models is unclear (e.g., notation unclear). For example, $w_i$ is not defined throughout the paper.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback. It's encouraging that the reviewer acknowledges the importance of our study on privacy risks in retrieval-based LMs and finds our mitigation strategies effective. We will carefully address your comments below, and have added the DP experiments in the rebuttal additionally. We are more than willing to engage in further discussion should any follow-up questions arise! \n\n**“The privacy definition (e.g., memorization) is not formally defined. ”**\n\nWe hereby clarify our privacy definition: Our definition of privacy leakage extends the standard definition of the Model Knowledge Extraction (Definition 1 in Carlini et al., 2021)). Specifically, a string $s$ is extractable from an LM $f_θ$ if there exists a prefix $c$ such that: $s \\gets \\arg\\max_{s’} f_\\theta(s’|c)$. Namely, the model generates $s$ as the most likely continuation when prompted with some prefix $c$.\n\nSpecifically, in our untargeted setting, $s$ is an example in the training dataset, and in the targeted setting, $s$ is a subsequence of a training example. We will provide this definition in the revision. \n\n**\"The author uses ROUGE-L as the privacy metric, and the choice of privacy metrics does not make sense to me.”**\n\nROUGE-L measures the longest common subsequence (LCS) between the attack result and the ground truth, thus representing the fidelity of the attacker results. ROUGE-L score has been used to measure the reconstruction fidelity in previous work [1,2,3]. If the reviewer can specify why it does not make sense, we are happy to further clarify it.\n\n**“The threat model is unclear”**\n\nWe believe this is due to a misunderstanding, and we do not assume the attacker has access to the private dataset. The use of private datasets is only for automatic evaluation purposes.\n\nSpecifically, our threat model (defined in Section 3.1) assumes that the attacker only has black-box API access to the model: The attacker generates top-n candidates through the model's API and sorts them based on their defined score, all **without** utilizing private data. We then evaluate the attack’s performance by comparing top-n candidates against the private dataset, which aligns with standard practices in reconstruction attacks [1, 2, 3, 4]. \n\nNote that our threat model is indeed the same as that by Carlini et al. 2021 in our submission, but they only considered small-scale human evaluation while we used private data for automatic evaluation.\n\n**“Explanation for why kNN-LM is more vulnerable than parametric LM”**\n\nWe would like to clarify that we didn’t claim that the privacy risk of $k$NN-LMs is more serious than parametric LMs. Instead, our work focuses on understanding the conditions under which $k$NN-LM exhibits higher or lower vulnerability.\n\nTwo notable observations we made are:\n- When private data is incorporated into the datastore, the vulnerability of $k$NN-LM tends to increase, as indicated in Table 1. Our rationale behind this finding is that the datastore offers a more explicit way for potential privacy breaches compared to the parameters within traditional LMs. This inference gains support from our results in Figure 2, where we empirically illustrate that with a larger $\\lambda$ (resulting in increased emphasis on the $p_{kNN}$ term during final prediction), the model's vulnerability increases, and with a smaller $k$ (resulting in the model being influenced by a limited group of private nearest neighbors.) the model's vulnerability increases.\n- Conversely, incorporating public data into the datastore along with privately fine-tuned encoders renders $k$NN-LMs less vulnerable than parametric LMs. This reinforces the rationale that the private datastore primarily contributes to the leakage.\n\nWe believe that a theoretical understanding of this phoneme would be an interesting area of future research, but it may exceed the scope of current empirical investigation.\n\n**“Misses differential privacy (DP) as a defense”**\n\nWe appreciate the suggestion and provide the DP results below. Specifically, we fine-tune the pre-trained LM on the private dataset with DP-SGD. We vary the privacy budget $\\epsilon \\in \\{5.0, 10.0\\}$ and fix the failure probability $\\delta$ to be $1/N$, where $N$ is the number of training examples. It's important to acknowledge that due to the utilization of a private datastore, each inference query in the $k$NN-LM process incurs supplementary privacy costs, leading to the final $k$NN-LM model **not** satisfying the $(\\epsilon, \\delta)$-Differential Privacy criteria.\n\nAs demonstrated, when $\\epsilon$ is set to 5.0, the model showcases minimal utility alongside a marginal privacy risk. Conversely, with $\\epsilon$ raised to 10.0, the utility closely resembles that of utilizing Enc$\\_\\textrm{private}$ in conjunction $D\\_\\textrm{private}$, while concurrently slightly reducing the associated risk. These results suggest that DP is also a viable alternative in improving the utility-privacy trade-off in $k$NN-LM.\n\n| Model | Eval PPL | Targeted attack, total | Targeted attack, phone | Targeted attack, email | Targeted attack, URL | Untargeted attack (# Good Recon) |\n|---|---|---|---|---|---|---|\n| $k$NN-LM, Enc$_\\textrm{private}$  w/ $D\\_\\textrm{private}$| 16.12 | 54 | 25 | 23 | 6 | 656 |\n| $k$NN-LM, Enc$_\\textrm{DP, $\\epsilon$=5.0}$  w/ $D\\_\\textrm{private}$| 29.17 | 7 | 3 | 1 | 3 | 54 |\n| $k$NN-LM, Enc$_\\textrm{DP, $\\epsilon$=10.0}$  w/ $D\\_\\textrm{private}$ | 17.55 | 44 | 21 | 17 | 6 | 651 |\n\nWe also would like to highlight that DP and our suggested defense mechanisms are not mutually exclusive. We now provide additional evidence of the compatibility between DP and our proposed defense strategies: by disentangling the key and query encoders (with one of the encoders being the DP encoder), we effectively enhance the trade-off between privacy and utility.\n\n| Enc$_K$ | Enc$_Q$ | Eval PPL | Good Recon |\n|---|---|---|---|\n| Enc$_\\textrm{private}$ | Enc$_\\textrm{private}$ | 21.46 | 561 |\n| Enc$_\\textrm{DP, $\\epsilon$=10.0}$ | Enc$_\\textrm{private}$ | 21.62 | 594 |\n| Enc$_\\textrm{private}$ | Enc$_\\textrm{DP, $\\epsilon$=10.0}$ | 21.57 | 603 |\n| Enc$_\\textrm{DP, $\\epsilon$=10.0}$ | Enc$_\\textrm{DP, $\\epsilon$=10.0}$ | 22.83 | 540 |\n\n\n**In Tables 1 and 2, what is the evaluation dataset in terms of eval PPL, $D_{\\text{public}}, D_{\\text{private}}$?**\n\nTable 1 and 2 uses the evaluation set of Enron Email dataset ($D_{\\text{private}}$) for its evaluation. \n\n**Can you showcase what types of training examples are considered private in your case?**\n\nGreat question! When it comes to an untargeted attack, any effort that effectively reconstructs a significant portion of the original training example is deemed a successful attack, regardless of the type of training example. This is because the attack only needs to function in the worst-case scenario rather than the average scenario.\n\nFor the reviewer's reference, we've also included a few examples from the Enron Email dataset. (We've anonymized any identifiable information to protect against potential privacy breaches inherent in the dataset.)\n\n\n| Example |\n|---|\n| Request ID : 0000000000***** Request Create Date : */*/* 8:27:00 AM Requested For : ***.***@enron.com Resource Name : Market Data Bloomberg Resource Type : Applications |\n| You can reach me over the weekend and in the evening at either ***-***-**** or ***-***-****. |\n| This would give you a total loan of ****, total cost of **** for equity required of ****. |\n| Winter 2001 baseload traded as high as ** pounds a megawatt-hour and as low as ** pounds a megawatt-hour, before closing at ** pounds a megawatt-hour, ** pence lower than Friday. |\n| everything is correct except for password: ***. |\n\n**The description of nearest neighbor language models is unclear (e.g., notation unclear). For example, w_i is not defined throughout the paper.**\n\n{$c_i, w_i$} represents an entry in the $k$NN-LM’s datastore, where $c_i$ is the leftward context and $w_i$ is the next word for $c_i$. Thanks for your suggestion, and we will make it clear in the revision.\n\n**Typo in Table 2**\n\nThanks for the reviewer’s careful reading. We apologize for the typo and will fix it in revision.\n\n**References**\n\n[1] Deng et al., Tag: Gradient Attack on Transformer-based Language Models. EMNLP 21\n\n[2] Balunovic et al., Lamp: Extracting Text from Gradients with Language Model Priors. NeurIPS 22\n\n[3] Gupta et al., Recovering Private Text in Federated Learning of Language Models. NeurIPS 22\n\n[4] Huang et al., Are Large Pre-Trained Language Models Leaking Your Personal Information? EMNLP 22", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and well-motivated. The privacy risks of retrieval-based LMs are important.\n\n2. Experimental results demonstrate that the proposed attacks are effective and the authors provide countermeasures to mitigate the privacy risk.", "reasons_to_reject": "1. The privacy definition (e.g., memorization) is not formally defined. Carlini et al. (2021) provide a rigorous privacy definition (e.g., k-eidetic memorization). However, the authors do not provide such a definition. Instead, they only use whether to extract segments or a whole training example to differentiate targeted and untargeted attacks. The categorization is strange to me. Besides, in the evaluation of untargeted attacks, the author uses ROUGE-L as the privacy metric, and the choice of privacy metrics does not make sense to me. \n \n2. The threat model is unclear.  For example, in the privacy measurements, the author implicitly assumes the attacker could get access to the private dataset $p_i$; otherwise, there is no way for the attacker to evaluate the top-n candidate. However, this does not make sense to me. The authors should provide more justification for this.\n\n3. The authors do not provide insights into *why* the privacy risk of kNN-LM is more serious than parametric LM empirically or theoretically. \n\n4. The paper misses differential privacy (DP) as a defense in the experiments. Though the authors acknowledge this point in the limitation section, I still think it should be included in the experiment since DP is a standard defense against privacy attacks.", "questions_to_authors": "1. Could you please provide a justification for the above weaknesses?\n2. In Tables 1 and 2, what is the evaluation dataset in terms of eval PPL, $\\mathcal{D}_{\\text{public}}$ or $\\mathcal{D}_{\\text{private}}$?\n3. In untargeted attacks, the goal is to reconstruct the entire training example. Can you showcase what types of training examples are considered private in your case?\n4. The description of nearest neighbor language models is unclear (e.g., notation unclear). For example, $w_i$ is not defined throughout the paper.", "paper_topic_and_main_contributions": "The paper studies the privacy risk of retrieval-based language models (LMs) by designing data extraction attacks. In particular, the authors propose two types of attacks: (1) targeted attack to reconstruct certain segments of text and (2) untargeted attack to recover the entire training example.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Table 2, the caption indicates the best utility under sanitization is highlighted in boldface. However, I do not see any number highlighted in boldface. Is it a typo?"}, "extra_text": {}}, {"review_id": "pCM7ncSXmw", "reviewer": "Reviewer_EAGn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback. It's motivating to see the reviewer’s recognition of our study's significance, solid formulation, comprehensive empirical evaluation, and interesting insights. We are now taking the opportunity to carefully address the detailed comments provided by the reviewer, as this will aid us in revising the paper.\n\n\n**\"The use of perplexity as a measure of utility is not well motivated.\"**\n\nWe appreciate the question. To clarify, in our context, \"utility\" refers to the performance of the language model. We use perplexity as a measure of utility following previous works (Khandelwal et al., 2020; He et al., 2021; Zhong et al., 2022; Shi et al., 2022a; Xu et al., 2023, as referenced in our submission), and we will leave the impact on downstream tasks as future work.\n\n\n\n**“It would be useful reflect privacy implications in other types of retrieval based language models”**\n\nWe acknowledge this is a limitation, but we believe that our work paves the first step toward understanding the trade-off between privacy and utility in retrieval-based LMs. Notably, we observe that $k$NN-LMs are more prone to privacy breaches from their private datastore compared to parametric models. We further explore mitigations of privacy risks, including sanitization of the private datastore, and decoupling key and query encoders.\n\nWe plan to leave the exploration of other retrieval-based LMs to future work. We also would like to note that $k$NN-LMs have received wide attention in the last few years, and many works have been proposed to improve their efficiency, their training methods, or their downstream adaptations (He et al., 2021; Zhong et al., 2022; Shi et al., 2022a; Xu et al., 2023, in our submission). Hence, we believe that understanding the privacy risks of kNN-LMs itself is important, and the scope would suffice in this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well written paper. The topic is interesting and important. The authors motivate the paper well and provide a solid background and problem definition. The experiments are comprehensive and the insights from the results are interesting. ", "reasons_to_reject": "The use of perplexity as a measure of utility is not well motivated. \n\nThe nearest neighbor variation of retrieval based language models is one of many different types of retrieval based language models. It would be useful reflect privacy implications in other types of retrieval based language models. \n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors investigate the privacy risks associated with retrieval-based language models. The main objective of the paper is to find ways to improve the balance between privacy and utility of these models. The authors finds that Including a private datastore in kNN-LMs increases both targeted and untargeted privacy risks compared to parametric LLMS trained on private data. For targeted attacks, sanitizing private information can eliminate privacy risks without sacrificing utility. For untargeted attacks, using a public datastore and training the encoder on a combination of public and private data can reduce privacy risks but leads to a decrease in utility.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lwTT38X51P", "reviewer": "Reviewer_KGUM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback. It's encouraging to see the reviewer recognizing the significance of our study, the well-structured approach we proposed, and the systematic evaluation we conducted. We are now taking the opportunity to carefully address the detailed comments provided by the reviewer, as this will aid us in revising the paper.\n\n\n**“The framework provided would be hard to use with other retrieval augmented LLMs like RETRO, or more recent chain-of-thought based approaches with LLMs.”**\n\nWe acknowledge this is a limitation, but we believe that our work paves the first step toward understanding the trade-off between privacy and utility in retrieval-based LMs. Notably, we observe that $k$NN-LMs are more prone to privacy breaches from their private datastore compared to parametric models. We further explore mitigations of privacy risks, including sanitization of the private datastore and decoupling key and query encoders.\n\nWe plan to leave the exploration of other retrieval-based LMs to future work. We also would like to note that $k$NN-LMs have received wide attention in the last few years, and many works have been proposed to improve their efficiency, their training methods, or their downstream adaptations (He et al., 2021; Zhong et al., 2022; Shi et al., 2022a; Xu et al., 2023, in our submission). Hence, we believe that understanding the privacy risks of $k$NN-LMs itself is important, and the scope would suffice in this work.\n \nWe are confused by why chain-of-thought is discussed here, as CoT is more of a prompting technique and hasn’t been really investigated in retrieval-augmented LMs (to the best of our knowledge). \n\n\n**“The results presented are applicable only to the specific kind of training data used, both public and private.”**\n\nWe respectfully disagree with this point. We would like to note that we also evaluated our approach with the Medical Transcriptions dataset as $D_\\textrm{private}$, as shown in Appendix C. We also varied the choice of public dataset $D_\\textrm{public}$ among WikiText and PubMed among WikiText and PubMed (see Appendix C.1, Table 7), and found that PubMed was a more suitable public dataset.\n\nThe results in Table 8, 10, and 11 for the Medical Transcriptions dataset are consistent with our findings in Table 1, 3, and 4 for the Enron Email dataset. This suggests that our approach is generalizable to different types of private datasets.\n\n**“The mitigation strategies – like sanitizing the data – would be very data set specific.”**\n\nWe respectfully disagree with this comment. Given a privacy-sensitive application, as long as we can define “privacy leakage” at the verbatim level, we can operate sanitization. \n\nWe further note that the verbatim level definition of “privacy leakage” is general and widely adopted. Notably, regulations such as HIPAA and CCPA offer explicit definitions of privacy-sensitive data. Consequently, these regulatory frameworks can serve as the basis for establishing the verbatim-level definition of \"privacy leakage.\" For example, HIPAA defines 18 identifiers that are considered personally identifiable information (PII), including names, addresses, phone numbers, etc.\n\nTherefore, the strategy of sanitization, despite its prerequisite for a verbatim-level privacy leakage definition, remains applicable on a broader scale.\n\n\n**“What w_i is in the Inference part of section 2.1”**\n\n{$c_i, w_i$} represents an entry in the kNN-LM’s datastore, where $c_i$ is the leftward context and $w_i$ is the next word for $c_i$. Thanks for your suggestion, and we will make it clear in the revision.\n\n**Rephrasing a sentence**\n\nThanks for your suggestion, and we will fix it in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper delves into a concern around large language models that is becoming increasingly important – that of data leakage, especially when private data is used to train or tune the models. Strengths of the paper are the following:\n\n• For a class of retrieval augmented LLMs, it takes a structured approach at determining leakage of private data for specific kinds of threats.\n\n• It provides a framework that can be used for systematic exploration of privacy leakage in general with LLMs.\n\n• It identifies certain threat models that are realistic representations of what an LLMs might experience and shows the leakage of private data. \n\n• It suggests some approaches to mitigate the leakage, along with the impact it likely has on the performance of the LLM, that way giving the developers of LLMs ways to tradeoff performance and risk of private data leakage.", "reasons_to_reject": "While the ideas presented in the paper are very pertinent to ongoing research on privacy and LLMs, the major weaknesses of the paper are the following:\n\n• The framework provided would be hard to use with other retrieval augmented LLMs like RETRO, or more recent chain-of-thought based approaches with LLMs. \n\n• The results presented are applicable only to the specific kind of training data used, both public and private. While the results show different ways in which privacy and performance tradeoffs can happen, it is very likely the results would be different for different LLMs, different data for the same kNN-LMs.\n\n• The mitigation strategies – like sanitizing the data – would be very data set specific as well and might not be applicable more generally. \n\nFor these reasons, while the framework clearly demonstrates the possibility of data leakage, in its current form, it is not very generalizable and its applicability to other LLMs, other datasets might be limited.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Large language models have seen tremendous growth more recently. This paper explores privacy implications for a class of large language models called kNN-LMs. As a part of the processing, this class of LMs use a retrieval system to fetch k tokens that are used along with the context in generating the next token. This paper explores how private information that was a part of the training data of this model can leak and suggests ways to mitigate the leakage.\n\nMain contributions of the paper:\n\n• The paper shows how kNN-LMs are susceptible to leakage of private data that could have been a part of the training data of encoders, or available at inference time through a retrieval system.\n\n• The paper shows approaches to mitigate some of the leakage and quantifiably shows the impact. These include sanitizing the training data, using combination of encoders that are training only on public data among others.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* It would be great if the authors can also explain what w_i is in the Inference part of section 2.1\n\n* Section 5.1 - Fine-tuning encoders on a mixture of public and private data (page 7) - The first sentence \"However, adding public data may cause ...\" needs to be rephrased."}}]

Author rebuttal:

Meta-review: The manuscript studies how to balance training of retrieval-based Language Models between utility (represented by perplexity) and privacy tradeoff. The comprehensive study reveals that kNN-LMs are more susceptible to leaking private information than parametric model as more private data and fewer public data are incorporated into the datastore. The authors propose how to quantitatively mitigate the leakage issue, either by sanitizing the training data or by mixing public and private data. All reviewers agree that this paper provides valuable framework that can systematically explore privacy leakage for a class of retrieval-augmented LLMs and mitigating solutions. To further improve the study, it is encouraged to navigate other types of retrieval-augmented language models and other types of utility, especially multi-task solving capabilities.
Decision: Accept-Main