Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions
Authors: Michele Miranda
Elena Sofia Ruzzetti
Andrea Santilli
Fabio Massimo Zanzotto
Sébastien Bratières
Emanuele Rodolà
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) represent a significant advancement in artificial intelligence, finding applications across various domains. However, their reliance on massive internet-sourced datasets for training brings notable privacy issues, which are exacerbated in critical domains (e.g., healthcare). Moreover, certain application-specific scenarios may require fine-tuning these models on private data. This survey critically examines the privacy threats associated with LLMs, emphasizing the potential for these models to memorize and inadvertently reveal sensitive information. We explore current threats by reviewing privacy attacks on LLMs and propose comprehensive solutions for integrating privacy mechanisms throughout the entire learning pipeline. These solutions range from anonymizing training datasets to implementing differential privacy during training or inference and machine unlearning after training. Our comprehensive review of existing literature highlights ongoing challenges, available tools, and future directions for preserving privacy in LLMs. This work aims to guide the development of more secure and trustworthy AI systems by providing a thorough understanding of privacy preservation methods and their effectiveness in mitigating risks.

Published in Transactions on Machine Learning Research (01/2025)
Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions
Michele Miranda
mirenda Odi www.ooo..............
Sapienza University of Rome Translated
Elena Sofia Ruzzetti
elena. hanaa.ooffu.uuaaatiiinnmmmmoee
University of Rome Tor Vergata
Andrea Santilli
cantilli@ddi.umrrmaa
Sapienza University of Rome
Fabio Massimo Zanzotto
fabio. mas.imuu...mmmtttttoeinrron.eee
University of Rome Tor Vergala
Sébastien Bratières
sehastien&&transsated.com
Translated
Emanuele Rodolà
rodola Adii uniromal.
Sapienza University of Rome
Reviewed on OpenReview:
Abstract
Large Language Models (LLMs) represent significant advancement in artificial intelligence, finding applications across various domains. However, their reliance massive internet- sourced datasets for training brings notable privacy issues exacerbated in critical domains (e.g., healthcare). Moreover, certain application-speeffic scenarios may require fine-tuning these models on private data. This survey critically examines the privacy threats associated with LLMs, emphasizing the potential for these models to memorize and inadvertently reveal sensitive information. We explore current threats by reviewing privacy allacks on LLMs and propose comprehensive solutions for integrating privacy mechanisms throughout the entire learning pipeline. These solutions range from anonymizing training datasets to implementing differential privacy during training inference and machine unlearning after training. Our comprehensive review of cxisting litcrature highlights ongoing challenges, available tools, and future directions for preserving privacy in LLMs. This work aims to guide the development more secure and trustworthy AI systems by providing a thorough understanding of privacy preservation methods and their effectiveness in mitigating risks.

---

Published in Transactions on Machine Learning Research (01/2025)
Contents
Introduction
Preliminaries
7
2.1
Large Language Models
7
2.2
Dillerential Privacy
2.3
Deep Learning with Difterential Privacy
3
Privacy Attacks
14
Training Data Extraction
Non-adversaria extraction
3.1.2
Adversarial prompting
3.2
Membership Inference Attacks (MTA)
18
3.2.
MIA with Thresholds
18
3.3
Model Inversion and Stealing
19
3.3.1
Model Output Inversion and Model Stealing
20
3 3.3.2
Gradient Inversion
21
3.3.3
Model Stealing
21
4
Privacy Threats at Inference Time.
22
4
Solutions for Preserving Privacy in LLMs: Data
24
Anonymization
25
4.1.1
Classical Anonymization
25
4.1.2
Anonymization with Differential Privacy
27
Solutions for Preserving Privacy in LLMs: Model
31
Training Large Language Models with Differential
................................
31
5.1.1
Pre-training with Differential Privacy
................................
32
5.1.2
Fino-tuning with Differential Privacy
................................
34
5.1.3
Parameter- Efficient Efficittt-- Fine- Tuning with Difterential Privacy
................................
36
5.2
Inference with Differential Privacy
37
5.3
Federated Learning with Diflerential Privacy
40
5.4
Machine Unlearning
41
Homomorphic Encryption
6
Tools & Frameworks
46
Limitations and Future Directions
50
2

---

Published in Transactions on Machine Learning Research (01/2025)
Conclusion
51
A Going Further
82
B Summary Tables
84
C Discussion Federated Learning
87
Federated Learning with Local DP
89
3

---

Published in Transactions on Machine Learning Research (01/2025)
1 Introduction
Large Language Models (LLMs) represent significant advancement in artificial intelligence (AT), demon- strating remarkable capabilities in understanding and generating human-like text, and fucling innovation However, their reliance on massive internet-sourced datasets for training raises significant privacy concerns, considering that these modols are prone to memorize segments of their training data. (Feldman 2020 Feldman & Zhang 2020).
across various domains (Brown et al. 2020; TTouvro el al. 2023 Zhao et al. 2023
This further exacerbated if we consider certain application-specíie scenarios (e.g. healthcare) that may require fine-tuning publicly available models on user private data (Tramer et al.
2024a). Since language
models are specifically trained generate text, this poses fundamental threat considering that they might inadvertently disclose memorized private data at any moment (Carlini et al. 2019 2021
Privacy is fundamental human right that encompasses the protection of personal information from unau- thorized access and misuse (United Nations 1948: Nissenbaum 2010). Preserving privacy in the context of language models thus means implementing measures that ensure the protection of personal and sensitive information from being exposed or misused, following the permissions granted by users to use those data, whether implicit or explicit (Brown ct al., 2022a). Preserving privacy not only crucial to preserve users and public trust in AI technologies but also a legal requirement in many countries such the EU with the General Data Protection Regulation (GDPR), the US with the Privacy Act, and China with the Personal Information Protection Law and Data Security Law (Weidinger et al. 2021 Solaiman et al. 2019 Veale ct al. 2018). These regulations may include additional guarantees, such as the 'right to be forgotten', which allows individuals to withdraw their consent and request the deletion of their personal data, posing an addi- tional challenge for LLMs as erasing a data point from a trained model might require retraining the model from scratch or discarding the entire model (Zhang et al. 2024b).
Understanding the distinction between privacy and copyright is crucial in this context. Privacy focuses on protecting and controlling personal information, ensuring that individuals' sensitive data remains confidential and is not shared or exploited without their consent. Copyright, on the other hand, pertains to the legal rights of creators over their original works, such as text, music, or art, and governs how these works can be used, distributed, and reproduced by others [Torremans] 2007 Karamolegkou et al. 2023). While privacy is about protecting personal data from unauthorized access, copyright protects intellectual property and ensures rights holders receive recognition and compensation for their work. This study focuses on addressing privacy issues related to the potential exposure of personally identifiable information (PIT) in LLMss rather than exploring copyright infringement or intellectual property rights.
PII
Personal Identifiable Information (PII) refers to any data that can be used to identify a specific individual. This may include direct information such as a person's name, address, email address, social security number, phone number, genome, fingerprints, and face, as well as any other data that, when combined, could lead to the identification of an individual (National Institute of Standards and
Technology 2017 General Data Protection Regulation 2016
This survey aims to provide a holistic view of the problem of preserving privacy in LLMs by combining the current privacy threats and the current available solutions and tools to implement them. We believe this approach presents both aspocts tandem and is an essential resource for rescarchers and practitioners seeking a comprehensive understanding of this field. Specifically, this survey targets (i) readers interested in understanding the current privacy risks associated with LLMs ($3) (ii) readers seeking to learn about the available solutions to defend against privacy attacks 4 and (iii) readers looking for tools to make their
LLMs resistant to privacy attacks
The survey is organized as follows and divided into two main categories: privacy attacks ($3) and solutions 5)
The taxonomy in Figure I provides structured view of our survey on preserving privacy LLMs. We provide a preliminaries section for readers unfamiliar with the terminology ($2) that explains the basic concepts related to language models, differential privacy, and federated learning.
4

---

Published in Transactions on Machine Learning Research (01/2025)
Non-adversarial extraction 3.1.1
Training Data Extraction $3.1
Adversarial prompting 33 1.1.2
Membership Inference
3.2
Thresholds 3.2.1
Model Output Inversion
3.3. 3.3
Privacy Attacks
Model Inversion and Stealing 3.3
Gradient Inversion 3.3.2
Model Stealing 3.3.
Preserving Privacy in Large Langnage Models (LLMss
Privacy Threats as Inference Time 3.4
Classical Anonymization 641.1
Privacy Data 64
Ancaymization with DP 1.1.2
Pre-tminnig with DP 6.1.1
Solutions 4
Fine-tuning with 5.1.2
LLMs Differential
Privacy (DP) $5.1
PEFT DP 6. 5.1
Inference with DP
Privacy in Model $
Federated Learning DP 5.3
Machine Unlearning
Homomorphic Eneryption
5.5
ARX. OpenData Anouymizer
Presidio
Data Anorymization
Spark NLP for Healthrare
Tools & Frameworks 46
Differential Privacy
private- Cenusteetaaneoormerss dp-trrusformerr Dill dep
Chorus
OpenDP
lew- shot geneeeatio ion.
Opacis
Fedeeraedd Learning
Flower.
pfl
TF Federated,
PySyh.
FATE
PaddleFI.
Figure 1: Taxonomy of Preserving Privacy in Large Language Models. Red indicates various attack techniques, and blue represents current possible solutions to preserve privacy by acting on the training data model. Finally, in orange we highlight the currently available tools to preserve privacy.
Privacy allacks discuss how it's possible to extract training data from a model and encompass Training data extraction ($3.1), membership inference ($3.2), and model inversion and stealing ($3.3). Training data extraction involves techniques such non-adversarial extraction ($3.1.1) and adversarial prompting ($3.1.2). where attackers can potentially extract sensitive information from the training data. Membership inference (3.2 utilizes shadow models or thresholds ($3.2.1) to infer whether specific data points were part of the training set. Model inversion includes model output inversion ($3.3.1) and gradient inversion ($3.32) methods that can reconstruct the input data from model outputs while Model stealing is proposed to reconstruct a model's parameters from output ($3.3.3.. However, as prompting LLMs with private input such as in-context demonstrations an increasingly popular approach, thrcats inference time are also discussed 3.4
Solutions to these privacy challenges are categorized into privacy in data ($4 and privacy in model ($5) Privacy in data includes classical anonymization techniques to anonymize data before it is used in train- ing (11.1.) and anonymization with differential privacy (DP) (44.1.)). which ensures that individual data points cannot be distinguished from the dataset. Privacy in model encompasses the application of DP during different phases such as pro-training (55.1). fine-tuning (55.1.2, Paramcter-Effieet
Fine-Tuning (PEFT) ($5.1.3), and inference (...2.. Federated learning with DP ($5.3) is implemented maintain privacy across distributed datasets, and machine unlearning
5.4) involves techniques to effectively remove specific data
5

---

Published in Transactions on Machine Learning Research (01/2025)
from trained models. Homomorphic Encryption ($5.5) involves cryptographic methods that allow com- putation on encrypted data without decrypting it. Finally, we present available tools and frameworks to implement privacy in language models (
In summary, this survey serves as a critical resource for researchers and practitioners, providing insights into the current privacy threats LLMs and the currently available solutions address these challenges. By systematically categorizing privacy issues and their corresponding mitigation strategies, this paper highlights the importance of preserving privacy in LLMs and paves the way for future advancements in the field. All the papers in this survey are also available at the following GitHub repository

---

Published in Transactions on Machine Learning Research (01/2025)
2 Preliminaries
In this section, we present preliminary concepts essential for comprehending the subsequent content of the document. Readers already acquainted with these concepts may proceed directly to the next section. For a comprchensive introduction to the concepts of Machine Learning and Deep Learning, we direct the reader to [Bishop (2006) and Goodfellow et al. (016)).
In this survey, we use the term large language model (LLM) to indicate any model trained on broad textual data that can be adapted (c.g., via fine-tuning or in-context learning) to a wide range of downstream tasks [Bommasani el al., 2021). While the focus of the survey is mostly on generative LLMs (Zhao et al. 2023 models that are capable of solving a broad set of tasks by generating text, such as GPT-3 and GPT-4 it is not uncommon to encounter smaller models like BERT in our discussion. We have chosen to include some of these models when we believe that their associated findings, both in terms of potential threats and proposed solutions, remain still relevant and generalizable to larger models nowadays.
2.1 Large Language Models
Large language models are deep learning systems trained on vast amounts of not-annotated text data to predict the likelihood of word sequences. Currently, most of the generative language models trained in an autoregressive fashion, meaning they generate text by predicting each word soquentially based on the previous words in the sequence usually from left to right. A model trained in this way is capable of generating and manipulating language effectively, which has been shown useful across a broad set of tasks and scenarios (Zhao et al. 2023). However, this was not always the case. Other popular techniques train LMs by masking certain words a sentence and then prodicting those masked words based on the surrounding context, like in the case of BERT (Devlin et al. 2019 . These models are lypically much smaller and less capable than modern LLMs (for instance, BERT large has 340 million parameters). Language modelling is not new task: the groundwork for it was laid in 1948 by Claude Shannon (Shannon 1948). and the task was extensively investigated for decades, but the recent advances with the introduction of the Transformer architecture (Vaswani et al. 2017) and the rollout of GPT-3 in 2022 (Brown et al. 2020 shed new light on the task and this new class of very big language models, namely large language models. Training a Large Language Model from scratch, sometimes referred to as pre-traiming, is usually very computationally intensive given the scale of these models and requires several resources generally available only to a few organizations in the world (Sharir et al., 2020 Hadi et al. 2023). For example, to give sense of this scale, GPT-4 is rumoured to have around 1.8 trillion parameters with an estimated training cost of around $63 million at the time of its release.
With these models, a new training paradigm arose: the ccc-traiiing/fnnttuiiin paradigm. Pro-training is usually on the next token prediction lask, while fine-tuning is on the downstream task (which can still be the next token prediction, but can also be something else like a simple classification task). Organizations building these large language models typically pre-train them on extensive text corpora and then fine-tune them for specific tasks before making them accessible to the public. White-box access allows users to download the model's weights and see or modify the model's internal parameters and architecture, enabling detailed inspection and customization. In contrast, black-box access is usually offered through APIs or web interfaces, where users can input data and receive outputs without any visibility into the model's internal workings or the ability to modify its parameters. There's also a middle ground called grey-box access, which allows users a partial view and limited interaction with the model's 's internal workings. Users can gain some insights into specific aspects of the model, such as the architecture or certain parameters, but do not have full access to download modify the model's weights and internal components. This type of access might enable customization and tuning through pre-defined interfaces or tools, offering more flexibility and control than black-box access but not as much transparency or openness as whitc-box access. Rescarchers and scientists can efficiently adapt the models to their specific tasks and needs through fine-tuning, which uses a relatively small fraction of the original pre-training data and iterations. Since these are language models, it is possible to interact with them in natural language by feeding them sentences in input, which in this context are called prompts. This means that black box models can still be adapted by conditioning them with specific prompts, either in a shro-shot or few-shot fashion (Bronn et al. 2020 This process is known in-context
7

---

Published in Transactions on Machine Learning Research (01/2025)
3 Loss <X x)
Euput pertuabution
3 Labels
X2
Mast
Dutpl peraataiio
Label pemztotioo
Spet
latel
Nobe
3 Output
0Xtaaa0: CXX0XX0 octaact
Unembedding Matii
Oljecive fraciion Noiee
Objecivve pertetation
Gradient
Model
① Input
Notao pertefaatie
Embedding Marri
Figure 2: Diffcrential Privacy. One the left: classical DP applied to a neural network, picture from Baraheem & Yao (2022). On the right: DP applied on a Language Model (analyzed in detail in Section 5. Differential privacy can be applied by adding a perturbation (noise) on different positions of its architecture: 1) Input noise perturbation added to the input; 2) Gradient noise perturbation is added during training to the gradient update step; 3) Output noise perturbation is added to the output; 4) Labels noise perturbation is added to the labels used during training; 5) Loss noise perturbation is added to the loss function during training.
learning: in a zero-shot setting, the LLM takes only an instruction as input, while in few-shot setting, it also takes a few examples, significantly enhancing its conditioning.
Here, we offer the reader a comprehensive explanation of several terminologies frequently used in the context of language models throughout the paper. These explanations aim clarify key concepts and enhance understanding of the material presented:
Sampling: LLMs are trained to predict the next token (word or punctuation) when given a text (prompt). To do so, LLMs learn a probability distribution over every possible token. Sampling con- sists of selecting the next word from this distribution, given the current context. Various methods, such as greedy sampling, random sampling, top-k sampling, and top-p sampling, allow for different balances between coherence and creativity in the output. These techniques essential for tai- loring the model's responses, impacting the diversity and appropriateness of text generation across applications like chatbots and content creation.
Instruction tuning: Instruction tuning is a technique used enhance the performance of large language models (LLMs) by training them to follow specific instructions. It involves fine-tuning pre-trained model on dataset that includes tasks described in natural language, along with the desired outputs. This technique allows the model to shift from general next-token prediction to more specialized tasks (e.g., text summarization). By using instruction tuning, the model learns to interpret and execute tasks based on user-given instructions, thereby improving its ability to generate relevant and accurate responses given a user prompt.
PEFT (Parameter Efficient Fine-Tuning): Parameter Efficient Fine-Tuning is a class of methods to adapt pre-trained language models by altering only a small fraction of the model's paramoters. This approach maintains the general capabilities of the model while enabling customization to new requirements with minimal computational cost. and memory overhead. Tt's particularly useful for deploying models in resource-constrained environments or for applications requiring rapid adaptation to new data.
RLHF (Reinforcement Learning from Human Feedback): Reinforcement Learning from Human Feod- back is a technique used train large language models (LLMs) by incorporating human preferences into the learning process. After model has been instruction-tuned, its outputs evaluated by
8

---

Published in Transactions on Machine Learning Research (01/2025)
humans, who provide feedback on the quality and relevance of the responses. This feedback is used to fine-tune a reward model that assigns scores to the outputs based on human preferences. The main model is then fine-tuned using reinforcement learning, guided by the reward model, to better align with human values and enhance its performance on tasks requiring maancdd understanding and judgment.
2.2 Differential Privacy
According to Dwork & Roth (2014): 'Differential privacy describes a promise, made by dala holder, or curator, to a data subject: "you will not be affected, adversely or otherwise, by allowing your data to be used ony study or analysis, no matter what other studies, data sets, or information sources, are available." The fundamental goal of Differential Privacy is to enable the extraction of useful insights from data while minimizing the potential impact on any individual's privacy within that dataset. The core idea is to introduce controlled noise or randomness into the analysis results in way that prevents the identification of specific individuals. In simpler terms, when system or algorithm is differentially private, the inclusion or exclusion of any individual's data should not significantly alfect the overall outcome or conclusions of the analysis. There are several kinds of Differential Privacy definitions, but the original one is (e, Diifferrent Privacy.
(eff)--ifferentiall Privacy
A randomized algorithm M with domain N is (c, (eiff-iiiierenntiall private if for all S C Range(M) and for all x, E NI* such that | yll I 1
Pr[M(a) e s] < Pr(M()) E 6
The formal guarantee provided by (s, Di-Diffellltial Privacy is that for any two datasets x and x' that differ by only one clement (these often referred to 'adjacent datascts"), and for any subset of outputs S of the algorithm's 's range Y, the probability that the algorithm outputs something in S given is not substantially higher than given x!.
c is the parameter that measures the privacy loss associated with a query in the system. A smaller c provides stronger privacy guarantees. The use of e in the inequality indicates that the probability of a certain output does not increase by more than a multiplicative factor of e when any single individual's data added or removed.
The privacy budget is the total loss of privacy sustained by the system during the execution, and it can be computed by compounding the privacy loss indicated by over the total number of executions/iteration This is done with the composition theorems, which are explained in more detail later in this section.
is the parameter that allows for a small probability of failure of the privacy guarantee. Ideally, 8 should be very small, close to 0, which indicates a lower chance of the algorithm failing maintain privacy. When 8 is 0, it defined s-differential privacy, which is the strongest kind of differential privacy.
This means that for all subsets S of the output range Y, the probability that the algorithm on input gives output in S is at most raised to the power e times the probability that the algorithm on input x' gives an output in S, plus the small probability 6.
The simplest method to achieve differential privacy in a query is by adding random noise to its answer. The challenge lies in balancing the noise added to meet differential privacy standards without rendering the answer too noisy be useful. To assist with this, foundational mechanisms have been developed the field, detailing the type and amount of noise required. One such mechanism is the Laplace mechanism.
9

---

Published in Transactions on Machine Learning Research (01/2025)
Laplace Mechanism
According to the Laplace mechanism, for function f(z) which returns real number, the following definition F(x) satisfies e-differential privacy:
Af
= - + Lap
where Aʃ the sensitivity of S, and Lap(b) denotes sampling from the Laplace distribution with centre 0 and scale b.
Sensitivity Af refers to the maximum change in the output of a function or query when a single individual's data the input dataset is added or removed. It quantifies how much a query's result can vary with the smallest possible change in the dataset, which is crucial in determining the amount of noise that needs be added ensure differential privacy. A function with high sensitivity requires more noise to protect individual privacy, while function with low sensitivity requires less. Sensitivity is key factor in calibrating the noise for mechanisıns like the Laplace mechanism to ensure that individual data points do not significantly affect the output, thereby maintaining privacy.
Global Sensitivity
For a function f - IR mapping datasets (D) to real numbers, the global sensitivity of f is defined as follows:
GS(f) max 2, 2,0''d(a,'' <<==
(x')
Here, d(x, x²) denotes the distance between two datasets and x', with the notion that two datasets are neighbours if their distance is 1 or less. The way this distance is defined significantly impacts the privacy definition we achieve, and we'll explore the dataset distance metric in more detail later.
Global sensitivity is defined such that for any two neighbouring datasets æ and x', the difference between f(x) and f(x') does not exceed GS(f). This referred to as 'global' sensitivity because it independent of the specific dataset being queried, applying to any pair of neighbouring x and x'.
Based on the noise added, there are several mechanisms in the literature. The Gaussian mechanism is an alternative to the Laplace mechanism, which adds Gaussian noise instead of Laplacian noise.
Gaussian Mechanism
The Gaussian mechanism does not satisfy pure e-differential privacy but does satisfy (e, d)dfferrrnt privacy. According to the Gaussian mechanism, for function f(z) which returns number, the following definition of F(x) satisfies (c, 3diffferrent privacy:
F(x) = f(x) + +N0,00 where As 22og(
where Af is the sensitivity of and (f),,²)) denotes sampling from the Gaussian (normal) distri- bution with mean 0 and variance o², Note that here (and elsewhere in these notes), log denotes the natural logarithm.
The primary mechanisms we've discussed (Laplace and Gaussian) are designed for numerical outputs, adding noise directly to the results. But what if we need to provide an exact answer without injecting noise yet still mainlain differential privacy? The exponential mechanism addresses this by enabling the selection of the "best" element from a set while ensuring differential privacy. The "best' element is identified by using a scoring function that assigns scores to cach item within a given set of options. The exponential mechanism ensures differential privacy by approximately maximizing the score of the chosen element. To preserve privacy, it might sometimes select element that doesn't 't have the highest score.
10

---

Published in Transactions on Machine Learning Research (01/2025)
Exponential Mechanism
The exponential mechanism satisfics e-differential privacy:
1. A set R possible outpuls is selected
2. A scoring function is specified u D R with global sensitivity Au
3. The exponential mechanism outputs R with probability proportional to:
cu(x,r) r)
exp
2
The main practical distinction between the exponential mechanism and previous mechanisms like the Laplace mechanism that the exponential mechanism's 's output is always an element from the specified set. This is particularly advantageous when choosing item from finite set, where noisy result would be inappro- priate.
Another prominent definition of Differential Privacy, which actually relaxation of the original definition,
is Rényi Differential Privacy (Mironov 2017).
Rényi Differential Privacy
A randomized algorithm M D - R. is said to have c-Rényi differential privacy of order o, (a, r)PDD for short, if for any adjacent D, D' E E it holds that:
Da(M(D)LM(()) S6
S
Here, (PIlQ) represents the Rényi divergence of order a between two probability distributions P and Q, and is defined as:
Da(l)QQ)
1
log Σ P()) T
for all 1, where X' is the support of the distributions P and Q. The parameter here controls the scnsitivity of the divergence mcasure to differences between P and Q, with higher values of focusing events that are more probable under P than under Q. The parameter still quantifies the privacy guaranlee, with smaller values indicating stronger privacy.
By providing a mathematically rigorous definition of privacy guarantecs, Diffcrential Privacy has become a key principle for designing privacy-preserving algorithns, especially in scenarios where personal data is involved. From this mathematical definition, some interesting properties arise. Amoug these, there are composition theorems that allow the computation of the exact privacy loss through many applications of the DP algorithm, which corresponds to the privacy budget. Sequential composition theorem (Kairouz al. 2015) applies when multiple differentially private mechanisms are applied to the same dataset in sequence. The theorem quantifies how the privacy guarantees accumulate over the sequence of operations. Parallel composition theorem (McSherry) 2009) applics when multiple differentially private mechanisms are applied to disjoint subsets of the dataset. This theorem shows that the worst-case privacy guarantee of any single mochanism governs the privacy guarantec of the overall process. Differential Privacy also cxhibits Post- Processing property that states that once data has been processed through a DP mechanism, the output retains its privacy guarantees regardless of any further processing.
Differential privacy was born a statistic analysis technique for databases. Initially, it was used to query databases without compromisnng the privacy of anybody enrolled, and that is why we discussed queries and compounding privacy over them to compute the total privacy loss or privacy budget. Luckily, all this came in handy recent years, with the worldwide diffusion of deep neural networks, when differential privacy was applied them. That when databases became datasets and queries became (mostly training) iterations.
We will more in detail what the result. of this transition in the next paragraph 2.
11

---

Published in Transactions on Machine Learning Research (01/2025)
2.3 Deep Learning with Differential Privacy
Disferential privacy was initially devised for database analysis, but it is possible to apply it to deep learning models in several ways. In practice, since we primarily work with deep learning models, we will often refer to datasets rather than databases. However, this distinction does not entail any practical difference in the context of our discussion. Figure(2.1]shone how Differential Privacy can be incorporated in 1) the imput data, 2) the gradients, 3) the output data, 4) the labels or even in the loss function 5). The most used technique is by training using DP-SGD (Differentially Private Stochastic Gradient Descent), while DP-Adam is less used we discuss the reasons for this in $5.1.2 Both of these techniques involve gradient manipulation, falling under point 2) in Figure 2.1 Initially presented by [Song et al. (2013). DP-SGD is variation of standard Stochastic Gradient Descent (SGD) that enhances privacy by incorporating gradient clipping and adding Gaussian noise. In DP-SGD, gradients are first clipped to a maximum norm to limit the influence of any single data point. Then, Gaussian noise is added to the average of these clipped gradients, ensuring that the updates to the model parameters maintain differential privacy by obscuring the contribution of individual data points. It is possible to consider the several layers separately in order to have different clipping thresholds, and the amount of noise should be calibrated over the sensitivity of the function (in this case, the network), explained by Dwork et al. (2006b). The sensitivity of a function f indicates how much the function's output changes in response to changes in its input. In DP, there is also focus on privacy costs, so DP-SGD needs a way to compute them. In Abadi et al. (0166), along with a refined version of DP- SGD, the authors present a novel technique for privacy accounting during training, which is called moments accountant. The composability property of DP allows us to compute the privacy cost at every iteration and then sum them up. Privacy cost is computed in an asymptotic way, and while the mathematical details are beyond the scope of this survey, it is worth noting that moments accountant saves a non-negligible factor in the computation with respect to previous techniques, allowing to guarantee lower privacy budget with the same magnitude of injected noise.
Also gradient clipping is an important step in the DP learning process. Gradient clipping can be done in several ways. It is possible to use a per-sample clipping (Lee & Kifer 2021) and this can even be done automatically (Bu et al. 2023ab, or a flat clipping style (also called all-layer) (Abadi et al. 2016 or something in between, like group- wise clipping (McMahan et al., 2017b). As stated by Bu et al. (2024), the first leans towards better memory accuracy while the second favours memory efficiency. Another option to enforce DP is PATE (Private Aggregation of Teacher Ensembles) (Papernot et al., 2017) and its variations (Papernot et al. 2018 Tian et al. 2022). In this method, several teacher models are trained on private data, and then a student model is trained on the predictions given by a majority vote with added noise made by the ensemble on public data. In this way, the actual model never sees the data, and the individual models (and, for extension, their data) cannot influence the outcome too much. A new, alternative way to enforce DP, as shown by Du et al.] (203b)) is to perturb only forward pass embeddings, and according to the experimental evidence, it offers stronger performance with a lower vulnerability to attacks and more efficient memory usage. DP has been shown to work in several settings like NLU (Qu et al. 2021 Dupuy et al. 2022), image classification (De et al. 2022), diffusion models for image generation (Dockhorn et al. 2023 Ghalebikesabi
ct al. 2023). neural machine translation [[gamberdiev et al. 2024h and, of course, language modelling (Dinh & Fioretto 2023) and LLMs in particular (Carranza et al. 2024). All these related works prove the interest of the scientific community in the problem and the commitment to finding an effective solution. Differentially private models are alrcady in place with several solutions, and they work to different extents based on modcls, data and privacy budgets.
Federated Learning Federated learning (McMahan et al. 2017a) is machine learning approach that enables model training across decentralized devices or servers holding local data samples without exchanging them. Instead of centralizing all data in one location, the model is trained collaboratively across multiple devices or servers. Each local device computes updates to the model based on its local data, and only these updates are shared and aggregated to improve the global model. Federated learning aims to preserve privacy and security by keeping sensitive data localized while still benefiting from the collective knowledge gained during the collaborative training process. It allows to be involved in the training of a model without fully trusting the external entity training the model (which can be government or a company), because you are not giving up your actual data. As we will see more in detail later, this technique does not really manage to
12

---

Published in Transactions on Machine Learning Research (01/2025)
protect privacy by itself because gradient updates, even after aggregation, can be reversed to recover training
data 3.3.2 A big advantage of federated learning is allowing to train better models thanks to the higher
availa bility of high-quality local data, like the case of next word prediction for smartphone keyboards (Hard
ct al. 2018
Zhao et al, 2019a) a communication protocol that enables several parties to jointly compute a function over their private inputs, without revealing those inputs to each other. This way, each participant's 's data remains confidential while still allowing for the collective computation of the function. It is particularly useful in scenarios where data privacy is crucial, yet there is need for collaborative computation, such in privacy- sensitive data analysis and secure voting systems. There are several ways to enforce SMPC, like secret
Secure Multi-Party Computation Secure Multi-Party Computation (SMPC) (ao
1982
sharing (Shamir, 1979 Blakley 1899), homomorphic encryption (Gentry 2009 Kim & Yun 2021 Joo &
Yun 2014), zeeoeeeroooree knwwledgeee proofs (Goldreich & Oren 1994 Feige et al. 998) and oblivious transier (Rabin
2005 Even et al. 1985: Brassard et al. 1986 Chailloux et al. 2013 Cheong al.) 2009). Homomorphic
encryption is particularly interesting in this context as it allows the computation of function on encrypted
data. This makes it possible to perform training and inference of machine Icarning models encrypted
data. While these techniques can enforce strong data security, they also add considerable computational overhead that makes them difficult to apply in situations where the learning process is already heavy.
13

---

Published in Transactions on Machine Learning Research (01/2025)
3 Privacy Attacks
As the training data becomes huge; it is increasingly urgent to understand whether models tend to expose the data on which they have been trained since they are more likely to capture sensitive information. Moreover, the leakage of the original training data is not the only possible threat. In fact, the versatility of LLM is enhanced by the use of in-context demonstrations of the task that a user wants to perform (Brown et al, 2020j. While being a useful technique, unconscions use can cause sensitive information - that should not be disclosed to be included in these examples in conlext as well. For commercial systems, those data can also be leaked by exploiting vulnerabilities of the systems storing them (OpenAI 2023). Even cyber security measures are insufficient since those data can be used in further training steps and subsequently leaked by a future version of the model. This aspect is also concerning for data with commercial value from business perspective (Coles) 2023 and companies have decided, in some cases, to warn their employees about the responsible use of LLMs Kim 2023).
A broad research area is devoted to understanding whether deep learning algorithms memorize training examples and, hence, can be used to reconstruct (potentially sensitive) training information. While some works question the effect of memorization during generation mcaning that texts generated starting from prompls extracted from training are different from the original ones (McCoy al.) (023) the discussion around memorization and its consequent risks is growing larger models become more popular. In some cases, memorization is discussed as the key to good performance since long-tailed distributions often char- acterize data. Hence, for a model, it is hard to distinguish between outliers and actually informative data points (Feldman & Zhang 2020). However, memorization of training data by a model allows an attccer to reconstruct sensitive information or to understand whether a certain example was used during training. It has boen shown by Carlini ct al. (2023a) that it is possible reconstruct training data with only black-box access.
For LLMs, it might be sufficient to input the right prompt, which is not necessarily an adversarial one (Carlini et al. 2021), in order to condition a model to generate sequences that are exactly memorized from the training data. Áttacks that allow verbatim extraction of training data are generally referred to as Training Data Extraction attacks. When LLMs are fine-tuned with Reinforcement Learning with Human Feedback (RLHF), they tend to be more robust and avoid revealing the training set examples and, in particular, revealing sensitive information. However, some adversarial prompts have emerged to condition model to show some unintended behaviour. Since sensitive information is memorized by model, RLHF is used to prevent the model from disclosing such information, but it does not ensure that new attacks cannot retrieve it. We discuss both non-adversarial and adversarial prompling in 1
In some cases, an attacker may not only have access to a model but also to some data possibly included in the training phase of the target model. In this scenario, a Membership Inference Attack (MIA) can be performed: the attacker aims to claim that the information is included in the training set. of a threatened model. The initial formulation of those attacks (Shokri et al., (017)) has also been extended against larger models and Transformer-basee Language Models in particular. Since some sequences are memorized by a model, overfitting over them can be detected and used to attest to the presence of certain data in the original training data. The effectiveness of MTA is also discussed in light of the training practices generally used to develop LLMs, such as training for a single epoch on a huge amount of data, which may influence the measure of overfitting used in these attacks. In $3. we discuss that kind of attack.
When private data owned by different holders need to be used to train a joint model on them, protocols like federated learning ($2.3) seem to be the appropriate solution: in this scenario, data are not distributed across multiple servers, only model snapshots and updates are shared. However, model outputs and gradients can also be used to infer the input that justifies the observed model behaviour. The model outputs or gradients can be inverted to obtain the sequence that justifies them, allowing an attacker to reconstruct private data without them being explicitly shared. We discuss Model Inversion Atlacks as a method to track back model input from the model's output or gradients in (3.3). Moreover, from those rather opaque sources, like model outputs, an attacker can gain information regarding a model's parameters: those attacks, called Model Stealing attacks, while still not able to reconstruct entire LLMs, can potentially be directed against A API-gated models, making them less opaque and, hence, more vulnerable against privacy attacks.
14

---

Published in Transactions on Machine Learning Research (01/2025)
+12
Model Ourguss
Ve Gradients
Language Model
Language Model
Language Model
Malicious Prompt
included not
Data indiuded
User Prompt
a)
b)
C)
Figure 3: Attacks against LLMs aim to reconstruct private training data. Training Data Extraction Attack (a): The attacker has no internal access to the model (black box) and can only design a malicious prompt to force the model generation to reveal private and potentially sensitive training data. Membership Inference Attacks (b): The attacker has access to data that might have been included in the training set. The goal is to determine whether specific data points were indeed part of the training set. This is achieved by detecting behavioural differences in the model's responses data that were included in the training set versus data that were not. Model Inversion Attack (c): In black-box or grey-box setting, the attacker observes the model's outputs gradients on unknown inputs. The attacker then uses this information to reconstruct the original input from these seemingly opaque sources.
Finally, notice that since fine-tuning is more expensive than in-context learning, many systems use in- context demonstration to specialize an LLM on specific tasks. Despite the flexibility of this method, some risks may emerge from using private information for in-context demonstrations. For example, a system that answers user query by leveraging a prompted LLM could be vulnerable to privacy attacks performed by a malicious user (sce 3.4 for further details.)
3.1 Training Data Extraction
In Training Data Extraction attacks, an adversary aims to reconstruct training data having only access target Language Model. In particular, those attacks aim to extract the entire example by querying the target model (Carlini et al. 2021). Retrieval of examples is based the assumption that the target models memorized them, and the definitions of memorization underlying these attacks may vary. However, the various definitions revolve around the key concept that a textual example or part of it can be extracted from a language model. In the original definition of Carlini et al. (2021). , an example is defined to be extractable if there is a prefix used to condition the generation the model so that the string can be exactly generated from the model. This definition is first introduced by Carlini et al. (2021) and then largely adopted with
slight modifications (Huang et al. 2022 Carlini al. 2023a Nasr et al. 2023
Training Data Extraction
Extracting Training Data consists of conditioning an LM to the right prompt so that it will likely generate a training example to complete it. Formally, a textual example x is extractable from LM g if there exists a prompt such that: x = arg max, g(z'pp) Once the right prompt has been identified, this procedure can be used to reconstruct training data, and hence, it can be used to extract sensitive information like Personally Identifiable Information (PIT) from a model.
15

---

Published in Transactions on Machine Learning Research (01/2025)
3.1.1 Non-adversarial extraction
Using training data extraction techniques, it has been shown that Language Models are prone to reveal private information when prompted with sentences requiring disclosing such information. While trying to recover memorized training examples, Carlini et al.) (202I) demonstrated that GPT-2 tends to memorize personal information such as Twitter handles, emails, and UCIDs (Universal Unique Identifiers). In particular, in their experiments, the language model generation conditioned to some prompts that the attacker predefined: to maximize the probability of success of the attack, they choose prompts that are likely to be in the training data, like samples from the Common Crawl D Then, once the generation is complete, the attacker tries to figure out which sequences are most likely to be generated as result of the memorization, that is, which of the generated strings exactly matches the training data. In order to identify sequences that are more likely to be generated because of memorization, Carlini et al. (2021) compare the likelihood that is assigned to each sequence by two models: one is the model under attack - GTP-2 - and the other is a reference model, another LM. The attacker retains as possibly memorized sentences only those that have a high likelihood in the target model when compared to the likelihood assigned by the reference model. This is form of Membership Inference Attack $3.2 Finally, once the allacker has retained a subset of strings that are likely to be generated due to memorization, the success of the attack must be verified by checking whether these high-likelihood sequences are actually part of the training set. However, for this step, Carlini et al. (2021) could only manually verify the effectiveness of the attack by searching the Internet for matching sequences, as the tested model did not have accessible training data.
The choice of the right prompt is crucial in this setting: in fact, when prompted with a prefix already seen during training, models tend to complete the prompt with the rest of the training sequence (Carlini et al. 2023a Huang et al. (2022) further explore this direction and show that conditioning model to prompt already observed during training can lead to Icakage of PII, like emails, and that is more effective than designing new, previously unseen prompls. In particular, they distinguish between memorization and association. A model memorizes personal information if a prompt exists- already part of the training data for that model- that Icads the model to generate personal information. The association roquires that a prompt not necessarily observed during training and containing some reference lo an individual exists that can lead to the generation of PIT by the model. Huang et al. (2022) show that the former approach is more effective than the latter, quantifying the number of e-mails that a model from the GPT-Neo family can predict when conditioned by the two different kinds of prompts. Moreover, they notice how these attacks get. more effective as the size of the models increases, and - as also noted in as also noted in Carlini et al. (2023a) and Nast et al. 2023 larger models tend to be more exposed to training data extraction attacks.
Those attacks can also be used to reconstruct the dataset used to fine-tune the model. To reconstruct personal information from a generative language model like GPT-2, Tukas et al. (0023) propose to solve a masked language modelling task to reconstruct masked personal information from a sentence, conditioning a model on both the prefix and suffix of the masked information: they call this approach PII Reconstruction. However, the attacker must gain access to masked training data before performing the attack. Panchendrarajan & Bhoi 20211) reconstruct the training data of finc-tuned GPT-2. This work stems from the observation also shared by Membership Inference Attacks as described in $3.2- that sentences that have been used in the fine-tuming phase will be given different probabilities ies by the fine-tuned version of a model and by the base model. They generate candidate sentences by starting the generation of the fine-tuned model with an empty string. Then, they observe the generation of the base model over the same sentence. Results show that it is possible to reconstruct about 75% of informative words in the fine-tuning dataset.
These experiments were the foundation for exploring other datasets and larger models. Nasr et al. (0023 demonstrate that the method proposed by Carlini et al. (2021) is more effective than initially thought: querying open-source models, they were able to verify the success of the attack procedure by accessing the training data only to verify the attack. They performed the attacks on models belonging to GPT-Nco, Pythia and RedPPaamma-INCTT INCITE families and using as initial prompts samples from Wikipedia. [Wang et al. (2024a) explore the leakage of training data of OpenAl's GPT closed models evaluating the accuracy of information extraction of sensitive information contained in pre-training data. The private information they
Phttps//sommmoncrav
16

---

Published in Transactions on Machine Learning Research (01/2025)
are able to extract is emails from the Enron Email dataset (Kimm & Yang 2004). They hypothesize that this dataset is used during the training phase of GPT-3.5 and GPT-4 since it is included in the Pile dataset (Gao et al, 2020): while no information is available on the training set of those models, the Pile was used in the pre-training of the GPT-Nco family of models and hence, possibly, also in the closed models under investigation. They adopt the same prompls proposed by Huang el al. (2022) and show results also in few- shots prompting settings. The experiments demonstrate that GPT-3.5 and GPT-4 can predict, respectively, around 5% and 4% of the email addresses accurately. Moreover, in the few-shot prompting setting with emails extracted from the same dataset, these models accurately predict up to 48.19% of emails correctly. The same trend is not observed for prompts that contain emails that have a different domain from the target one, suggesting the importance of the right choice of demonstrations for training data extraction attacks. While LLMs are generally more resistant to attacks where association is required, the work of Shao el al. (2024) shows that it is possible to recover 3% of the training emails from a 20 billion parameter model with attacks that require association, mcaning that the extraction procodure does not rely on prompts already observed during training. It has also been proposed to extract largeted examples from a model pre-training data: Zhang et a (2023b) propose
attack to recover a certain suffix given a precise prefix known to be in the pre-training data. They suppose that the attacker can also obtain some pairs of prefixes and suffixes and train an attack model on them. They tune soft prompt embeddings and then estimate the confidence that a generated suffix is, in fact, as it is in the corresponding prompt in the pre-training data. A similar approach, based on soft prompt tuning, is also adopted by (Ozaayi et al. (2023). As the size of the models increases, this type of attack appears to be more effective (Nasr et al. 2023). Moreover, the same techniques have also been discussed for other models, like diffusion models (Carlini et al. 2023b).
As concerns about the potential leakage of private information via training data extraction attacks grow, nunber of tools are being investigated to assess and potentially prevent the leakage of private information by LMMs before it spreads. On the evaluation side, ProPILE (Kim et al., 2023) is a tool designed to test models with black-box and white-box attacks against their tendency to release PII: the black-box approach is designed to allow data subjects to test whether their PII is leaked; the white-box approach is based on soft prompting and allows providers to conduct their model surveys to prevent leaks of private information. Differential privacy (see S for further details) is also effective against the memorization of private information and can successfully defend against these attacks. Downcy et al. (2022) demonstrate that by hand-crafting canary sentences containing sensitive content in the training data, the last two words of each canary are designed to contain the most sensitive information, and the LM is asked to complete the canary. The GPT model they train with Differential Privacy with some loss in model performance successfully preserves privacy.
3.1.2 Adversarial prompting
There have been initiatives to ensure LLMs are aligned with human values and used appropriately. For example, the OpenAl usage policy requires that users do not use the models to compromise the privacy of others, including collecting or generating personal data. Morcover, OpenAT's ChatGPT has been aligned with Reinforcement Learning from Human Feedback (Ouyang et al., 2022 and avoids answering questions regarding its training data. In this scenario, the techniques based on non-adversaria prompts may not be effective: Nasr et al. (2023) discuss how ChatGPT tends not to expose training data when prompted with prefixes that are likely seen during training. For these reasons, some attacks against LLMs define adversarial prompts that force the model to diverge from its originally intended use. For example, asking the model to repeat word indefinitcly, Nasr et al. (2023) notice that ChatGPT diverges and starts emitting training data, also containing personally identifiable information.
A new form of adversarial prompt attacks is jailbreak prompts. These prompts are designed to cireumvent safeguards and manipulate LLMs into generating harmful content and are largely used agaist closed LLMs alligned via RLHF. In some cases, a goal hijacking is performed, meaning that the original goal of a prompt is changed to a new goal, like generating a target phrase, by injecting malicious text in the original prompt (Perez & Ribeiro 2022). Shen et al.] (2024) characterize a large set of jailbreak prompts and evaluate their effectiveness against different LLMs. They notice that most of these prompts encourage the model to act like another fictional assistant, such as DAN (short for Doing Anything Now), which has no limitation in
17

---

Published in Transactions on Machine Learning Research (01/2025)
the answers. Along with other potentially harmful scenarios, they show that this type of prompt allows user to receive suggestions on how to collect personal information.
In this context, Li et al. (2023c) focused on the usage of adversarial prompts to pose privacy threats: they propose a multi-step jailbreak prompt to extract personal information, such as emails, based on Chain-of Thought (Kojima et al. 2024). They study the effect of adversarial prompting against ChatGPT. They first notice that the model tends to avoid sharing private information if direct data extraction prompts are used. Then, they propose the Multi-step Jailbreaking Prompt in three steps: they assume the role of the user to imput the jailbreaking prompt, then act as the assistant to confirm that the jailbreak mode is enabled Lastly, they play the role of the user and query the assistant with non-adversarial prompt. They show that more frequent emails are less difficult to recover: experiments on the Enron dataset show that more than 50% of frequent emails in the dataset can be extracted.
3.2 Membership Inference Attacks (MIA)
Membership inference attacks (MIA) Shokri et al. 2017 are currently de facto standard when it comes to
verifying model privacy (Murakonda & Shokri 2020). In this kind of attack, the attacker has access to (1) some records (supposed to come from a similar distribution of the training data) and (2) to the model, which could be provided as an API, in a black box setting, or be fully available as a white box. The aim of the attacker is to find out if these records (1) are actually part of the training set for that model (2). The initial formulation of this attack by Shokri et al. (2017) required the attacker to build several "shadow models' that mimic the original model, trained datasets that may resemble the distribution of the original one. Then, the attacker trains classifier on the shadow models' outputs to assess the usage of some records during the training phase by the shadow models (where the ground truth is known since the allacker also designed the training set of those models). The classifier, trained on shadow models' outputs, is then used against the original model with the objective of identifying data that are part of its training set.
MIA as Classification of Behavioural Changes
MTA can be framed as a classification problem:
the attacker determines whether a certain example has been used during the training phase of the model or not. Attacks are based on detecting changes in the target model when it is exposed to data points that are in the training set versus when is exposed to examples that are not part of it.
Stemming from the general setting described in Shokri et al. 2017 MTAs have been successfully tailored for Language Models to extract sensitive information from them (Song & Shmatikov 2019 Hisamoto et al. 2020). Carlini et al. (0022) observe that MIAs could be better framed as the procedure of comparing models that are trained on a given example and models that are not trained on it, and to find a difference in their behaviour, for example, having a likelihood of the example much higher in the first case rather than in the other. They train numerous shadow GPT-2 models to measure the probability of observing certain likelihood of an example in models trained and not trained on it.
However, training shadow models may be too expensive in the context of LLMss: the original formulation of MIA can be used as long as it is feasible for the allacker to construct shadow models that are similar to the target model. For this reason, in the context of LLM the formulation for MIA has been adapted, generally climinating shadow model developments.
3.2.1 MIA with Thresholds
Since the construction of shadow models can be too expensive, MLAs have also been formulated without relying on them. In fact, some MIAs against LLMs have been designed to exploit some measure of overfit over the training data in order to understand whether a given example has already been seen by a model: overfitting the training data is a sufficient condition to detect memorized examples (Yeom et al. 2018). Shi et al. (2024) observe that the probability distribution is more spread over low probability words for an unseen example: they design attacks that threshold the average log-likelihood of the top-k rarest words to ascertain whether not the example is part of the training data. Language models trained clinical
18

---

Published in Transactions on Machine Learning Research (01/2025)
data have been shown to be sensitive to MIAs. Mireshghallah et al. (2022) demonstrate the effectiveness of MTAs designed against Language Models trained with Masked Language Modeling objectives, obtaining a substantial gain both in AUC and in recall of the attack over the baseline. Their method is based on thresholding the ratio between the likelihood of a sentence on the target model and the likelihood of the same sentence on a model trained on a general population. While their experiments are conducted on mranformer-based networks trained on Masked Language Modcling objective, the method proposod can be, at least in principle, generalized also the generative context. Mattern et al. (2023) argue that the limited availability of training data for the reference model poses a strong restriction the applicability of previous work. They propose to generate perturbations of a sentence and to measure the average loss of target model on the original sentence and on the perturbations: they hypothesize that if the sentence is not in the training set, the model should behave similarly also on the perturbed examples, while it should have a lower loss conditioned on the original sentence it is part of the training set. These kinds of attacks are also useful to understand, given two snapshots of a model - for example, a pro-trained model and a. finc-tuned version of it on which data the fine-tuning phase was performed. In particular, the difference ween the probability assigned from different snapshots to each token in sentence can help identify whether an example was used during later training phases Maanlla-Béguelinn et al.) 2020). Duan et al. (0223) show that LLMs, while being extremely versatile in a number of tasks thanks to their in-context capabilities, can be subject to MTAs that target datasets used during prompt training. They demonstrate that an LLM trained to answer prompts exposes more information than the corresponding fine-tuned model.
However, a recent body of research poses some critical challenges to the use of MIAs to assess their reliability in demonstrating the inclusion of a sample in LLM training data. Tn some cases, the theoretical foundation of MIA is discussed. Zhang al. (024c) argue that, since it is no longer possible to use shadow models that do not include the data point in their training set, the statistical framework behind MIA is no longer applicable since other approaches are statistically unsound. Duan et al. (024b) demonstrate empirically that range of MIA is not reliable against LLMs. According to their findings, training for a single epoch- as is becoming standard practice for LLMs- and increasing amounts of training data may negatively affect MIA accuracy. They also notice that MTA become more difficult as the difference between data included in the training set and data not included becomes more subtle. However, as training corpora become larger their statistics are based on the Pile (Gao et al., 2020)- this similarity increases: for example, non-members Github samples share on average 76.9 % of their 7-grams with the Pile. Hence, the combined effect of training techniques that diverge from the one defined for other models and the huge training sets may challenge MIA's 's effectiveness against LLMs.
3.3 Model Inversion and Stealing
Model inversion attacks generally exploit the output of the model and use this information to reconstruct the input data that generated the observed outputs. In the initial formulation (Fredrikson et al. 2014 2015 adversary is given access to the trained model and to a partially masked input, for which only non-sensitive features are disclosed. The attacker aims to infer the value of the sensitive attributes that serve as input to the model: since deep learning models create dense, vectorial representations for textual data, model inversion attacks against, deep learning architectures aim to decode human-readable features from them. While the initial formulation is implemented for lincar regression models (Fredrikson et al., 2014) and a neural network trained for image processing (Fredrikson et al. 2015), this type of atack has been successfully used to extract sensitive input information from different models, including LSTMs (Welleck et al. 2018). Those kinds of attacks are challenging against LMs, especially since, in principle, the optimization process to reconstruct the initial input of a model has to deal with the discrete nature of text (Parikh et al. 2022).
Model inversion attacks are often performed against distributed models; in fact, some protocols like federated learning ensure that private data are not shared across multiple servers (sce $2.3 for further details). However, with some access to model outputs, it is possible for an attacker to reconstruct the input data that causes the observed output.
19

---

Published in Transactions on Machine Learning Research (01/2025)
Model Inversion Attacks and Model Stealing
This family of attacks threatens models, especially when they are distributed across multiple servers: in fact, while data holders do not directly share private data which are thus maintained locally- they may be willing to share seemingly opaque information related to a machine learning model, such model outputs, embeddings gradients. However, Model Inversion attacks are designed to reconstruct private data from these apparently opaque sources. In some cases, also centralized private models parameters can be directly reconstructed: this is the objective of Model Stealing attacks.
In some cases, from those opaque sources, it's also possible to reconstruct a model's parameters from its output: in this case, the attack technique often defined as Model Stealing ($3.3.).. These attacks aim to recover model weights from an APl-gated model, making it less opaque and, hence, less robust against other types of privacy attacks. While the stealing of entire LLMs has not yet been described, some works efficiently reconstruct a subset of the model parameters in black-box attacks.
3.3.1 Model Output Inversion and Model Stealing
The outputs of a model, like the representation of textual data that a Transformer-based LM produces in the last layer, can be totally partially inverted to reconstruct the text that is originally the input of the model. In particular, these representations often called embeddings - are only apparently opaque and can be inverted to reconstruct the entire sentence or the words that it contains. |an el al. (2020) design the first model inversion attack against Transformer-basee Language Models. They observe that directly framing the inversion problem as an optimization problem is challenging due to the discrete nature of sentences. They train an attack classifier that is designed to extract fixed patterns and keywords- like, for example, location medical information from sentence representations of models. Similarly, [Song & Raghunathan (2020) predict unordered sets of words from sentence embeddings: their attack model is neural network trained with the multiset prediction loss. While the experiments in Pan et al. (2020) and [Song & Raghunathan (2020) are performed on relatively smaller Transformer-basee network compared to more recent models, their methods can generalize to larger networks with some additional difficulty due to the increasingly larger embedding dimension in LLMs.
In some cases, the entire input text is also reconstructed. Li et al. (023d) implement the inversion as a generative language modelling task with decoder-only model, conditioned on the embedding of the sentence to invert as the first token representation. They show that the model successfully inverts meaningful tokens containing a large fraction of the named entity. The threatened model in their experiment is a BERT model, but the general framework can be extended given model representations for sentences. Morris et al. (2033) propose an iterative method to reconstruct the input text of Transformer- embeddings for documents: their allack is based on the idea of iteratively generating different hypotheses that may justify the observed embeddings. They learn the distribution of texts in a dataset given their embeddings: their approach starts by assuming adjusting the hypothesis to progressively align its embedding to the target one. This approach has also been extended to a multilingual setting, adding translation step before iterating again over the generated hypothesis Chen et al.] (2024b). Only observing the logits of LLMs can also be sufficient to reconstruct the original input: Morris et al. (2024) train an inversion model on a collection of instruction datasets to reconstruct the prompt of an LLM in their experiments the seven billion version of Llama-2 and T.ama-2 Touvron et al. (2023) given the probability distribution of the token following the prompt. In a completely black box scenario in which only the model's output. is observed an allacker may be able to reconstruct the original prompt training an inverter another LM- on previously observed outputs on known input prompts Zhang et al. 2024a
models that produce
initial hypothesis and then iteratively improves it by
This kind of attack has also been applied to Language Models fine-tuned on classification tasks to reconstruct private training data. In this case, the output of the model is a probability distribution over each of the possible classes. This information can be effectively exploited: greater confidence that a model has in classifying sentences that are already part of the training set. [Elmaddy et al. 20022 inject canaries in the training data (sequences that are intended to be later extracted and out of
particular, an attacker may exploit the
20

---

Published in Transactions on Machine Learning Research (01/2025)
distribution with respect to the other data) and then reconstruct a partially masked sentence to search for tokens that maximize the probability of the target label. They perform the reconstruction by enumerating tokens in the vocabulary, weighted by the number of occurrences that they have in the training data. ZZhag ct al. (022c) propose instead to build a dataset that resembles the unknown training data and initially train a lext generator on these collected data, similarly to what done with shadow model MIA ($3.2.. Then, they further train the text generator via word embedding perturbation with the objective of minimizing the classification loss on the target model. The idea is that the target model's prediction on training sentences will exhibit lower loss, and this information can be used to perturb the text generator model and obtain data closer to the private one. Although the use of generative models as classifiers is not widespread at the time of writing, the results of this works could be generalized to the setting of generative LLMs with some additional difficulties in defining the larget class of fine-tuned generalive models.
3.3.2 Gradient Inversion
In Federated Learning [Ray et al. 2021,, multiple nodes train a model cooperatively without directly sharing data samples: the system is designed to make training feasible while exchanging model updates rather than data across nodes (see $2.3 for further details). Gradient Inversion attacks are directed against federated learning environments that perform training by sharing gradients. Those attacks can be used to reconstruct training data from seemingly opaque sources.
Zhu et al. (019) demonstrated that if the gradients are openly accessible, it is possible to reconstruct the training data. The attacker aims to reconstruct the training data by minimizing the distance between some generated gradients and the ground truth onc. In particular, the attacker initially generates gradients from random inputs and labels. Then, having access to the ground truth gradients solves the optimization problem of minimizing the L2 distance between the generated gradients and the ground truth one, parameterized by real inputs and labels. In their experiments, ZZhu et al.] (2119) applied this attack against a BERT model. This idea was further explored by inverting gradients for image classification tasks (Zhu & Blaschko 2021
Geiping et al. 2020 Yin et al. )
In recoustructing textual data, Deng et al. 2021 further attempted to perform a gradient atlack against Transformer- based language models, introducing also L1 term in the reconstruction loss. Experiments on a Transformer model, BERT model and BERT demonstrate the effectiveness of this gradient attack, recovering up to 50% of the original tokens on average. In LAMP (Balunovic et al. 2022), more effective results are achieved by simultaneously training the attack model to minimize the difference between the reconstruction gradients and choosing at each iteration only sequences that have low perplexity according to external language model (like GPT-2). Reconstructing the correct word order is, in fact, one of the main challenges: Gupla et al. (2022) proposes to recover from the gradients a bag of words for the sentence to extract and then perform beam search to effectively reconstruct the sentence.
Should be noted, however, that the moment of writing the application of those attacks to larger modcls is mostly limited by the increasing usage of centralized pretrained models.
3.3.3 Model Stealing
As exposing only black-box access to a LLM is standard practice for popular commercial models, an adversary might want to reconstruct the gated model with the aim of imitating its behaviour. A family of attacks called model stealing attacks have been designed with this goal. The adversary, in this case, can have no information regarding the original training data. The attack usually requires observing the model outputs on inputs that the attacker fed to the gated model reconstruct its internal parameters. Tramer et al. (016)) introduce this family of attacks and demonstrate that attack could lead to an accurate reproduction of logistic regression, neural networks, and decision trees. They also successfully demonstrate on those systems that model inversion as defined by [redrikson et al. 2015) could be applied to the stolen model to reconstruct. private input, with direct privacy implications.
Krishna et al. (2020) demonstrate the fcasibility of those attacks against Transformer- models. They use random words as input for gated fine-tuned BERT models and use their outputs for fine-tuning an equivalent model. They also demonstrate that the quality of the stolen model increases if the attacker has
21

---

Published in Transactions on Machine Learning Research (01/2025)
some background knowledge regarding the domain of the original fine-tuning dataset. Moreover, Zanella- Beguelin et al. (2021) notice that an attack that exploits the characteristics of the Transformer's last layer can effectively reconstruct a fine-tuned BERT model with frozen pre-trained weights and learnable embedding projection matrix. In particular, they reconstruct the embedding projection matrix by feeding the gated model random inputs and collecting the corresponding output. They reconstruct the embedding projection matrix optimizing its weight on the observed embedding and outputs. Their technique, in combination with other techniques like the one proposed by Krishna et al. (2020), can lead to a refined model that resembles the target one.
Recently, similar idea has been applied to thrcat LLMs: Carlini et al. (2024) proposed an attack to recover the embedding projection matrix at the last layer of a transformer language model. They attack a model in a black-box fashion, collecting the model's output logits on random prompts. They notice that since the embedding projection matrix is low rank since it projects the hidden dimension to a logit vector that has higher dimensionality- it's possible to apply the SVD algorithm to the collected logits to obtain a linear transformation of the original embedding projection matrix. Moreover, they demonstrate that the logits over the entire vocabulary are not necessary, and that the probabilities over predetermined number of words are sufficient for successful attack. A similar approach has also been proposed by Finlayson ct al. (2024) and they describe the utility of the attack in discovering whether a gated model has undergone some update.
3.4 Privacy Threats at Inference Time.
In the LLM context, the leakage of the original training data is not the only possible threat. The inclusion of additional, potentially private information at inference time presents a number of challenges. Here, we summarize three of the major challenges derived from such systems. These threats have pushed great deal of research to focus on how to perform calculations securely on sensitive user input, either by obscuring them through differential privacy (5.2) by keeping them encrypted during the entire calculation process 5.5
Computation over Private Input. Given the challenges of training such models from scratch, the option of using LLMs in Machine Learning Service modality is becoming increasingly popular. In this setup, an organization hosts the model and users simply submit their data for processing. This approach raises significant privacy concerns for user data: vulnerabilities in the systems storing this data could lead to breaches (OpenAl 2023), and model providers must offer strong assurances to comply with privacy regulations, such as GDPR. Similar concerns even led to a temporary ban on ChatGPT in Italy (Lomas) 2023).
This issue is particularly concerning for businesses dealing with proprietary data or commercially valuable information (Mitchell 2023 Coles 2023 and companies have decided, in some cases, to warn their employees
about the responsible use of LLMs Kim 2023)
Prompting Private Data. At inference time, the versatility of LLM is enhanced by the use of additional data, which are often provided in the form of demonstrations in the context of the task the user wants to perform (Brown el al. 2020). While being useful technique, some sensitive information could be included in these examples in context as well, and an increasing body of research suggests that some risks can be identified during inference. In particular, since in-context learning is more cost-effective than fine-tuning, some systems may deploy LLMs that handle user requests by utilizing pre-loaded in-context demonstrations, which may include private or sensitive information. This efficacy, however, comes with an increased risk terms of privacy leaks, which can also outweigh the risks against fine-tuned models (Duan al., 2023). Wu et al. (024b) propose threat model in which a user of a system submits some data containing private information, such medical records, and another user of the same system could potentially access that sensitive data. They focus on addressing this threat through the application of Differential Privacy during inference (see $5.2 for further details). Duan el al.] (2024a) demonstrate the elficacy in a membership inference attack (MIA) against prompts containing sensitive, in-context learning demonstrations: they suppose that a model is prompted for a classification task and that the adversary may want to understand whether a certain example has been used or not as in context demonstration. They argue that an adversary can test whether an example has been used in context since the probability for the corresponding class will be much higher for
22

---

Published in Transactions on Machine Learning Research (01/2025)
members (in context examples) than for non-members. Similarly, in this scenario, malicious user -instead of performing the intended task on the system- can potentially extract the in-context demonstrations via jailbreaking attacks (Tang et al.) (024). Chu et al. (2024) demonstrate that entire conversations can be reconstructed via jailbreaking when they are used as contextual information for prompting an LLM.
Contextual Privacy. As LMs' capabilities increase and their role as assistants becomes more prominent, it is important to understand whether they are able distinguish if piece of certain sensitive information is safe to be shared in a given context. For example, there is no privacy leakage when doctor shares with his patients information about their own health. However, the disclosure of such information to an unauthorized insurance company violates patients' privacy. Those considerations are recently emerging as language models can be reliably used as assistants by groups of users that want to share some information while preserving others. Brown et al. (2022b) discuss the discrepancy between data sanitization and differential privacy that assumes a structure private data and cannot solve the necessity to share private information only with the correct group of people. They also argue that some data protection techniques like data sanitization and differential privacy make assumptions on the nature of the data that should be protected: implicitly or explicitly, they state that private information is structured information that should not be shared. However, every bit of language can potentially contain secrets that are not meant to be shared indiscriminalely.
Mireshghallah et al. (2024) formalize the necessity to share secrets in the appropriate context stemming from the contextual integrity theory [Nissenbaum] 2004). In particular, the information to be protected is not only in the training data but also during inference. They introduce a new benchmark to assess the ability of such models to disclose private information only to users who should be allowed to access them. They measure that multiple LLMs include information that should not be disclosed when asked to generate summaries of meeting conversations. Priyanshu et al. (2023) similarly cxplore the tendency of ChatGPT to disclose personally identifiable information when asked to produce summaries of hiring decisions and medical notes, a behaviour that they notice can be mitigated by explicitly asking the model to comply with privacy regulations and feeding step-by-step prompts that suggest which information should be deleted. [aang et al. (0244) also discusses the necessity of measuring the privacy leakages during inference. They focus on PII like emails and phone numbers: they propose to initially feed the model with prompts that contain PIT with the indication that those should not be shared; then, they prompt the model asking for the same information. They notice that GPT-3.5 is prone to violating privacy restrictions, while GPT-4 is more robust. Both models, however, tend to reveal private information when prompted with some demonstrations containing leakages in the shot scenario.
23

---

Published in Transactions on Machine Learning Research (01/2025)
4 Solutions for Preserving Privacy in LLMs: Data
After identifying the key vulnerabilities and potential attacks on LLMs as outlined in f3 in this section, we introduce the reader to the current possible solutions to the problem of preserving privacy in the context of LLMs and text generative models. We have to say that the field is fragmented as this is recent concern, and the field is evolving fast. At the moment of writing this survey, there is no single gold standard solution to the problem of preserving privacy in LLMs and text generative models. The main solutions available can be broadly categorized into two categories: i) Preserving Privacy in Data $ and ii) Preserving Privacy in Model $ In the former section, we collected all the methods that try to solve the problem at its source, i.e., they act directly on the privacy-sensitive information in the dataset before the training happens, while in the latter, we selected all the methods that address the problem from the model perspective, ie., try to remove from the model problematic information after the training. The two approaches fit different use cases, so it is important to be aware of both.
In this section, we propose several solutions that allow for preserving privacy from the data perspective, i.e., we preserve privacy directly in the training data used in machine learning models. This includes any approach that involves manipulating, carefully choosing, or generating training data in order to avoid the memorization and subsequent leakage of private information inside large language model any model that learns from data. This basically puts the focus preserving privacy in the data used to train the model. Given that a machine learning model is moulded by ils training dataset, training it on privacy-preserving dataset ensures that these methods offer equivalent privacy assurances to the dataset itself. The methods presented here are thus not limited to LLMs as they can be used to anonymize textual data more generally. That is, it is possible to use them to strip datasets of sensitive information before using these datasets to train LLMs Making the training data safe means making the trained model safe.
In the context of LLMs, there are very few technical reports that thoroughly describe the precautions necessary to preserve privacy via data anonymization, and even fewer that release the preprocessing software used for this anonymization. Among the notable examples, the LLaMA3 technical report (Dubey et al. 2024) mentions 'filters designed to remove data from websites [to eliminate] unsafe content or high volumes of PIT' but does not detail the exact procedure while noting the use of "Microsoft Presidio Analyzer' in the Similarly, (oldaini et al. (2024), in the context of training OLMo (Grocneveld et al., 2024), also considered using Microsoft Presidio but ultimately opted for custom regular expressions, given the scale of the pre-training dataset.
preprocessing of speech data (See our overview of the tool in $6
Preserving Privacy in Data
Key Idea: Preserve privacy directly on the dataset used to train the model.
Overall, the available open reports suggest. that for removing PTT from the pre-training datasel, the preferred solution at scale tends to involve custom regular expressions (Penedo et al., 2024). The broader subject of preserving privacy in pre-training data, however, remains largely unexplored beyond these rudimentary techniques (Subramani et al. 2023). The scope of this section is thus two-fold: i) propose to the reader possible new avenues to preserve privacy in the pretruining dataset by surveying existing approaches proposed in the literature that have not been applied yet in the context of the pretraining phase of LLMs; ii) propose to the reader anonymization techniques that can be used to anonymize data for the instruction-tuniin / fine-tuning phase where the L.Mss are possibly fine-tuned on sensitive in-domain data for the downstream task (c.g., healthearc). Since most of these techniques were not tested with training datascts for LLMs, it is difficuul to discuss about the scaling of these methods for big datasets, but seeing examples like the previously cited OLMo (Groeneveld et al.) )024), it is quite clear that complex methods at big scale tend to not be chosen. However these techniques can be applied to data regardless of the subsequent usage, which means that they could (and will) be used to clean data for inference, before feeding it to a model.
The section is structured in the following way: we first show classical anonymization techniques $1.1 then we focus on anonymization techniques built with Differential Privacy 11.1.2 Putting this last scction in relation with Figure the point of application of DP would be (1), the input, because all these techniques aim apply Differential Privacy on the data before feeding it to the LLM (for either training or inference).
24

---

Published in Transactions on Machine Learning Research (01/2025)
Anonymized Data
Data
Preprocessing
Identification
Anonymization
Figure 4: Classical Anonymization Pipeline. The figure illustrates the standard process for anonymizing data in natural language processing tasks. The pipeline consists of three sequential blocks: Preprocossing, Identification, and Anonymization. The Preprocessing block prepares the raw data for further analysis. The Identification block involves the use of a procedure to detect potential Personally Tdentifiable Information (PII) within the data (e.g., Named Entity Recognition). Finally, the anonymization block replaces identified PII with anonymous substitutes, ensuring data privacy preservation. Arrows indicate the flow of data. through each stage of the process.
4.1 Anonymization
Data anonymization is not a new concept, with seminal work dating back to the 1990s: a pivotal develop- ment was the introduction of k-anonymity in 1997 (Sweeney 1997), which means ensuring that data cannot be re-identified to fewer than individuals. This foundational concept has spurred extensive research, lead- ing to plcthora of anonymization tochniques. Given the breadth and depth of the field, this section is organized into two main parts: the first, titled Classical Anonymization 4.1.1 explores traditional hot such as rule-based approaches and includes discussion of how recent advancements in language models have been leveraged for anonymization purposes. These traditional methods have been the cornerstone of anonymization for decades. In more recent developments, the rapid adoption of Differential Privacy has marked a significant evolution in the field, offering robust privacy guarantees that, depending on the appli- cation context, can surpass those classical methods. The second part of this section, Anonymization with Differential Privacy 122., discusses these techniques. It also introduces the concept of Metric Differential Privacy, highlighting its unique applications and benefits within the field of LLMs and beyond.
4.1.1 Classical Anonymization
As already mentioned, anonymity definitions can be traced back 1997 with k-anonymity. Anonymiza- tion aims to make the ro-identification of individual impossible, that is, the process of reconstructing the identity of an individual given a record that contains their information. Different attributes could be anonymized. While effectively masking PII - like names, emails, and social security mumbers - is necessary, some features that are not exactly identifiers can be used in combination to identify an individual: those features are called quasi-identiffers. Quasi-identiffers are apparently non-personal attributes - like gender, date of birth, and zip codes that can be used in combination with external resources to identify a person from its anonymized records. The definition of k-anonymity (Sweeney) 1997) ensures that a quasi-identiffer cannot be used for ro-identification. Essentially, datasct is said to have k-anonymity if any given record in the dataset is indistinguishable from at least k- 1 other records with respect to certain identifying attributes called quasi-identifiers. Upon this concept, several implementation techniques were proposed by Samarati & Sweeney [1998), like generalization (modifying attribute values to less specific but accurate representations), suppression (i.e., entirely removing data that are outliers or too identifiable), minimal Generalization (en- suring data are not generalized more than necessary to meet k-anonymity). Of course, this method is highly susceptible to the choice of k (even inapplicable sometimes) and gives coarse data, making it impossible carry out fine-grained analysis. It is also important to mention that, due to the reliance on quasi-identiffers, this method is much less effective with unstructured text. Since anonymization has such long history with the first paper theorizing k-anonymity dating back to 1997 (Sweency 1997) much rescarch was devoted to this field, resulting in the development of several techniques. Larbi et al. (022)) give an overview of many anonymization techniques, with their different pros and cons based on the use case. In this section, we will show the most relevant ones that are also available in a synthetic tabular format in Table 2
The standard process is outlined in Figure 4 After any necessary preprocessing phase, the first challenge solve is to identify the entities that may cause violation of privacy. This be done via Named Entity
25

---

Published in Transactions on Machine Learning Research (01/2025)
Recognition (NER) tool. To maintain the coherence of the text, Co-reference Resolution (CRR) system can be used to resolve references between entities should be identified if the anonymization framework aims to ensure consistent anonymization across the document. Finally, private information should be replaced with anonymous information. The anonymization methods can vary. Suppression is used to completely remove the information. Tagging allows the replacement of sensitive information with artificial labels like the one obtained by the NER system- that still retain information about the class of the data and an identifier. The sensitive information can also be replaced with other textual information: random substitution and generalization can both be used. Random substitution uses another random entity of the same class, while generalization replaces an entity with a more general term. It should be noted that, while this pipeline is intended for general data anonymization, regardless of which model is going to be trained on it (be that an LLM or not); it is specific to textual data, as it relies on some specific features of text.
Anonymization before DNNs Mamede ct al. (0166) present modular anonymization framework for text documents in Portuguese that closely resembles the general workflow for this task. The system consists of four modules: pre pre-processing, Named Entity Recognition (NER), Co-reference Resolution (CRR), and Anonymization. They notice that the effectiveness of automatic text anonymization methods relies heavily on the performance of Named Entity Recognition (NER) and Co-reference Resolution (CRR) modules and while CRR shows good performance, the NER can be very lacking depending on the dataset. They also implement and test a number of different anonymization strategies: all the anonymization methods presented can be effective in some cases, but they all exhibit some drawbacks. Suppression, while dependent on NER accuracy, often compromises text comprehension due to loss of information, challenging readers to grasp entity references. Tagging helps maintain references between anonymized entities but detracts from natural text flow, necessitating improved NER and CRR accuracy. Random substitution maintains natural language output but can lead to semantic drifts due to arbitrary replacements, suggesting a need for a curated list of vague yet contextually appropriate entities. Generalization offers balanced approach by replacing specific entities with broader categories and improving readability and relevance, especially for location entities, though it struggles with accurately linking entities with multiple knowledge base entries.
Another prominent anonymization technique is perturbation, as proposed by Zuo et al. (2021) It involves subtly altering the original data to preserve statistical significance, aimed at reducing the vulnerability of the dataset to linkage and enhancing privacy protection. Techniques like micronggregation and data swapping are commonly used for the implementation of perturbation. Microaggregation involves grouping data and using representative values, while data swapping involves interchanging values between records to maintain the overall statistical properties of the dalaset. While perturbation effectively anonymizes raw data and retains its statistical utility, it may compromise the accuracy of anonymized data.
While there is no doubt that these methods could be used to clean datasct for LLM training or inference, it is not very clear how they would scale for an LLM-sized training dataset, as they are developed for much smaller datasets.
Anonymization with DNNs More recent anouymization methods also leverage Deep Neural Networks. Zhou et al. (2022) use BERT to impute previously masked sensitive information. This framework is called DataSifter'ees Text, and it works with a whitelist that holds sensitive terms that need to be masked and a blacklist for meaningful terms that should be retained. After a proportion parameter is chosen, sensitive tokens are masked according to that proportion, and then BERT used to replace the masked tokens. These two steps in the procedure respectively cover the identification and anonymization step shown in 4 Results show that even when the obfuscation level is high, the framework retains significant data utility, demonstrating the method's effectiveness in maintaining informative content.
In some cases, the author of a text is meant to be identified: re-identification of an author can be done by spotting some fral that characterize an author's writing. Romanov el al. (2019) analyze this scenario for the Russian language. They extract features like lemmas, parts of speech and morphological features. Then, a fast correlation filter multivariate feature selection method where the class relevance and the dependency between each feature pair are taken into account is used to select the most informative features. Those informative features are the more likely to be identifying features. To preserve the privacy of the author, those
26

---

Published in Transactions on Machine Learning Research (01/2025)
features are smoothed by making targeted adjustments. For example, words punctuation marks that are characteristic of the author's style are replaced or altered based on average statistics derived from broader corpus. The final step is to further process the smoothed text with a deep learning model: in this work, a Transformer-basee model is chosen. This approach reduces the probability of authorship identification to random guessing.
Mosallanezhad al. (011 use an attention- based task-aware text representation learner for extracting latent embedding representations with reinforcement-lemmine learning-based based privacy and utility preserver for manipulating these embeddings. While it effectively balances privacy and data utility and is adaptable and efficient, it has notable drawbacks, including complexity, potential data-specific limitations, and dependency on procise parameter tuning. Complexity in particular is relevant to this context, as it is likely to make this method less suitable for large-scale datasets.
More recently, some works proposed the use of LLMs for anonymization. Vats et al. (2024) propose to hide sensitive tokens via a masking process and impule them by using a Large Language model (LLM). In relation to Figure the masking acts the identification part, as it finds and masks sensitive information, while the imputation represents the proper anonymization, where non-sensitive terms replace the (previously masked) sensitive ones. To carry out the masking process, several techniques are proposed: handcrafted list of safe words (the ones that do not carry sensitive information) called allowlist created by linguistic experts, this is similar to the blacklist of Zhou et al. (2022) and identifies the terms that do not need to be masked; another method to identify safe words vocabThres, where the N more frequent words in a vocabulary are considered safe; the last masking criteria is to find the unsafe words and involves using a NER module that identify potentially private tokens to mask them (similar to Mamede et al. (2016)). After choosing masking method, the chosen tokens are masked and then imputed by LLM. Also for the imputation, three separate methods are proposed: Top-1, which uses the one token with the highest probability from the LLM's output; Top-K, which randomly selects one of the first K tokens with the highest output probabilities; Fine- Tine-TTming, which selects the records that do not contain sensitive information and uses them to fine-tune the LLM, then just takes the highest, probability output (so after fine- Lune-tuning it circles back to the Top-1 approach). They compare models trained on the anonymized datasets against models trained on the masked dataset, showing significant improvement in both tasks. Intuitively, this method is quite sophisticated and complex, which means that for LLM-sized training dataset it is going to make up for a quite heavy computation, but it could be suitable for smaller dataset that might be used for fine-tuning or inference.
As the usage of closed models increases, privacy concerns also emerge when sending prompts containing private information. Chen et al. (2023b) focus on sanitizing prompts before sending them out to some unsafe third party like a black-box model like ChatGPT, thus enforcing prompt privacy. Moreover, they argue that standard anonymization techniques like masking may be useful to ensure the privacy of prompts, but the model output could not be effectively de-anonymized to improve its utility. They propose new framework called HaS (Hide and Seek) with two local small LMs, one trained for anonymization and one trained for deanonymization. The first one preprocesses the prompt before sending it to the black-box and the sccond one postpssesssss the model output. This method is specifically designed to use LLMs in inference without worrying about personal data in the prompts. The hide model is trained with original sentences and extracted privacy entities and is prompted to substitute them randomly, while the seek model is trained with the same data and the addition of the remote LLM answer. The framcwork is evaluated using both adversarial models (black box and white box) to measure privacy protection effectiveness and usability experiments to assess the impact on LLM task performance, including translation and classification. Adversarial models are unable to recover original information, indicating stronger privacy protection. Usability is evaluated by metrics like BLEU for translation and precision, recall, and F1 scores for classification to see how well the framework maintains the utility of LLM outputs post-anonymization. The results demonstrate that HaS effectively balances privacy and utility, with label-based anonymization showing better privacy protection and generative methods offering slightly better utility retention.
4.1.2 Anonymization with Differential Privacy
In this section, we propose works that use Differential Privacy for data anonymization, i.c., before actually using the data in training. These methods are intended to anonymize datasets, regardless of the intended
27

---

Published in Transactions on Machine Learning Research (01/2025)
X1
X2
X3
X5
Unembeddiin Matrix
Neural Model
1 Input
Embedding Matrix
4 X2
4 X3
X1
Xs
Figure 5: Anonymization with Differential Privacy. This figure illustrates the procedure to anonymize data via differential privacy. A neural model is used embed data, data embeddings are then perturbed with noise using DP. The anonymized output is then returned by decoding the perturbed embeddings.
use of the datasets. This means that the anonymized datasets might be used to train LLMs, but also for other models and tasks. We refer the reader to $2.2 for introduction to Differential Privacy, and we will propose later in $5 methods that use Differential Privacy directly the model. The methods proposed in this section apply Differential Privacy to perform text rewriting, where the original text is modified by perturbing its representations to ensure privacy. The key advantage of this approach that it adheres to the formal guarantees of Differential Privacy. All the works in this section are also available in tabular format in Table 3
identify the works in this section examples of Differential Privacy applied to the imput (1) of neural model. Differential Privacy also be applied in separate, typically smaller, auxiliary preprocessing model to anonymize data before it is fed into an LLM.
Referring to Figure
Metric Differential Privacy Feyisetan et al. (019b)) propose DP mechanism to transform text data that constructs a hierarchical representation to identify private phrases in the input sentence and then randomly replace them with neighbouring words in a word embedding space. They build upon Metric Differential Privacy that, as explained by Chatzikokolakis et al. (2013), extend Differential Privacy principles to location-based privacy. This approach, originally developed for spatial data, is adapted for textual data by utilizing word embeddings as coordinates, transitioning its use from spatial to textual domains. For obfuscation, target words are selected by favoring locations near the original one with a higher probability over distant ones. For a graphic representation of this process, refer to 5) Building further on the theme of differential privacy in metric spaces, Feyisetan et al. (2019a) intuce a noise distribution tailored to the dx-privacy criterion, which calibrates values based on the geometric properties of the word embedding space to maintain plausible deniability of original words. This methodology was tosted across various tasks including binary sentiment classification the perturbed IMDB dataset, multi-class classification on Enron, and question answering on InsuranceQA, revealing less than 2% utility loss in binary classification tasks, even at low values, indicating a favourable balance of privacy and utility compared to other methods like Versatile (Arampatzis et al. 2015) and Incognito (Masood et al.) 2018). While the tests for this method do not include L.MMs and text generation in general, nothing would prevent traïning or fine-tuning an L.M with a dataset cleaned with it. This would definitely cause preprocessing overhead, but could be balanced by the missing training overhead from removing DP in the training phase, so is not clear whether this would be a favorable trade-off and it would probably depend on both dataset size and model size.
The Differential Privacy mechanism, in textual examples, should not disrupt the syntactic structure of a sentence. For this reason, Krishna et al. (2021) developed ADePT, an autoencoder-bassed differential privacy algorithm aimed at anonymizing text while preserving its utility. This method involves clipping and noise introduction-iitieee Laplacian or Gaussian-betweon encoding and decoding to form a differentially private latent representation. They hypothesize that data regeneration via an auto-regressive decoder maintains the syntactic structure of the sentence. Since they focus on the anonymization for Intention Classification (IC),
28

---

Published in Transactions on Machine Learning Research (01/2025)
an intention label is included before encoding to facilitate training annotation-awart autoencoder. Despite ADePT's design to enhance privacy against Membership Inference Attacks and its superiority over traditional privacy-sensitvve word replacement methods like |eyisetan et al. (2019b), Habernal (2021) point out that ADePT actually fails to achieve true diffcrential privacy and cxhibits higher sensitivity than reported, casting doubt on its effectiveness. In a similar lashion, Weggenmann et al. (0222) introduce a VA-VAE (Differentially Private Variational AutoEncoder) that also perturbs the latent space to secure privacy, following the model depicted in 5 This model contracts the mean vector of each latent representation towards the latent space's 's origin before adding isotropic Gaussian noise. Applied to anonymize IMDb and Yelp reviews, this technique successfully reduces authorship attribution accuracy significantly while only slightly impacting sentiment classification accuracy. Further testing with a GPT2-based model confirms that the anonymized texts remain coherent and understandable despite a slight increase in perplexily.
Also transformer- models can be used in this setting. Extending the research on Differential Privacy for text privatization in this direction, [gamberdiev & Habernal (2023) introduced DP-BART, a privatization model using the BART architecture instead of an LSTM, employing a similar strategy of clipping and noise addition in the latent encoder output, seen in 5 DP-BART tackles the challenges of high sensitivity and computational demands associated with transformers through a novel clipping technique (by value) and an iterative pruming and training method that simplifies the encoder's output vectors. Tested against ADePT on datasets like ATIS, Snips, IMDb, Drugs.com, and Amazon, DP-BART demonstrated superior performance in both privacy and utility. Another transformer- based approach is proposed by Du et al. (2023a), where they use BERT and propose the Purkayastha Mechanism to take advantage of the angular distance as a metric. They also propose to sanitize labels when too sensitive, using Randomized Response if the label space is small enough. Otherwisc, they prune the label space, leveraging prior knowledge and then use RR. This approach is tested on SST-2, IMDb, and QNLI datasets, representing tasks for sentiment analysis and -anteerngn based on movie reviews and Wikipedia content, respectively. Performances are close to the non-private version with an increased robustness to MIAs.
Levaraging the Exponential Mechanism Yue et al. 2021) argue that - due to the curse of dimensional- ity by-ding directtyy injecting noise in large high-dimensional word embedding may fail to strike a nice privacy-utility balance since it becomes exponentially less likely to find a noisy embedding close to a real one on every di- mension. Yue al. (2021) developed two strategies, SANTEXT and SANTEXT+, to replace original text tokens with 'sanitized" tokens via the exponential mechanism. Both methods utilize embeddings to map words to compute their semantic distance. Then, once word is identified to be replaced, the target word is selected with a probability that is inversely proportional to the distance with respect to the original one, SANTEXT indiscriminately replaces any word, often resulting in excessive utility loss and protection, while SANTEXT+ specifically targets and replaces the top w low-frequency tokens, identified as most likely to be private. These approaches enforce Metric Local Differential Privacy (MLDP), maintaining utility by lavouring tokens closer to the original, though requiring a high epsilon value, like 16, to achieve notable improvements in defence rate. Similarly, Chen et al.] (2023a) introduced CUSTEXT, another MLDP ap- proach where every token is replaced using a mapping function that determines the output set for an input token and a sampling function based on the exponential mechanism for selecting replacements. The main difference here is that every input embedding has a custom set of output embeddings, which is subset of the full output space used in SANTEXT, and thus, it reduces the chances that the output is semantically irrelevant to the input. They claim the best privacy/ baaecy/cccurar /accuracy tradcoff for these systems so far, achieving good results with relatively low epsilon values (up to 3), although there still a notable performance drop compared to using plain text.
SANTEXT and CUSTEXT, while being effective, show some limitations. Tong et al. 2024 critique the small adjacency list used in SANTEXT and CUSTEXT, which could expose the methods to embedding inversion attacks and inefficacy in open-ended text generation, where bias matters more. Tong proposes RANTEXT, which dynamically generates a "random adjacency list" for each loken, then adds noise to the token's embedding and selects nearby tokens based on the noisy embedding, leveraging the Laplace mochanism to ensure Differential Privacy. This method, developed within a framework for using black-box LLMs in private inference, includes a denoising module to reconstruct private outputs. RANTEXT, tested CNN /Diiy Mail (Hermann et al. 2015), Wikitext-103-vl (Merity et al. 2017) and the ArXiy dataset
29

---

Published in Transactions on Machine Learning Research (01/2025)
Clement al. 2019), generally outperforms other methods in both utility and privacy, with exceptions in some areas where CUSTEXT+ performs slightly better. This comprehensive approach enhances the robustness of textual differential privacy, especially against inference and embedding inversion attacks.
As already discussed, none of these methods is specific to LLMs but they all could be used to preprocess the training dataset for an L.L,, although with some computational overhead given by the usual (huge) size of LLMs' training datasets.
Beyond the exponential mechanism Carvalho al. (2021) critique the exponential mechanism's 's fit for Metric Differential Privacy, noting it fails to consider the density of the space from which outcomes are selected. They propose the Truncated Exponential Mechanism (TEM), which adjusts noise addition based on the density of the embedding space around an imput word, evaluated for utility with a sentiment classifi- cation model and privacy with MIAs, showing substantial improvements over the bascline (Feyisctan et al., 2019b). Adding to these developments, Xu et al. (2020) propose a Regularized Mahalanobis Metric for text perturbation to address the shortcomings of traditional noise injection methods that suffer from low utility due to the uniform application of noise in continuous embedding spaces. By using regularized version of the Mahalanobis metric to add elliptical noise, accounting for the covariance structure in the embedding space, this method aims to ensure sufficient. likelihood of replacement for words in sparse regions without sacrificing overall utility. Tested on Twitter and SMSSpam datasets, this method shows a lower number of unchanged words and a greater diversity of word substitutions compared to the Laplace mochanism, suggesting higher privacy while maintaining comparable text classification performances. These advancements highlight the ongoing challenges and innovations in achieving effective textual differential privacy without compromising the utility of the data.
Known Issues Expanding on the theme of embedding-based perturbation methods, Xu et al. (022) ac- knowledges common limitation in such approaches, where even minor noise could still leave the original embedding as the nearest neighbour. To counter this, they introduce a method inspired by Vickrey auctions to balance the selection between the first and second closest. neighbours using a tuning parameter. This parameter is optimized through a constrained optimization problem aiming to maximize utility while en- suring privacy guarantees. Conducted experiments on Product Reviews, IMDb Movie Reviews, and Twitter datasets reveal that this Vickrey mechanism can enhance utility by up to 50% on real text classification datasets compared to existing methods without compromising privacy, highlighting an innovative stride in balancing the privacy-utility tradeoff in textual differential privacy strategies. Further examining the meth- ods in the field, Arnold et al. (2023) critiques the focus on semantic- based embeddings in existing Metric Differential Privacy techniques, noting their failure to consider syntax specifically. Analyzing these tech- niques' ability to preserve grammatical propertics, they found that privatization mechanisms often privilege nouns even when replacing other grammatical calegories. To address this, they incorporated grammalical categories into the privatization process, using methodologies from Feyisetan et al. (2019a) or Carvalho et al. 2002)). Results indicate that this approach can enhance performance downstream tasks, although it also shows a slight increase in self-substitution, suggesting small decrease in privacy compared to previous ap- proaches. This development underscores the complex interplay between maintaining linguistic integrity and ensuring robust privacy protection in textual data. Adding to the critique, Mattern et al. (202b)) identify significant limitations of Metric Differential Privacy methods, such as the increased noise necessary for long texts or their inability to rephrase effectively. Proposing an alternative, they suggest a paraphrasing model obtained by fine-tuning GPT2, inspired by WWittveen & Andrews (2019). This model introduces noise by sampling temperature based on the exponential mechanism, aiming to balance privacy with natural language generation capabilities. Tested for its effectiveness using the Matthews Correlation Coefficient (MCC) for authorship attribution and sentiment analysis, as well as semantic similarity and perplexity metrics to assess how well the original meaning is preserved and the quality of the generated text, this paraphrasing model proves superior to traditional word-level DP mechanisms. Although these methods show promise, it would be interesting to see more of the generated data used in tasks such as text generation, which is quite rare in the current literature. Most of the presented works focus on smaller tasks and models, but there is no limitation in using these methods to generate anonymized data to train LLMss
30

---

Published in Transactions on Machine Learning Research (01/2025)
5 Solutions for Preserving Privacy in LLMs: Model
In this section, we explore the various contemporary methods employed to ensure privacy preservation within machine learning models. The focus is four primary approaches that operate directly the model. Firstly, explore the concept of Differential Privacy applied during the training phase ($5.1). This method involves training model with techniques designed to protect the privacy of the data used by injecting noise (for an overview on Differential Privacy, see $2.2). Secondly, we examine the application of Differential Privacy during the inference phase (5.2) where arivacy-preserving mechanisms are applied when the model is making predictions. Thirdly, we discuss federated learning with Differential Privacy ($5.3). which combines federated learning techniques with Differential Privacy provide data safety in decentralized environments. Lastly, we discuss machine unlearning ($5.4), technique that enables an existing model to forget or unlearn specific privacy-sensitive information, and Homomorphic Encryption (55,5,, cryplographic method that allows computation on encrypted data without decrypting it. Each of these methods offers unique benefits and challenges in the pursuit of robust privacy protection in large language models.
5.1 Training Large Language Models with Differential Privacy
Training LLMs with Differential Privacy
Definition: Training LLMs with Differential Privacy consists of all the approaches that apply Differential Privacy at training time during one of the training phases of Language Models.
Training Large Language Model (LLM) typically involves three sequential phases: pre-training broad collection of unannotated corpora (phase 1), fine-tuning procedures (phase 2), and final alignment according to human preferences (phase 3). We refer the reader to $2.1 for an in-depth overview of how these phases work. In this section, we propose methodologies that guarantee privacy by using difierential privacy during the pre-training phase (phase 1 $5.11.1 or apply it during the fine-tuning phase (phase 2 $5.1.2). We also analyze methods that focus on Parameter Efficient Fine-Tuning techniques ($5.1.3) while leaving out of the scope of this survey works that focus on alignment via Reinforcement Learning and differential privacy since, to the best of our knowlodge, there are very few works that just started exploring this aspect (Wu et al. 2024a).
specialized corpus using instruction-tuning
Despite the significant concerns over privacy, most of today's 's LLMs abstain from any privacy-specific measure for pro-training, relying instoad on publicly available internet data for pro-training. While this data is public, it often contains sensitive information, posing risks that might harm individuals and undermine trust. Public data, thus, carries inherent risks yet remains a widely used resource. On the other hand, fine-tuning often requires using private datasets that entities prefer to keep confidential to protect sensitive information or maintain a competitive edge. We refer to 'public training" when the training process involves using datasets that are available to the general public and to 'privale training" when the training process relies on restricted, confidential data that is not available to the public and must be carefully protected for privacy reasons. Here, Differential Privacy could offer a solution, minimizing risks associated with data memorization and safeguarding the privacy of individuals in these datasets.
We have already explained Differential Privacy (DP) in Scction 2.2 here, we contextualize DP with its usage in modern large language models. Specifically, working with DP oflen involves deciding where inject noise into the training process, typically by adding noise to the gradients, as shown in Step 2 of Figure 5.1] This approach generally involves using DP-SGD (Song et al. 2013). a variation of the classic SGD that clips gradient norms and then adds Gaussian noise to the average clipped gradient norm (see 2.2 for more details). It should be noted that, in principle, all the approaches involving DP-SGD are general, as DP-SGD is an optimizer that can be used with any network.
In the following sections, we will focus on how Differential Privacy can be applied to modern language models.
31

---

Published in Transactions on Machine Learning Research (01/2025)
5 Loss
(X2 X)
Labels
X2
X3
Xs+1
X2
X3
Xs+1
Output
Unembedding Matrix
Pretrained Weights
V Gradient Noise
Gradient
VV
Language Model
E Raxd
Input
M(D,
Embedcing Matrix
X1
X2
X3
Xs
Figure 6: Differential Privacy Applied on LMs. Differential privacy represents one of the main solutions used to preserve privacy when training a language model. We have already discussed Differential Privacy (DP) in Section 2.2 here, we contextualize DP with its usage in modern large language models that, most of the time, consist of adding noise in the training gradients (DP-SGD). On the right: Adding noise the gradients of all the training parameters in language model is often expensive. Parameter-offiiiet fine-tuning techniques can reduce the number of training parameters, thus lowering the cost of applying Differential Privacy to these gradients. Picture adapted from LoRA (Hu et al.) 2022), popular PEFT method.
5.1.1 Pre-training with Differential Privacy
Pre-training Large Language Model from scratch is usually a very cost-intensive procedure that just few companics and institutions in the world can afford. In addition to this, adding differential privacy to this pre- training phase further increases the difficulties since (i) adding noise increases the already exorbitant training cost, slowing the convergence time, and (ii) the training phase requires a vast collection of private data, which is often scarce and difficult to obtain. Thus, most models pro-trained from scratch with Differential Privacy are relatively small compared to current Large Language Model standards. Nevertheless, they offer both theoretical and practical insights into why fully private pre-training considered a sub-optimal strategy. Instead, a mixed solution involving pre-training on public data is recommended. In this section, we will explain why. All the works discussed are also summarized in Table 4
Hoory et al. (201) conduct fully private training of BERT, utilizing DP-SGD during both pre-training and fine-tuning stages. The pre-training employs the standard BERT dataset from Devlin et al. (2019). supplemented with the MIMIC-III dataset (Johnson et al. 2016). Fine-tuning is then carried out on the i2b2- 2010 dataset, sourced from the i2b2 National Center for Biomedical Computing and the NLP Shared Tasks Challenges, focusing on entity extraction within the clinical domain. The experiments are conducted with and without Differential Privacy, and the outcomes are measured by the F1-Score. Their findings indicate only minor performance decrement with privacy/utilityy tradeoff.
methods, thus demonstrating favourable
Yin & Habernal 22022) introduce a novel approach to fully pre-train BERT from scratch using a specialized legal tokenizer. They first perform conventional pre-training using BookCorpus and Wikipedia, followed by an additional pre-training phase using DP-SGD on a legal corpus, emphasizing their focus on safeguarding the privacy of the pro-training data. Subsequently, the model undergoes non-private fine-tuning on two downstream tasks: binary classification the Overruling dataset (Zheng et. al. 2021 and multiple-choice
32

---

Published in Transactions on Machine Learning Research (01/2025)
QA on CaseHOLD Zheng et al. 2021). These steps highlight the authors' intent to protect only the data used during the DP-SGD pre-training phase. The outcomes of this approach not only match but sometimes surpass those of non non-private benchmarks, demonstrating the feasibility of maintaining high utility while employing privacy-preeerving methods. However, the study lacks a specific privacy benchmark to assess any potential privacy leakage.
It is crucial to acknowledge that the discussed works were capable of conducting full training from scratch largely because they utilized BERT (Devlin et al. 2019). which possesses a relatively modest 109M parame- ters, figure significantly smaller than those of modern LLMs such GPT-3 with 175B parameters (Brown ct al. 2020). Currently, the standard practice, even in scttings where data are not private, and there is no need to perform DP tuning, involves leveraging & pre-trained model and fine-tuning it for specific tasks. This approach not only reduces computational demands but also allows the model to harness a broad and diverse initial training dataset, enabling it to subsequently adapt more effectively to a private dataset and its specific domain, often enhancing performance. However, the benefits extend beyond performance improve- ments. Recent studies, including Kerrigan et. al. (2020) that compare public pre-training with differentially private fine-tuning against entirely private DP training, demonstrate significantly better results with the former method. This provides empirical evidence that DP learning greatly benefits from an initial phase of public pre-training, Full training solely on a private dataset often results in high perplexity values, indicating a model that would likely be ineffective. In Kerrigan et al. (2020), the comparison of models fully trained privately versus those benefiting from public pre-training followed by private fine-tuning demonstrates that public pre-traiming is essential for enabling effective private learning. Additionally, since DP-SGD introduces noise during training, limiting this noise to the fine-tuning phase rather than the entire training process re- sults in less overall noise and better performance. This improvement is due to the general (clean) knowledge acquired during the pre-training phase. Applying DP-SGD to an LLM pre-training (which is a huge part of the training) would make it extremely slow and add a lot of noise, which could hinder the final utility.
Ganesh et al. (2023) present a study that compares the performance obtained via a public pro-training Tollowed by privale fine-tuning with a scenario where both stages are conducted in a ially private manner. This paper emphasizes the theoretical underpinnings of Differential Privacy in training models, arguing that for differentially private model to be effective on the task it is trained on, it is essential to first undergo pre-training public data. The theoretical basis provided in the study explains that optimization involves two phases: initially, the model transitions from a random initialization towards favourable basin, and subsequently, it addresses a convex optimization problem to find the local minimum within that basin. If attempted entirely within a private context, incorporating private data and noise addition, the initial stage would necessitate substantially more data to achieve comparable outcomes to those achievable with public data, as supported by experimental evidence. Moreover, given the typical scarcity of private data, it might be impractical or even impossible to gather sufficient data to effectively train a model privately from scratch, especially if the model is LLM. Therefore, pre-training on a larger public dataset not only aids in overcoming these limitations but also enhances the model's resilience to distribution shifts between the public and private datasets.
Tramer & Boneh 2021 offer another perspective, arriving at a similar conclusion: public pre-training is crucial for achieving desirable outcomes in a private learning environment. Although this study is centred on image classification using CNNs, the findings reinforce that the advantages of public pre-training transcend various contexts, including this one. In their approach, the researchers enhance the performances of fully private training by incorporating handcrafted features, which yield improved results. Nevertheless, these outcomes still do not match those achieved through a combination of public pre-training and private fine- tuning. The collective insight from numerous studies is that a public pre-training phase is indispensable in private learning scenarios.
While the works reviewed so far performed a "classical pre-training' (i.e., gaining knowledge from a large dataset), Yu et al. (2024) demonstrate that pre-training can also be conducted selectively, i.., by focusing on specific (smaller) subset of data that allows DP training to become more efficient. Their approach involves novel technique of selecting a subset of the pre-training dataset, OpenWeb (Gokaslan & Cohen 2019 whose distribution closely resembles the fine-tuning dataset's distribution, Enron email dataset (Klint &
Yang 2004). To filter the pre-training dataset, a classifier trained with Differential Privacy predicts whether
33

---

Published in Transactions on Machine Learning Research (01/2025)
pre-training sample belongs to the distribution of the fine-tuning (private) dataset. Each sample in the pre-training dataset is then scored for how likely it is to belong to the fine-tuning dataset, then the top-k samples are used for public pre-training. This is followed by differentially private fine-tuning on the private datasct, with the final privacy loss taking into account both the classifier and the ultimate model. This method of dataset selection is designed to minimize distribution shifts, enabling smaller models to achieve better results. The results demonstrate strong performances, surpassing more complex baselines and thus enhancing efficiency to the point where this can be considered a model compression tool. However, & potential oversight of this method is the non-consideration of biases within the fine-tuning dataset, which could lead to the inadvertent transfer of these biases into the model through the selected pre-training samples. This is not as likely when the pre-training dataset is fully utilized.
5.1.2 Fine-tuning with Differential Privacy
Pre-training a large language model from scratch, with or without Differential Privacy, has become wildly expensive, given the sheer size of these models. Currently, the prevalent approach consists of using an already available LLM pre-trained checkpoint and fine-tuning it for specific downstream tasks, including applications of Differential Privacy (DP). In this section, we survey several different works that apply Differential Privacy on top of already pre-trained LMs. These works are also available in tabular format in Table [ While the works in this section are all centered around LMs, they rarely adopt solutions specific only to the language domain, making these methods generalizable to other domains. Yue et al. (2023) show that using DP-SGD, it is possible to successfully train LMs for text generation. They apply DP-SGD to fine-tune various sizes of GPT2 using the Open Yelp Dataset. This is mainly done to generate synthetic reviews that cannot be traced to the actual reviewer. The generated reviews are then used to train classifiers to predict ratings and business categorics. The quality of the synthetic text is assessed by comparing its distribution to the original dataset (through F1-score, Fréchet Inception Distance Heusel et al. (2017), and MAUVE Pillutla et al. (2021) and the classifier performance against models trained the original data. In this task, the largest GPT2 models produce data that more closely resemble the original data, probably because they are less influenced by DP. Experiments indicate that DP introduces a minimal performance drop, often less than 0.01%, within a privacy budget of e = 4. Remarkably, the GPT2-Large model with DP nearly matches the performance of its non-private counterpart, with only a 0.008% difference in accuracy on downstream tasks, This leads to the notion that the pipeline is particularly suitable for LLMs, as it tends to scale well with model size. Privacy testing was performed following the method by Carlini et al. (2019): ; this consists of injecting canaries inside the training set in order to extract them later on, to prove that the model is capable (or not) of memorizing training data. When these canaries were injected into the training data, the model could not be prompted to reproduce them under DP conditions, even with a repetition rate of 100. In contrast, without DP, the canaries were reproduced 80% of the time, demonstrating effective privacy preservation. This work highlights the fcasibility of DP fine-tuning. When we investigate DP learning and the used techniques, there is a constant comparison with standard learning, as it is the most widely known and adopted. Papernol] lt al. (0200) specifically focuses on the differences between private and public training, trying to shed some light on the best conditions for private training. They investigate architectures, parameter initializations and optimizers. We are not very interested in architectures as they do not consider transformers but only convolutional architectures and we already know from $5.1. that it's best to start from parameters that were previously optimized through a public pre-training. The novelty that this research points out is that adaptive optimizers like Adam provide marginal gains (if any) in this context. Putting in direct comparison DP-SGD and DP-Adam, they find that while the latter converges faster initially, it eventually slows down to the point where DP-SGD can outperform it, reaching a lower loss. Another relevant difference they point out is how different a working learning rate is when tuning DP-SGD with respect to standard SGD (with all the other conditions being the same). Definitively this work highlights how DP-tuning, while effectively working, requires to be handled in a slightly different way than a standard tuning and, while this work only considers CNNs, there is no reason to believe that the findings would not apply to LLMs.
Li et al. (2022), building on the work of Papernot et al. 2020 on how to approach private-tuning, explore the effects of hyperparameters and fine-tuning objectives, identifying typical pitfalls of DP tuning when it's treated as normal tuning. Their findings are particularly relevant as they suggest that, with proper execution, the performance of DP-tuned models can rival strong non-private benchmarks They introduce
34

---

Published in Transactions on Machine Learning Research (01/2025)
technique known as Ghost Clipping. Ghost Clipping is an important step in the landmark of private tuning because it greatly reduces memory usage, aligning the efficiency of private learning with that of non-private learning. Before that, DP-tunnng was much slower and heavier than normal tuning to the point where it was almost not feasible (especially for large batch sizes, which are necessary for models like BERT) and even if it was, it would reach much lower performances. Ghost Clipping is an expansion of the concept described in (Lee & Kifer 2021), it avoids the nced to instantiate per-example gradients for each linear layer by handling one layer at a time and necessitating an additional backpropagation pass per batch. Extensive testing across various language generation (NIGG), and both GPT-2 and DialoGPT for chit-chat dialogue generation- that private models perform nearly as well as public ones, with only minor differences in perplexity and BLEU scores, showcasing competitive results. When compared to using vanilla Opacus, Ghost. Clipping shows a 3x improvement in efficiency. While Ghost Clipping is already a big step in efficiency, Ding et al. (2024) points out that this method cannot be applicd to standard Transformers bocause of non-plain food ford-forward (and back-propaggtion) lopology caused by embedding sharing that hinders Differential Privacy guarantees. The authors proposed Phantom Clipping, which is able to handle parameter sharing thanks to the manipulation of tensor operations. This ensures that the shared parameters' gradients are computed and clipped without redundancy, hence preserving the Differential Privacy guarantees. They also show and solve another problem typical of DP-tuning with transformers: attention distraction. Attention distraction is a phenomenon that occurs when noise added in DP-SGD to maintain privacy distorts the self-attention mechanism, typical of transformers networks, causing the model to inaccurately weight parts of the input data, This leads to less effective learning as the model might focus on irrelevant details or overlook important information. To address this, the proposed 'Re-Attention Mechanism recalibrates attention scores by tracking and adjusting for the variance introduced by the noise. This adjustment ensures that attention scores more accurately reflect the true underlying data relationships despite the noise, thus maintaining the model's performance and efficiency under differential privacy constraints. It should be noted that these issues and fixes are general to transformers model and not specific to LLMs. However, they can greatly speed up fino-tuning for LLMs, which quite important given their size. Ghost Clipping and Phantom Clipping are already a big step towards making private learning as efficient as standard learning, but there is still a gap to close. In order to get even closer, Bu et al. (023b) advance the methodology proposed by [i et al. (2022) with novel Book-Keeping (BK) technique. This new method requires only single back-propagation round and does not necessitate the instantiation of per-sample gradients. As a result, it significantly enhances throughput compared to Ghost Clipping while maintaining a similar memory footprint. This means that this new tochnique is a big step forward as it offers a more scalable solution for handling data of higher dimensions.
RoBERTa for natural language understanding (NLU), GPT2 for natural
It is often possible that there is some overlap between pre-training data and fine-tuning data. While this is usually not an issue, in a private setting, where only fine-tuning is done privately and pre-training is standard, this kind of contamination would result in data leakage that could expose sensitive data. In this sense, the work done by Kurakin et al. (2024) is important as it addresses the potential data contamination between pre-training and fine-tuning datasets. To tackle this issuc, they employ an LLM based on LaMDA (Thoppilan et al. 2022), initially applying deduplication techniques [Lee et al. 2022) to ensure fine-tuning data are excluded from the pre-training dataset. Following deduplication, they proceed with standard pre- training and subsequently conduct private fine-tuning using both full and Parameter Efficient Tine-Tuning (PEFT) methods. The model is then utilized to generate a synthetic dataset for a downstream sentiment analysis task, employing BERT and WordCNN for classification. Although the results are good, showing only minor performance drops for DP-tuning with respect to standard tuning, it is important to note the use of relatively high privacy budgets (c.g., c=1)) Moreover, this approach is tested only on binary classification tasks, as opposed to the more complex multi-class classification challenges addressed by Li et al. 2022
Mattern et al. (202aa) introduce a variant of DP fine-tuning by applying it to GPT2 for generating private versions of IMDb and Amazon reviews datasets, which are then used to fine-tune models for classification. Following the methods of SSchick & Schütze| (2020), during the fine-tuning of GPT-2, they enhance sample generation fidelity by incorporating instructions based on the original dataset's characteristics, such sentiment or category, into templates. Since the datasets are made for classification, the data is labelled. The idea is to use these labels to steer the generation, also during training, by asking specifically for a sample corresponding to a label, like, for example, asking for a "positive" review. This approach integrates
35

---

Published in Transactions on Machine Learning Research (01/2025)
classification labels directly into the LLM's fine-tuning prompts to prevent generating mismatched samples (like positive review that reads like a negative one) and introduces a novel loss function that penalizes such mismatches. Since this method takes advantage of prompts to steer the generation, it is specific to LLMs. After training, the model samples a subset of the original training set, which is then utilized for further fine- luning on binary classification lasks. For comparative analysis, the authors also fine-tune models using DP on the actual data. The main result the demonstration that models trained on synthetic data generated via DP and then fine-tuned in a non-private manner outperform those trained directly on real data with DP. Additionally, the researchers assess data privacy by examining duplicates between original and synthetic datasets, noting a significant reduction, with duplicates completely eliminated in some instances. In this study, similarly to what was shown by Kurakin et al. (2024), the downstream task is binary classification, which is not particularly hard.
Differential Privacy (DP) introduces various challenges, notably due to the addition of noise. One such issuc, the incrcased memory consumption from noise addition, is addressod by Ghost clipping (Li et al.) 2022). To further miligate these challenges, [Behnia et al. (2022) introduce a new framework designed to reduce the overall amount of noise required to maintain privacy guarantees. This framework utilizes the Edgeworth accountant (Wang et al., 2023), which provides a 'finite sample guarantee" unlike the typically asymptotic nature of DP guarantees. This is particularly advantageous for fine-tuning an LLM where the number of tuning steps is relatively limited. By calculating the necessary noise levels based on privacy parameters and the number of compositions for a specific dataset, this new framework siguificantly reduces noise impact by up to 5.6%. Additionally, it achieves a performance improvement of up to 1.1% across various natural language understanding (NTU) tasks compared to previous benchmarks. This makes the approach relevant as it not only enhances privacy but also improves efficiency in DP applications.
Although pre-training on public data followed by fine-tuning on private data has long been considered the standard and safest method for private model training, recent evidence suggests that even this approach carries cortain risks. Feng & Tramèr (2025) recently showed how privacy backdoors can be injected in pre-trained models, making them susceptible to different kinds of allacks. This research explores how neurons can be subtly manipulated to retain their primary function while gaining malicious capabilities. By cnabling neurons to hold a gradient for a single input and 'deactivate' to prevent overwriting during subsequent training, the model becomes vulnerable to gradient inversion attacks, similar to those used in federated learning models (see 3.3 for details). While such attacks typically require white-box access, the researchers also demonstrate attacks on black-box models. Specifically, they enable membership inference attacks (MIAs, see 3 by modifying units in the model's sccond-to-last layer, which are uniquely activated by specific training data points. By adjusting the model's activation, they can determine whether a sample was part of the training set by comparing logits before and after fine-tuning. Although the study only tests on MLPs, ViT, and BERT, and it remains unclear if these vulnerabilities apply to LLMs, it highlights a significant risk in the public pre-training and private fine-tuning paradigm: if the pre-trained model or its provider is untrusted, loose privacy budgets commonly assumed in the literature are unsafe. Any efforts to establish tighter privacy guarantees must account for the model's integrity.
5.1.3 Parameter-Efficien Tine-Tuning with Differential Privacy
Modern large language models (LLMs) often contain billions of parameters, making even full fine-tuning impractical due to high computational demands, particularly when paired with noise injection for Differential Privacy (DP). Instead, Parameter Effficient. Fine-Tuning (PEFT) methods like adapter tuning (Houlsby et al., 2,19), compacter tuning (Karimi Mahabadi et al. 2021 prompt tuning {Lester et al. 2021), prefix tuning [Li & Liang] 2021), and notably LoRA (Hu et al. 2022 have become more prevalent. Around the same time LoRA came out, also Reparametrized Gradient Perturbation (RGP) was introduced (Yu et al. 2021 although with less success. RGP is PEFT technique specific to the field of DP. It enhances efficiency by reconfiguring each weight matrix into smaller matrices, optimizing gradient storage and compulation. These techniques, which involve adapting a subset of the model's parameters or incorporating external parameters, facilitate faster training and enhanced modularity. They allow the model to manage various tasks without requiring complete re-training. Remarkably, even with fewer parameters being updated, these methods often achieve performance that is comparable to or even surpasses full fine-tuning, depending on the specific task
36

---

Published in Transactions on Machine Learning Research (01/2025)
and dataset (Sun et al. 2023 Suri et al. 2023 Because of the huge number of parameters in LLMs, they seem to suit particularly well these methods, even though there is no theoretical limitation in applying them to other models. In this section, we survey methods that apply Differential Privacy with PEFTT techniques in order to reduce the computational burden of fine-tuning while keeping privacy standards. A synthetic overview of these methods is available at
Yu et al. (002)) applied all the PEFT techniques just mentioned with DP RoBERTa and GPT2 without any particular challenge. As matter of fact, they achieve performances nearly equivalent to their public counterparts, especially with RoBERTa, where many PEFT techniques with DP notably outperformed full fine-tuning. The study also highlights the efficiency of DP when combined with LoRA in terms of speed and memory use, underscoring the higher viability of private learning in a PEFT framework. This is done by using DP-SGD with LoRA.
In a continuation of similar rescarch, Li ct al. 2022) conduct tests with both RoBERTa and GPT2, employing DP-Adam, which is similar to DP-SGD but based on Adam, for updates and moment accumulation) and the Ghost Clipping technique. This introduction reduces memory usage, making the difference in efficiency between private and non-private learning negligible (see $B for more info). At the same time, also the performance drop is proven to be negligible, making DP-PEFT viable option. GPT2 is evaluated in tasks like table-to-text and chit-chat generation, where the performance of full fine-tuning and PEFT proves to be closely matched, with full fine-tuning tue-tuning showing a slight edge both with and without Differential Privacy. This indicates that PEFT can match the efficacy of full fine-tuning even in a private setting. Additionally, the results challenge previous assumptions seen in the work of Yu al. (2022). where PEFT's superior performance was attributed to lesser noise addition. The findings here suggest that higher noise levels in full fine-tuning might not detrimentally affect the privacy/ / /utility tradeoff as previously thought. This observation prompts further inquiry into the mechanisms of private learning, offering a pathway to deeper insights into how different fine-tuning methods impact model performance under privacy constraints.
While LoRA remains the predominant technique for private tuning of language models, alternative methods also demonstrate effectiveness. Li et. al.] (2023e) introduce RAPT, a privatization framework designed for prompt-tuning and prefix-tuning on a model that does not necessarily need to be locally run. This is made specifically to make it possible to privately train a remote language model through prompt tuning APIs, like Nvidia NeMo, This involves adding noise to the prompts before they are sent, addressing the issue that direct prompt tuning on privatized data can significantly impair downstream performances. To counteract this, they propose a novel privatized token reconstruction task, which is trained concurrently with the downstream task. The experiments utilize BERT and T5 models across various datasets such as SST (Socher et al.] 2013), QQP (Lili Jiang, 2017), and TP-UK (Hovy al. 2015), with adjustable privacy parameters. Privacy is assessed through the defensive capabilities against both an embedding inversion attack (sce $3.3.1 and an attribute inference attack, where both prompt-tuning and prefix-tuning show enhanced robustness. Utility tests reveal that performances suffer without the use of the privatized token reconstruction, which significantly boosts performance and confirms its efficacy in helping LLMs develop better representations.
5.2 Inference with Differential Privacy
Inference with Differential Privacy
Definition: Inference with Differential Privacy consists of all the approaches that apply Differential Privacy at inference time without changing internal parameters. DP is used in two classes of DP- inference approaches: differentially private in-context learning (ICL) noising-denoising external modules.
The hods presented here, be them in the first. or second class, are all specifically devised for LLMs as they take advantage of prompting.
The first class of DP-inference models is based on In-context learning (ICL) (Brown et al. 2020 (see 22.1 Indeed, ICL may be used to increase privacy in LLMs since private data be given to LLMs only
37

---

Published in Transactions on Machine Learning Research (01/2025)
Query
Differentially Private Answer
Query
Output
Stochastic
Query
Output 2
Subsampling
Private Aggregation
LLM
Sensitive
Dataset
Quary
Output M
Figure 7: Differentially Private In-Context Learning. High-level dcscription of applying Differential Privacy during in-context learning: the original dataset is used to create several subsamples via stochastic sampling procedure; these subsamples are then combined with the user query (prompt) and fed to an LLM; all the generated outputs are aggregated in private fashion, producing single differentially private prediction.
inference time. Thus, this private data should not be included in generalistic LLMs and can stay under the control of the owners holding the data rights. For example, if a medical decision on a specific patient should be taken by considering all patients in specifie hospital, these patients can be given as part of the prompt in in-context learning (ICL) approach. However, nothing guarantees that a malicious user can try to extract this personal data by using specific prompts following the general ICL prompt (sce $3.1.2.. This is an active area of research that aims to embed Differential Privacy mechanisms in inference, performing what is called differentially private in-contert learning. The work presented here all follow a similar high-level structure represented in Figure 5.2; the original dataset is used subsampled or partitioned in several subsets; the subsamples are then combined with the user prompt and fed to an LLM; all the generated outputs are aggregated in private fashion, producing single differentially private prediction.
To address the issue rising with ICL, Wu et al. (2024b) introduce novel approach, called Private in-context learning, to ensure privacy in ICL operations by partitioning the private dataset into M subsets, each containing N samples. These subsets are used as N-shot examples for generating M different ICL prompts, which in turn produce responses for the next token prediction or output classification. Results from cach prompt. are then privately aggregated using techniques tailored to the specific task: (1) Report-Noisy-Max with Gaussian noise for text classification that ensures differential privacy; (2) aggregation methods for text generation. Two aggregation methods are explored: Embedding Space Aggregation, where outputs are mapped to an embedding space to compute and noise the mean output, followed by querying the LLM multiple times to find the closest match with cosine similarity, and Keyword Space Aggregation, which involves noisy top-k counting of words across outputs using mechanisms like the Joint Exponential Mechanism Gillenwater et al.) 2022) or Propose- Propose-Tess-Raaee Rest-Releasee (Zhu & Wang 2022). Despite operating within a low privacy budget (e-g., == 1), Private in-context learning (ICI.) achieves performance comparable to non- private methods across various tasks such as text classification, document question answering, and text summarization. This method not only demonstrates minimal performance drop with added privacy but also offers less computationally intense alternative to fine-tuning with DP, providing practical, lightweight solution for enforcing privacy in LLMs.
Tang et al. (2024) added another security level on top of the method of Wu et al. (2024b): their hoth generates samples with DP before giving the input as in-context learning to LLMs. Then, the rest is the same, that is, possibly dividing the private dataset into M subsets containing the N samples. Clearly, Tang lt al.] (2024) show that the distortion of the input private data samples has little influence with respect to target applications. The effectiveness of this method is examined over benchmarks on standard Text
38

---

Published in Transactions on Machine Learning Research (01/2025)
Classification and Information Extraction datasets. Performance results are presented as the mean and standard deviation across five runs. The private models demonstrate performance on par with non-private models, indicating that it is feasible to achieve private learning without sacrificing accuracy. The study also evaluates the performance of Gaussian and Exponential mechanisms in the aggregation phase, with the Gaussian mechanism consistently showing slightly better results. This reinforces the viability of using synthetic data for privacy-preserving ICL applications.
To improve the level of security in these ICL-based approaches, Duan et al. (0224a) propose to use teacher- student approach where real data are given to the teacher model in way similar to WW et al. (2024b). The teacher model is then used to give synthetic examples for the ICL of the student model. In this way, the student model, which is exposed to final users, does not have access to real data. The method is called PromptPATE, and follows the same general workflow of the standard Private Aggregation of Teacher Ensembles (PATE), presented by Papernot et al. (2018): first, an ensemble of teacher models is trained, then private knowledge transfer is performed, and lastly, a student model is trained. All this training has to be intended as ICL. In this case, instead of training the teacher models, the private dataset is used to generate set of disjoint prompts for the LLM, so teacher models (or the 'flock of stochastic parrots") are actually models performing ICL with disjoint parts of the original datasct. The private knowledge transfer is basically a majority voting from the teacher models made in order to choose the next token prediction. The noisy majority voting is done with the Confident GNMAX algorithm (Papernot et al. 2018) just like in the standard PATE. The chosen token added to the public input to create a single, differentially private example that can replace private data for ICL on a student model so that the student model does not actually need to undergo real training but can take advantage of the elasticity of prompting. Similarly to [Tang et al. (024), this approach also involves generating synthetic samples for later use in ICL. This method, too, achieves performance comparable to non-private methods, even under stringent privacy conditions.
Hong et al. (2024) present a similar work, with a different aim, which is prompt engineering, and some differences in the aggregation mechanism, called DP-OPT (Diffcrentially Private Offsite Prompt Tuning). In this approach, an ensemble of local models receives disjoint subsets of data to generate the next token. The Exponential mechanism is then employed to privately select the token that appears most frequently, Following this, a Deep Language Network, as described by [Sordoni et al. 2023), is used to identify the optimal candidate prompt, applying the Exponential Mechanism again to ensure confidentiality. A significant advantage of this method is that the I.I.-engineered prompts are transferable, often yielding results on par with non-private alternatives, depending on the task and the underlying model. The primary distinction here is that DP-ICL is utilized to develop prompts based on training data without any data leakage. This technique demonstrates competitive results on downstream lasks and effectively transfers across several models, including GPT3.5, Llama2, and Vicuna.
The second class of DP-inference models is introduced by Mai et al. (2024). Their method involves parti- tioning the network into three components: local encoder responsible for adding Laplacian noise to the data, a remote (often black-box) model such as GPT2, T5, BERT, or Llama2 that processes the privatized embeddings and returns the output, and a local denoising module that refines the output. This configuration enables the use of black-box models while maintaining privacy by adding local preprocessing and postprocess- ing steps. Since the text is perturbed locally, Differential Privacy is applied on the input, which corresponds to 1) in Figure 5.1 (DP on input). Testod on various downstream tasks, this method shows promising per- formances, closely trailing behind its non-private counterparts. Additionally, the study conducted inversion attacks to assess robustness, revealing that BERT is particularly resilient at lower noise levels, GPT22- XLarge performs better under higher noise conditions, and T5 appcars to be the most susceptible, requiring stringent privacy settings to ensure safety.
Similarly, Tong et al. (2024) propose InferDPT, which includes a local perturbation algorithm called RANTEXT (discussed in 11..22). The perturbed text is sent to a remote black-box LLM, and a local extraction module smaller model - processes the perturbed output, reconstructing it to ensure co- herence, consistency, and alignment with the original prompt. Users then combine the original prompt and the perturbed generation results to produce the final output. InferDPT, using the RANTEXT mechanism, achieves text generation quality comparable to non- non-private GPT-4, maintaining high utility despite the added privacy-preserving perturbations,
39

---

Published in Transactions on Machine Learning Research (01/2025)
Central Server
@ Noisy aggregation
Local gradients
Local gradints
Model updatees
Modal updaten
LLM
①
Local Training w/ Noise
LLM
Local Training
LLM
Local Training w/ Noise
w/ Noise
Outanst
Dateaet
Datenet
Figure 8: Fedcrated learning with Differential DiffereDiffe Privacy in federated setting can be applied according to 2 different models: 1) in the Local model, local gradients are clipped and noised before being sent to the central server; or 2) in the Central model plain gradients are sent to the central server which aggregates them in a noisy manner.
5.3 Federated Learning with Differential Privacy
Differential Privacy and federated learning are two complementary techniques in privacy-preserving machine learning that tackle different aspects of data security. Federated learning ($2.3) is a distributed computation method that enables learning on private data from varions sites without transferring the data offsite. How- ever, this alone is not sufficient for data security, as gradient inversion attacks can potentially recover the original training data by intercepting gradients sent from clients to the server (check discussion in $3..22.. While federated learning helps mitigate memorization issues (Takkarr et al.) 2021), ; real privacy guarantees require Differential Privacy. In federated learning, Differential Privacy can be implemented in different ways: Figure 5.3 illustrates its application either locally (local Differential Privacy) or globally (central Differential Privacy).
A clarification is needed: central and local models of Diffcrential Privacy (DP) are not exclusively related to federated learning (FL). In central Differential Privacy, perturbation is applied at a global level, typically during aggregation. In contrast, local Differential Privacy applies perturbation locally before the data leaves the site. While DP learning often aligns with FL and involves perturbing gradients, DP was initially de- veloped for the statistical analysis of databases, and both central and local models exist independently of learning processes. $ provides several examples of techniques that fall under the local model of DP, where text perturbation is applied to create noisy private dataset that can be shared later, while $ contains examples of techniques of central Differential Privacy, where the noise is applied at aggregation time
Unfortunately, at the moment of writing, the intersection of differential privacy and federated learning did not produce much that is very relevant for LLMs, mainly due the size of the models, which makes difficult parameter sharing and updating, and due to data dimensionality, which is an important constraint for text. There is still some interesting research came across, even though not very relevant for LLMs, hence we leave the discussion for that in the appendix at B
The most notable example of central differential privacy is Google's implementation in Gboard. Xu el al. (2023b) discuss and analyze twenty Gboard language models trained for Next Word Prediction using DP- FTRL (refer to $B for a thorough explanation), indicating that all future Gboard models will incorporate differential privacy guarantees. In their DP-FTRL setup, model updates undergo clipping, and noise is ap- plied during global model updates, ensuring Central Differential Privacy. Adaptive clipping, which adjusts
40

---

Published in Transactions on Machine Learning Research (01/2025)
the clip norm each round by privately estimating the model's norm at a targeted quantile described by Andrew et al. (202T), was also utilized. However, adaptive clipping did not show significant improvement and was less robust during experiments with large report goals, sometimes causing catastrophic failure or a lack of progress in the first 1000 rounds. Despite these issues, adaptive clipping can help reduce hyper- parameter tuning when the privacy budget allows. Public pre-training on C4, as explained by Wang et al. (2024b), reduced the rounds needed to achieve target utility by about 1000 under the same noise multiplier, thereby enhancing privacy guarantees. Additionally, DP-FTRL was combined with Secure Aggregation, a ctyptographic protocol ensuring the central server only accesses aggregated updates and not individual client updates, as described by Bonawitz et al. (2017). While Secure Aggregation supports DP, it introduces con- siderable computational slowdown and requires different system configurations for large report goals. The techniques for training language models with differential privacy in Gboard have resulted in models with stronger privacy guarantees than some established standards, such as those used by the US Census Bureau. In Europe, these differentially private neural network models outporform older N-gram models in terms of utility. In the US for English and in Brazil for Portuguese, the new models are comparable to or slightly better than their non-private versions.
For the local model of differential privacy, most works (as discussed in $ use it for telemetry data collection, which means that they use very low dimensionality data. Data and model dimensionality are very problematic in this setting. Luckily, some recent developments can mitigate these problems. There is a very recent work (Liu et a 2024b) that shows how this model can be applied to an LLM thanks to LoRA. In this work, the authors developed a federated version of LoRA with Differential Privacy, which allows the sharing of a much smaller number of parameters while keeping the privacy guarantee. The method makes use of LoRA to locally train the model, performs clipping and noising at local level and then uploads the weight updates to the central server. They use this method to fine-tune Llama-7B and ChatGLM-6B for general and medical question answering, respectively, using SlimPajama and medical dataset crawled from two online healthcare services flatforms. The results for several and 5 show that, once again, there is a degradation in performance when using DP, but it is mostly negligible, especially considering the privacy gain. Unfortunately, in this paper, there is no concrete experiment to test to which extent this method prevents training data leakage.
5.4 Machine Unlearning
Machine Unlearning
Machine unlcarning refors to the process by which a machine learning model is able to forget or remove knowledge about certain data points from its training (Cao & Yang 2015).
Recent regulations like GDPR guarantees the "right to be forgotten', ; which allows individuals to withdraw their consent and request the deletion of their personal data (Zhang al., (024b)). Unfortunately, differential privacy (DP) cannot guarantee this because, for a data sample to be truly 'forgotten', it must have zero inffuence on the training process. This requirement translates to privacy budget of e = O, which would prevent the model from learning anything.
Machine Unlearning (Coo & Yang 2015) is an orthogonal field to Differential Privacy, focusing on how to climinate the influence of specific data samples. The key works related to this topic are summarized in Table 7 in the Appendix. The aim to make the model equivalent to one trained on a dataset that excluded unwanted entries, as illustrated Figure 5.4 While test-time prediction to mcasure the influence of training data have been proposed defining influence functions (Koh & Liang 2017), this technique does not reveal the impact of individual data point model parameters. Although training strategies designed to ensure unlearning have been proposed and tested for different architectures (Bourtoule et al. 2019), most of the unlearning solutions for LLMs adopt the idea of modifying model parameters after the training phase.
Jang et al. (002)) propose a method focused on unlearning specific information, particularly for language models. This method negates the training loss function performing gradient ascent maximize loss on target sequences, enabling unlearning with minimal performance loss on unrelated sequences. Sequential unlearning proves more effective than batch unlearning. Wang et. al. (202cc) propose to perform the un-
41

---

Published in Transactions on Machine Learning Research (01/2025)
Learning
LLM
Remove
Unlearning
Relearning
LLM
Figure Machine Unlearning. Machine Unlearning refers to the process of removing the influence of specific training samples from machine learning model. This process aims to make the model if it never trained on the removed dala. Essentially, it ensures that any impact the specified data had on the model's parameters is reversed, effectively erasing its contribution. The image shows that the only real alternative to Unlearning is Relearning, which theoretically works but is extremely impractical for big models like LLMs.
learning only of problematic spans in the sentence, rather than on entire instances to minimize the impact on performance. Despite its elfectiveness, Eldan & Russinovich (2024) show limitations of gradient ascent in certain scenarios, such as making Llama2-7B forget about Harry Potter. The challenge lies in the high likelihood of several candidate tokens related to Harry Potter, necessitating many gradient descent steps. To address this, they map Harry Potter terms to generic equivalents and then compare predictions from a baseline model with mapped input and reinforced model trained on Harry Potter with the original input. By reducing the likelihood of tokens predicted by the reinforced model, they finc-tune the bascline model using generic labels. This results in the model losing familiarity with the targel knowledge, with minimal leakages resembling secondary knowledge. Performance drops vary by task but remain generally tolerable. Recent findings suggest that gradient ascent should be combined with other techniques to be effective while maintaining model utility: Yao et al. (2024a) describe unified unlearning framework for LLMs and find that gradient ascent. more robust when combined with gradient descent on a retain dataset.
Chen & Yang 2023) propose creating unlearning layers and training them with selective saudent-teacher objective based on KL-divergence. The student model (the additional layers) maximizes divergence from the teacher model (the LLM) the data to be forgotten while minimizing it on the rest. Different unlearning layers handle different data, but they also provide a method to fuse these layers into one. The model is evaluated on three sets: the Retained Set (where high performance is desired), the Forgot Set (where low performance is desired), and a Test Set. This method yields excellent results, outperforming other unlearning techniques on the Test Sct., maintaining raw performance while effectively unlcarning the target data.
Unlearning also be achieved with Reinforcement Learning through an approach called DeMemorization [Ksssem et al. 2023). This method takes samples from the pre-training dataset, divides them into prefix and suffix pairs, inputs the prefix, and uses negative similarity metric to evaluate how different the generated sullix is from the real one. This metric serves as reward signal, encouraging the language model to develop a paraphrasing policy that generates different tokens to minimize memorization. DeMemorization achieves better performance on downstream tasks because doesn't remove samples from the dataset. However, for the same reason, shows less robust privacy performance than actual unlearning methods, though the privacy performance remains high.
42

---

Published in Transactions on Machine Learning Research (01/2025)
In Ishibashi & Shimodaira (2023) it is argued that unlearning approaches can lead to hallucination and may not make the generation harmless. To address this, they propose a different approach called knowledge sanitization. This approach uses LoRA to instruct the model to respond with a sanitization phrase like 'I don't know" when encountering potentially harmful prompts. To achieve this, they build a training set where answers to harmful quest are removed. This method, similar to Chen & Yang (2023), is evaluated on both data to be retained and data to be forgotten. The performance is high for both retaining and forgetting targets.
For large language models, In-Context Learning can be used for unlearning, demonstrated by ICUL (In- Context UnLcarning) (Pawelezyk et al. 2024). This technique builds context where the data point to be unlearned is included but with a flipped label (e.g., changing positive label to a negative) alongside correctly labelled examples. Compared to Jang et al. (2022), this approach achieves similar performance for both unlearning and the primary task without requiring any parameter updates. While this method was tested on a binary sentiment classification task, theoretically, could be appliod to other tasks by incorroctly labelling categorical samples, though no experiments have been conducted in such contexts to our knowledge.
Model Editing can also be used, in principle, to perform unlearning of private information. Building the idea that the linear layers in the Transformer architecture can be interpreted as key-value memories that store information (Geva et al. 2021]; ; different model editing techniques have been developed to modily without retraining - these specific parameters. Meng et al. (2022) introduce ROME (Rank- One Model Editing), allowing factual predictions to be traced to specific layers for individual manipulation. Furthermore, a similar approach called MEMIT (Meng et al. 2023) proves to effectively modify multiple examples without affecting the utility of the model. While grey-box attacks that examine intermediate hidden states can still recover deleted data 38% of the time (Patil et al. 2024), and being mostly focused on factual associations, ROME and MEMIT show potential for tracing other data types, substituting the private information with an anonymous one.
With & more specific focus on privacy, Wu et al. 2023) introduce DEPN (Detecting and Editing Privacy Neurons) inspired by ROME. This method assume that privale information, like factual information, might be encoded in specific neurons. They propose detecting these neurons and setting their weights to zero to block information flow. Results show good validation perplexity, sometimes surpassing models trained with Differential Privacy but with higher privacy risks. However, DEPN significantly reduces training time, offering better balance between computational efficiency and performance. |Vnditti et al. (0224) focus on the idea of breaking the association between an individual and their private information inadvertently included in the training data. Their method is called PAE and is based on MEMIT: PAE aims to make multiple edits feasible, both in a sequential and batch fashion, by taking into account the norm of the update matrix with respect to the original weights without sacrificing the model's utility.
5.5 Homomorphic Encryption
Homomorphic Encryption
Homomorphic Encryption (HE) is a cryptographic technique that allows computations to be performed directly on encrypted data without the need to decrypt it first. The result of these computations, when decrypted, matches the result of operations performed on the plaintext data.
Definition:
Homomorphic Encryption (HE) offers thooretical framework for preserving privacy machine learning by enabling computations on encrypted data. In the context of Large Language Models (LLMs), HE can be considered for performing inference sensitive inputs without exposing the raw data to the model provider. This could be particularly useful when the LLM is hosted on external servers or cloud services, and users wish to keep their input data confidential. Regarding training, HE is likely to not be able to prevent memorization as the patterns in the plaintext that are usually memorized by LLMs should be preserved by the eneryption step.
The application of Homomorphic Eneryption to LLMs aims to address privacy concerns during the inference phase by ensuring that sensitive user inputs remain encrypted throughout the computational process. Users
43

---

Published in Transactions on Machine Learning Research (01/2025)
encrypt their queries using an HE scheme and send the encrypted data to the server hosting the LLM. The server performs computations directly on the encrypted data and returns encrypted result, which the user can decrypt to obtain the plaintext output. While this approach theoretically provides strong privacy guarantecs, several significant challenges hinder its practical implementation with LLMs.
First, computational overhead and latency are substantial. HE schemes, especially those capable of supporting the complex operations required by deep neural networks, introduce significant computational overhead. Operations on encrypted data are significantly slower than their plaintext counterparts, often by several orders of magnitude. This performance penalty makes real-time inference with LLMs impractical using current HE technologies. Second, model complexity and compatibility pose challenges. LLMs typically consist of billions of parameters and involve complex architectures with non-linear activation fune- tions, such as GELU or Softmax, and operations like layer normalization and attention mechanisms. Many of these operations are not directly compatible with standard HE schemes, which primarily support arith- metic operations (addition and multiplication). Approximating these non-lincar functions with polynomial functions compatible with HE can lead to loss in model accuracy and additional computational complex- ity. Additionally, memory and storage requirements increase significantly. Encrypted data under HE schemes can be much larger than plaintext data due to encryption overhead. This ciphertext expansion results in higher memory and storage requirements on both client and server sides, making it challenging to handle large encrypted inputs and model parameters. Finally, there is limited support for training. While HE can, in theory, be applied to both training and inference, the vast computational demands make training LLMs under HE impractical with current technology. Consequently, most considerations of HE in the context of LLMs focus solely on the inference phase.
Current Research and Limitations Research into applying Homomorphic Encryption to neural net- works has shown some success with smaller models and simpler tasks. For instance, Gilad-Bachrach et al.] (2016) introduced CryptoNets, demonstrating the feasibility of performing inference on encrypted data with neural networks for basic classification tasks like recognizing handwritten digits (MNIST dataset). Similarly, Al Badawi et al. (2021) proposed optimizations for running convolutional neural networks under HE but were limited to relatively small models and datasets. Extending these approaches to LLMs presents additional challenges.
Recent efforts have attempted to apply HE to Transformer models and LLMs. Hao et al. 2022 proposed Tron, method that modifies Transformers to be more compatible with HE by approximating non-linear op- crations with polynomial functions, aiming to reduce computational complexity while proserving accuracy. Pang et al. (2024) introduced BOLT, which provides ers by reformulating attention computation to be more HE-friendly. Similarly, Chen et al. (2022a) presented THE-X, a framework that rodesigns critical components of the Transformer architecture to suit HE con- straints, optimixing encoding schemes to minimize performance degradation. Zheng el al. (2023) developed Primer, focusing on fast private Transformer inference on encrypted data by introducing optimizations like efficient polynomial approximations and streamlined encrypted computations.
and efficient inference for Transform-
Recently, Hou et al. 2023) introduced CipherGPT, framework designed for secure two-party GPT inference that addresses privacy concerns in LLMs. CipherGPT enables a client (holding the input data) and a server (hosting the model) to collaboratively perform GPT inference without revealing the client's input or the server's model parameters to each other. This is achieved through the use of cryptographic protocols such as secure multiparty computation and homomorphic encryption, optimized specifically for GPT models.
CipherGPT introduces several technical innovations to improve the efficiency and practicality of secure inference with large models like GPT. These include customized secure matrix multiplication protocol using subfield vector oblivious linear evaluation (sVOLE), an efficient method for computing the GELU activation function using spline-based approximations, and protocols for secure top-K selection and sampling required for the text generation process. By addressing the computational and communication overheads typically associated with homomorphic encryption, CipherGPT demonstrates the feasibility of performing privacy- preserving inference with large-scale language models. However, despite these advancements, CipherGPT still faces challenges related to computational overhead and scalability. The secure protocols, although optimized, introduce latency and resource consumption that may hinder real-time applications, especially
44

---

Published in Transactions on Machine Learning Research (01/2025)
with larger models. Additionally, the approximations used for non-linear functions may affect the model's performance and the quality of the generated text. Implementing CipherGPT also requires specialized expertise in cryptography and deep learning, which can be a barrier to widespread adoption.
While these works represent significant progress, they are often constrained by computational overhead and are applied smaller models or simplified versions of Transformers. One major challenge approximating non-linear functions. LLMs rely heavily on non-linear operations not directly supported by HE schemes. Researchers have explored using polynomial approximations of these functions, but such approximations degrade model performance. For example, approximating the Softmax function used in attention mechanisms with a polynomial can lead to less accurate attention scores, affecting the overall quality of the generated text.
Another issue is the pronounced performance trade-off between privacy and utility, With LLMs, due to their size and complexity, this tradc-off is more significant. While HE provides strong privacy by keeping data encrypted, the resulting computational costs and potential loss in accuracy make it less feasible for practical applications involving LLMs.
Furthermore, there is a lack of practical implementations for full-scale LLMs. Although recent works like Rho et al. (2024) have provided implementations for encoder-only models, they still face scalability issues when applied to large models. Li et al. (2023a) introduced MPCFormer, which enables private Trans- former inference using Secure Multiparty Computation (MPC) instead of HE, optimizing the Transformer architecture to be compatible with MPC protocols. While MPCFormer shows improved performance over traditional HE methods, it still encounters challenges with communication overhead and scalability when applied to large-scale LLMs.
While Homomorphic Encryption presents an attractive theoretical solution for preserving privacy during inference with Large Language Models, current technological limitations render it impractical for real-world applications involving LLMs. The substantial computational overhead, memory requirements, and challenges in handling complex non-linear operations make HE unsuitable for deployment with current models. Alter- native
techniques, such as Secure Multiparty Computation (SMC) and Trusted Execution Environments (TEEs), face similar challenges when applied to LLMs. Differential Privacy (as discussed in 5.2 and Federated Learning ($5.1) currently offer more practical approaches for preserving privacy in the context of LLMs, balancing efficiency with privacy guarantees. Ongoing research in cryptographic tech- niques and hardware acceleration may, in the future, alleviate some of the computational burdens associated with Homomorphic Encryption. Until such advancements are realized, HE remains a topic of interest in theoretical discussions but is not a practical solution for privacy preservation Large Language Models.
45

---

Published in Transactions on Machine Learning Research (01/2025)
6 Tools & Frameworks
After discussing the approaches one may take to tackle the problem of privacy-preserving LLMs, we briefly see what frameworks and libraries are already in place for researchers and practitioners interested in the topic.
Data Anonymization. Data anonymization has been extensively used to tackle privacy problems in ML for years, which means several frameworks and libraries can apply anonymization to datasets. The most prominent examples are:
ARX Data Anonymization Tool ARX is a comprehensive open-source tool that provides scala- bility and usability for data anonymization. It supports variety of anonymization techniques, data quality analysis methods, re-identification risk assossments, and privacy models like k-anonymity, I-diversity, (-closeness, and differential privacy. It's updated regularly and is available in Java.
Open Data Anonymizer Aimed at Python users, this library focuses on data anonymization and masking for data science tasks. integrates well with Python's data- hata-handling libraries, such as pandas, making suitable choice for projects involving data frames and machine learning.
Microsoft Presidio Presidio offers detailed anonymization capabilitics for text, including the
identification of personally identifiable information (PII) and its subsequent anonymization or de- identification. It features two main modules: the Presidio Analyzer for detecting PII in text and the Presidio Anonymizer for removing detected PII using varions operators like redaction, replacement, hashing, or encryption.
Spark NLP for Healthcare by John Snow Labs: Specifically designed for the healthcare sector, Spark NLP for Healtheare provides state-of-the-art accuracy for extracting, classifying, and struc- turing clinical and biomedical text data. It's optimized for Databricks, offering solutions like OCR for document processing and NLP models for detecting and obfuscating protected health information (PHT) in unstructured text.
Differential Privacy. We have seen that Differential Privacy has been considered a gold standard for privacy protection in machine learning for years, with several techniques to enforce it and big players using it in very used applications. Naturally, this means that there are many frameworks to implement, some of which are open-source. Most of these frameworks, like most of the research in the field, focus differentially private training. Here are presented the most prominent ones, for the tabular version of this list, refer to G
Similar to Opacus but for TensorFlow, is a Python library that includes implementations of TensorFlow optimizers for training machine learning models with DP. The library has butorials and analysis tools for computing the privacy guarantees provided.
TensorFlow Privacy:
Vycy PyTorc translation of TensorFlow Privacy.
OpenDP project incubated by Harvard University's Privacy Tools and Privacy Insights projects, the OpenDP project (OpenDP 2020) aims to make DP tools available to everyone freely. includes several components: OpenDP Library, a core component consisting of a collection of algorithms for generating differentially private statistical releases, developed in Rust with Python bindings; SmartNoisc,, designed for creating DP reports, dashboards, synopsos, and synthetic data releases. It supports SQL queries over Spark and popular database engines and includes a variely of synthesizers for generating privacy-preserving synthetic data; DP Creator, web-based application, is focused on enabling the calculation of DP statistics for statistical queries intended for public release.
Diffprivlib; IBM's genoral-purpose library (Holohan et al. 2019), stands out for focus on providing a wide range of DP tools for ML and data analysis tasks, supporting experimentation and development.
46

---

Published in Transactions on Machine Learning Research (01/2025)
Google DP: Provides broad set of DP tools in Java, Go and C++. (1) Privacy Beam: end-to-end DP framework built on top of Apache Beam. (2) Three 'DP building block" libraries in C++, Go, and Java implement basic noise addition primitives and DP aggregations. Privacy on Beam implemented using these librarics. (3) A stochastic tester is used to help catch regressions lhat could make the DP no longer hold. (4) A DP accounting library is used to track the privacy budget. (5) A command line interface for running differentially private SQL queries with ZetaSQL. DP Auditorium library for auditing differential privacy guarantees.
tip-fee-shoot-geeratioo Microsoft's contribution to the DP landscape TTng et al. 2024 is the only inference-DP framework you will find in this list. As explained in $5.2 DP in-context-learning a relatively new idea, and this is the only public library to do it, to the best our knowledge.
EKTELO made by Zhang et al. 2018 it stands out for its focus on a flexible, extensible frame- work for privacy-pree preeecvirg data analysis, enabling complex queries and analytics while ensuring differential privacy. As described on the GitHub page, they have 2 main objectives: Isolate private interactions with data in a compact and secure kernel and modularize privacy-related algorithms into operators, which promote code reuse and assist in keeping the kernel compact and secure.
PyTorch Opacus; made by Yousefpour et al. 20021) it enables training PyTorch models with differential privacy. It supports training with minimal code changes required on the client, has little impact on training performance, and allows the client to track online the privacy budget expended at any given moment. This is all taken care of by an abstraction called PrivacyFngine.
privale-rransformerrs Provides a privacy engine built off Opacus but rewritten specifically to facilitate integration with HuggingFace transformers.
dp-transformers It is toolkit that provides simplified integration of transformers training with DP. It also provides some examples and tips for DP training.
Chorus Chorus (Johnson et al. 2018 is unique for its DBDS-indeeenndet architecture, focusing on scalable, DP statistical queries through a cooperative query processing system. This library was actually deployed at Uber.
autodp Automates the process of calculating the privacy guarantees for complex algorithms and supports several standard mechanisms. Users can define their own privacy mechanisms or modily existing ones.
Transformers. The main focus of this work is LLMs, which are based on the transformers architecture (Vaswani et al.] 2017). Most transformers nowadays are implemented with the Transformers Library by HuggingFace, This library quickly became a cornerstone in Al because it provides most state-of-the-ar the-rrt models alrcady pro-trained and ready to be used in a plug-and-play fashion, allowing rescarchers to fast and efficient prototyping. It provides tools for pre-training and fine-tuning like (SFT for supervised learning and TRL for reinforcement. learning. The main reason for the success of this library is its versatility: there are also integrations for distributed training with Accelerate and DeepSpeed and it even supports Parameter- Efficient Fino- Fino-Tuning with an ad-hoc library called PEFT which are all fundamental elements when you use big models like modern L.Mss. There has been some work to try and integrate DP libraries and Transformers, and particular there are two public implementations of Opacus with Transformers: prrvatt--rransfommer and [dp-transformers which both provide implementations of some recent LLMs and other models with the integration of Opacus for Differential Privacy.
Federated Learning. Federated learning is standard when the task requires handling sensitive data, as it makes it possible to keep the data on-site without sending them to a central server and, therefore, gives (false (.3) sense of security. There are several frameworks for federated learning. Let's quickly see the most famous ones:
47

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Comparison of Differential Privacy Frameworks. Categorized here based on the components they offer: Optimizer means that the framework offers some optimizer that be used in combination with any model to train it with DP; Mechanism means that the framework provides noise injection mechanisms like the Exponential Mechanism; Models means that the framework provides some DP models ready to be used; Tn- mn-Contextt-Leammiig means that the framework offers the possibility to perform ICL with your model.
Framework
Optimizer
Mechanisms
Models
ICL
Language
Tensorflow Privacy
X
X
Python
VaVacy
X
X
Python
OpenDP
/
X
Rust
Diftprivlib (IBM)
V
X
Python
Google DP
V
X
X
C+++JJvva, Go
dp- sen.-rttttnnnioonnn
X
X
Python
EK
V
X
X
Python
Opacus
X
X
Python
private- tranoteerransformers
X
X
Python
ap-transformers
X
X
Python
Chorus
X
Scala
autodp
X
X
Python
Flower: Flower (Beutel et al. 2022) is an poon-ssourrc..fanoooo---aagg FL library designed to enable the building of distributed systems with any ML framework. focuses on usability, scalabil- ity, and compatibility, aiming to make FL accessible to researchers and practitioners working with different tools. Flower provides flexible and casy-to-use solution since it allows for experimenta- lion across different frameworks, making it ideal for researchers and developers exploring federated learning in diverse environments.
pfl Apple's pfl library is federated learning library specifically devised for quick research sim- ulations. It also provides tight integration with local and central DP mechanisms, allowing quick prototyping of differentially private federated networks.
TensorFlow Federated (TFF) TensorFlow Federated is an open-source framework for machine learning and other computations on decentralized data. TFF enables developers to simulate FL in their machine learning models using TensorFlow. It's designed for flexibility and ease of use when experimenting with FL algorithms. TFF a natural choice for those who use TensorFlow since it provides scamless integration with existing TensorFlow models and extensive documentation and tutorials from Google.
PySyft PySyft a Python library for secure and private deep learning that extends Pororch. It supports FL, DP, and SMPC, aiming make PySyft part of the OpenMined initialive, which fosters an open-source community focusing on wnivacy-preservin AI.
machine learning accessible.
FATE (Federated AI Technology Enabler) Webank's FATE is an open-source project intended to provide secure computing framework to support the federated AI ecosystem. It's designed for financial inst utions, enabling them to build machine learning models without sharing their data, thus complying with regulations. FATE is particularly suited for organizations in the financial sector looking to engage in federated lcarning. provides a robust framework that emphasizes security and privacy, making it ideal for use cases that require strict data protection measures.
Bindings for Python 'Bindings for Python
OOpacus provides a privacy engine that encapsulates the model and the optimizer Buill off Opacus, which provides privacy engine that encapsulates the model and the optimizer Buill off Opacus, which provides privacy engine that encapsulates the model and the optimizer
48

---

Published in Transactions on Machine Learning Research (01/2025)
P Baidu's PaddleFL is open-source federated learning framework based on PaddlePad- dle, Baidu's ML platform. It supports various FI, strategies and provides flexible definition of federated learning tasks. It's designed with industrial applications in mind, offering tools for model training and inference in a privacy-preservin manner. This is the go-to for those who are already using PaddlePaddlee
49

---

Published in Transactions on Machine Learning Research (01/2025)
7 Limitations and Future Directions
In this section, we provide discussion on current limitations and future directions for preserving privacy in large language models, analyzing all the different approaches to preserving privacy proposed in this survey.
Preserving Privacy in Data. Incorporating privacy directly within data before it is input into large lan- guage model offers a practical alternative to more complex, model-based privacy-preserving techniques. For example, the large-scale pretraining of prominent models like LLaMA3 and OLMo leverages data anonymiza- tion techmiques, applying privacy measures at the data level before it enters the LLM. However, given the vast scale of pretraining data, these projects have largely relied on basic regular expressions to filter out PIT and streamline the preprocessing (Dubey et al. 2024 Soldaini et al. 2024). While efficient, this approach is quite limited as it lacks comprehensive privacy protection and does not offer formal privacy guarantees such as those provided by DP. As SSuramanii et al. (2023) notes, "Rumning an out-of-the-box tool such as Presidio is the bare minimum that should be applied to all new and existing corpora". To support progress in this area, sections 4.1 and G of this survey present practical insights for researchers and practitioners aiming to build the next generation of privacy-preseving LLMs. While current models primarily rely on based filtering for pretraining data, several approaches discussed in section [..1 including metric Differential Privacy methods, can be scaled and applicd to enhance privacy preservation in LLMs. We believe these methods hold significant untapped potential for advancing privacy-preserving techniques in data handling for LLMs and that we are just beginning to explore their full possibilities.
Preserving Privacy in Model. Model-based approaches based on DP offer mathematically parooeeeiilll--ooeeen pri- vacy guarantoes within defined privacy paramcters 5), but they come with significant computational costs and require substantial, often complex, modifications to models and training procedures. Of all the ap- proaches discussed in section 5 DP is considered the gold standard for privacy-preserving machine learning, as it is the only method that provides formal mathematical privacy guarantee. However, this assurance typically comes at the expense of some utility loss, especially when DP is applied throughout the entire learning pipeline (Tramèr et. al. 2024b Feng & Tramer 2025 DP's applicability spans data preparation, training, fine-tuning, and inference. In contrast, the privacy guarantee provided by other approaches like Machine Unlearning ($5.4) is only empirical but necessary to implement the 'right to be forgotten' of pri- vacy laws like GDPR (Zhang et al. 2024bb. Future research could focus on building a rigorous mathematical basis for Machine Unlearning, developing techniques to seamlessly combine it with DP, and implementing accessible interfaces that allow users to specify data they wish to be crased from the model.
Despite the extensive application scope of Differential Privacy, it assumes a structure of private data where every bit of language can potentially contain secrets that are not meant to be shared indiscriminately. Differential Privacy cannot directly preserve secrets - information that can be shared only to the correct group of people (Brown el al. (0022)). The capability of L.MSs to keep secrets is still little explored, with the development of the first benchmarks to assess the ability of such models to disclose private information only to users who should be allowed to accoss them (Mireshghalaah et al.) 2024; Priyanshu et al.) 2023; Wang et al. 2024a). Future research should focus on advancing DP to selectively protect specific language elements, enhancing utility by applying privacy constraints only where necessary. As LLMs are increasingly integrated as assistants in applications, functioning as agents among groups of users who wish to share certain information while keeping other details private, addressing this issue will become more critical.
Looking ahead, applying DP to the training, tuning, or conditioning of larger, more capable models presents both a challenge and an opportunity. Although most current studies focus on smaller models, the advance- ment of quantization and Parameter Efficient Fine-Tunin (PEFT) techniques promises to make DP viable in larger models as well. These advancements reduce computational overhead and have facilitated DP-tuning in smaller models, such as RoBERTa and GPT-2 via LoRA, which could soon extend to modern models like Llama-3 and Mistral with negligible impact on computational resources ($5.1.). We believe that the next step in this field will be the development of with statc-of-the-art models. With the existing building blocks (techniques and modcls) in place, this devel- opment hinges primarily on computational resources for private fino-tuning. Our ideal vision includes models pre-trained entirely with DP, which would significantly mitigate privacy risks from the outset. However, we
LLMs that leverage these techniques, paired
50

---

Published in Transactions on Machine Learning Research (01/2025)
recognize that fully DP-compliant pre-training remains challenging, as it demands significant computational resources and can reduce model effectiveness. A practical compromise could involve pre-training the model using data anonymization techniques, which are less computationally intensive (as discussed in Section (.1} while reserving DP methods for the subsequent stages of instruction tuning and alignment. In the short term, we believe this approach offers a reasonable balance.
Federated Learning ($5.3) offers another promising avenue for enhancing privacy in LLMs, albeit with lim- itations. Although FL with Central DP is feasible, constraints device capacity mean that edge devices, such mobile phones, often lack the processing power to maintain locally stored LLM. This limitation is slowly diminishing, as illustrated by devices capable of running small-scale models like Gemini-Nano locally (Team et al., 2024.. With the ongoing minialurizatiinn and optimization of LLMs, we anticipate that FL- driven, coooompliant training on local devices will become increasingly feasible. Nevertheless, implementing a local DP model remains challenging due to the high dimensionality of the data and model size; PEFT tochniques provide some relicf by minimizing parameter sharing, but more rescarch needed refine utility and privacy outcomes within this framework.
Inference under DP constraints has shown initial success, suggesting that LLMs may offer privacy- privacy-preservine preserving inference without requiring task-specific fine-tuning ($5.2) While current research in DP-constrained infer- ence is limited, early results are promising, especially as models become more compact without sacrificing capability. In practical terms, this progress allows for models are preferable or where specific fine-tuning is unnecessary. Future work could explore larger models, potentially introducing DP to inference on locally stored models, where the option of performing inference without any data sharing would be highly ad advantageous.
inference in cases where offsite
Despite the appeal of Homomorphic Encryption (HE) for privacy-preserving inference with LLMs, significant challenges limit its practicality. As outlined in $5.3 HE introduces considerable computational overhead and latency, making real-time inference with large models unfeasible. Issues with approximating non-lincar func- tions can degrade performance, while ciphertext expansion adds memory demands that complicate deploy- ment on limited-resource systems. Recent advancements like CipherGPT (Hou et al., 2023) have optimized HE for LLMs, yet scalability, performance trade-offs, and complexity remain concerns. Future research should aim to reduce overhead through more efficient protocols, improve non-linear approximations, and explore hybrid methods that combine HE with other privacy techniques.
In conclusion, our findings reveal that while privacy-preservice techniques for ML are advancing, efficiency remains a central focus, as cxisting methods often involve tradc-offs in training speed and model performance. Recent innovations have brought privacy-preeeringg approaches closer to practical deployment facing models, reducing these tradc-offs and making DP increasingly accessible. We are confident that this trend will persist, paving the way for the widespread use of privacy-preserving ML as a practical alternative in the near future.
public-
8 Conclusion
In this work, we conducted an extensive survey on preserving privacy in Large Language Models (LLMs), meticulously analyzing the various dimensions of privacy concerns and the corresponding solutions.
Firstly, our investigation began by exploring the different types of privacy attacks ($3) that these models are susceptible to, including Training Data Extraction (3..),, Membership Inference ($3.2) and Model Inversion ($3.3). Each type exposcs significant vulnerabilities. Training Data Extraction attacks compromisc privacy by enabling adversaries to reconstruct sensitive training data such as personal identifiers or confidential communications, which erodes trust in LLM applications. Membership Inference attacks reveal whether specific data was used in training, potentially disclosing sensitive personal details like medical conditions, which could lead to discrimination. Model Inversion further extends these risks by allowing attackers to deduce personal characteristics from model outputs, promoting unauthorized profiling and significant privacy breaches.
Secondly, we presented an extensive view of the state-of-the-art solutions for preserving privacy in large language models 5 These solutions include approaches that inject privacy directly into the training
51

---

Published in Transactions on Machine Learning Research (01/2025)
data, like anonymization techniques ($4) or approaches that act the model ($5) While there are many possible approaches to the problem, no single solution fits all scenarios. We presented solutions for applying privacy mechanisms at several points in the learning pipeline: from data preparation (before training $4
to private training (during training $5.1. : to private inference (5..2..5 and finally after training ($5.4)
We refer the reader section for a thorough discussion on current limitations and future directions for preserving privacy in large language models.
This work highlights the critical need to address privacy vulnerabilities in Large Language Models (LLMs) by systematically analyzing various privacy attacks and their profound risks to data confidentiality and user trust. We presented current statc-of-tho-art solutions such as data anonymization, Differential Privacy, and Machine Unlearning, emphasizing the importance of continuously advancing and integrating these tech- niques. As LLMs become integral to sensitive domains like healthcare and finance, developing robust privacy safeguards essential. Future research must enhance these methods, improve the privacy-utility trade-off, and ensure that larger models adhere stringent privacy standards, ultimately fostering the creation of secure and reliable LLM technologies that uphold user privacy and trust.
Acknowledgements
The authors gratefully ackuowledge the support of the EU-funded project DataTTolls4Heart (grant No. 101057849), which provided crucial funding and resources for this research. This work has also been carried out in the scope of Project ECS 0000024 Rome Technopole CUP B83C22002820066, NRP Mission 4 Component 2 Investment 1.5, funded by the Europcan Union NextGeeerationEEU
We also exlend our sincere thanks to the reviewers and the action editor of TMLR for their invaluable feedback and constructive comments. Their insights have significantly enhanced the quality and clarity of this paper.
52

---

Published in Transactions on Machine Learning Research (01/2025)
References
Martin Abadi, Andy Chu, lan Goodfellow, H. Brendan McMahan, 1yya. Mironov, Kunal Talwar, and Li Zhang, Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference Computer and Communications Security, CCS*16. ACM, October 2016. doi: 09.1445/297674922788888 URL http: //dx.doi org/10..00. 1145/2976749, 2978318
Ahmad Al Badawi, Chao Jin, Jie Lin, Chan Fook Mun, Sim Jun Jie, Benjamin Hong Meng Tan, Xiao Nan, Khin Mi Mi Aung, and Vijay Ramaseshan Chandrasekhar. Towards the alexnet moment for homomorphic encryption: Henn, the first homomorphic enn encrypted data with gpus. IEEE Transactions Emerying Topics in Compuling, 9(3):1330-1343, 2021. doi:
Rami AI-Rfou, Marc Pickett, Javier Snaider, Yun hsuan Sung, Brian Strope, and Ray Kurzweil. Conversa- tional contextual cues: The case of personalization and history for response ranking, 2016.
Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially private learn- ing with adaptive clipping. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan (eds.), Adoances in Neural Information Processing Systems, volume 34, pp. 17455-17466 17466. Curran Associates, Inc., 2021. URL Paper.
neursippp cc///appr._.lllsssppee/////// 90CCf00af66602227777777548888888 Paper..--fapppp.p.pp.ppp .pff
Avi Arampatzis, George Drosatos, and Pavlos S. Efraimidis. Versatile query scrambling for private web search. Inf. Retr., 18(4):331 -58, aug 2015. ISSN 1386-4564. doi: 10.1007/4107991-015-99564 URL
/rtp::///oi.rgg/// 1007 1000///1791-015-925555
Stefan Arnold, Dilara Yesilbas, and Sven Weinzierl. Guiding text-to-text privatization by syntax. In Anaclia Ovalle, Kai-Wei Chang, Ninarch Mchrabi, Yada Pruksachatkun, Aram Galystan, Jwala Dhamala, Apurv Verma, Trista Cao, Anoop Kumar, and Rahul Gupta (eds.), Proceedings of the 3rd Work- shop on Trustworthy Natural Language Processing (TrustNLP 2023). pp. 151-162, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1 /1216553/112023.rrstttl/ 1.14. URL https://aalaanthology 2023/ trustnlp-1.14/
Mislav Balunovic, Dimitar Dimitrov, Nikola Jovanović, and Martin Vechev. Lamp: Extracting text from gradients with language model priors. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 7641 7654. Curran Associates, Inc., 2022. URL https: htttps://proceediee http://ooccee...eeee/// heeurips.cc/ppper...lle///pppppp ort
Samah Baraheem and Zhongmei Yao. A survey on differential privacy with machine learning and future outlook. arXie preprint arXiv:2211.10708 2022.
Priya Basu, Tiasa Singha Roy, Rakshit Naidu, Zumrut Muftuoglu, Sahib Singh, and Fatemehsadat Mireshghallah. Benchmarking differential privacy and federated learning for bert models. ArXiv, abs/2106.1397m 2021. URL https:://ppi.semanticcc http:/aar.rggg/rosst:::333000 235658799
Björn Bebensee. Local differential privacy: tutorial, 2019.
Rouzbeh Behnia, Mohammadreza Reza Ebrahimi, Jason Pacheco, and Balaji Padmanabhan. Ew-tune: A framework for privately fine-tuning large language models with differential privacy. In 2022 IEEE International Conference on Data Mining Workshops (ICDMW). IEEE, November 2022. doi: 10.1109/ 2tlmw558026.2022.00000 000078. URI. http::/tppp...//////////////tttt awoi.org//1) I1M09/ICDMW5802.. 2022. 00078
Daniel J. Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusmão, and Nicholas D. Lane. Flower: A friendly federated learning research framework, 2022.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, New York, 2006. ISBN 978-0387310732.
53

---

Published in Transactions on Machine Learning Research (01/2025)
G. R. Blakley. Safeguarding cryptographic keys. 1979 International Workshop on Managing Requirements Knowledge (MARK), pp. 313-318, 1899. URL
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint orXim:2108.0720 2021.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, machine
Daniel Ramage, Aaron Segal, and Karn Soth. Practical secure aggregation for learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Secu- rity, CCS '17, pp. 1175-1191, New York, NY, USA, 2017. Association for Computing Machinery. ISBN URL https://doi.orgg wtt../////11..113133333 114533339666 3133982
9781450349468. doi: 10.1145/313395631333999
Amine Boulemtafes, Abdelouahid Derhab, and Yacine Challal. A review of privacy- techniques for deep learning. Neurocomputing. 384:21-45, 2020. URL http:p///ppp...eeettt..ccoooooop
2111159333
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159, 2019. URL https://api...mmmniiccchote htapcchhhaaarooogoouussssssoeeee /rtp.rrrr//rrru11022299999
Gilles Brassard, Claude Crépeau, and Jean-Mare Robert. All-or-nothing disclosure of secrets. In Advances in Cryptology CRYPTO '86, Santa Barbara, California, USA, 1986, Proceedings, volume 263 of Lecture Notes in Computer Science, pp. 234-238. Springer, 1986. doi: 10. 10007773-50044721--111 17.
Alissa Brauneck, Louisa Schmalhorst, Mohammad Mahdi Kazemi Majdabadi, Mohammad Bakhtiari, Uwe Völker, Jan Baumbach, Linda Baumbach, and Gabriele Buchholtz. Federated machine learning, privacy- enhancing technologics, and data protection laws in modical rescarch: Scoping review. Journal of Medical Internet Research, 25, 2023. URL https: httpps///apl.semmnnicccoooo... http://mnnnnnniioooooooooooororp 257834709
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does it mean for a language model to preserve privacy? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '22, pp. 2280 2292, New York, NY, USA, 2022a. Association for Computing Machinery. ISBN 9781450893522. doi: 01444/55531146.33335 URL https:
//doi.org/10. 1115/35311/444 3534642.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does it mean for a language model to preserve privacy? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 2280-2292, 2022b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877 1901. Cur- ran Associates, Inc., 2020. URL Paper .pdf
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Automatic clipping: Differentially private deep learning made easier and stronger. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (ods.), Advances in Neural Information Processing Systems, volume 36, pp. 41747 Curran Associates, Inc., 2023a. URL http:: https:://rroceedings www.ssnnuuiiisieeeeeeeee http://///ccapeeeeeeee/////////p 14446/27777001111ee///aaccceeeee
Zhiqi Bu, Yu- Yu-Xiang Wang, Sheng Zha, and George Karypis. Differentially private optimization on large model at small cost. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato,
54

---

Published in Transactions on Machine Learning Research (01/2025)
and Jonathan Scarlett (cds.), Proceedings of the 40th International Conference Machine Learning, PMT.R, 23-29 Jul 2023b. URT,
volume 202 of Proceedings of Machine Leurning Research, pp.
https://proceeddings www.ggssmlr.... prt.serre///2222u22aaaa http:///s..mmm..e///////////22/0 html.
Zhiqi Bu, Ruixuan Liu, Yu-Xiang Wang, Sheng Zha, and George Karypis. On the efficacy of group-wise clipping in differentially private optimization, 2024. URL httpss//ppperrevieeeenttfrroooo
gmH87p
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. 2015 IEEE Sym-
posium on Security and Privacy, pp. 463 480, 2015. URL httpss//aap..ssmanticcccoo..ol www.schhoo.a...rrrrrorrrlsll
5945696
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: evaluating and testing unintended memorization in neural networks. In Proceedings of the 28th USENIX Conference on Security Symposimm, SEC'19, pp. 267-284, USA, 2019. USENIX Association. ISBN 9781939133009.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Securily Symposium (USENIX Securily 21), pp. 2633 2650, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference atlacks from first principles. In 2022 IEEE Symposium on Securily and Privacy (SP), pp- 1897-1914. JEEE, 2022.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, Quantifying memorization across neural language models. In The Eleventh International Conference on Learnöng Representations, 2023a. URL http://ppweeeveeeeeeeeerrrrrraaa
Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick, and Florian Tramèr. Stealing part of a production language model. In Forty-firsl Internalional Conference on Machine Learning, 2024. URL https:// wttp://www../e.ooommmd/d//te....
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, 2023b.
Aldo Carranza, Rezsa Farahani, Natalia Ponomareva, Alexey Kurakin, Matthew Jagielski, and Milad Nasr. Synthetic query generation for preeeccirr deep retrieval systems using differentially pri- vate language models. Tn Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies (Volume 1: Long Papers), pp. 3920-3930 Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/vi 1005553/12/// 220... https://aclaatholloo orgpe/24.0/222222///
URL
httt::://oc.mthoooley..o./2222. Longe 217/
Ricardo Silva Carvalho, Theodore Vasiloudis, Oluwascyi Feyisetan, and Ke Wang. TEM: High Ubility Metric Differential Privacy on Test, pp. 883-890. 2021. doi: 10.1137/1.7711611976663..8 URL httpss//epbbs.. siam. wttp://oo/bbb//.11117//111111111 01137771.78811119775333 ch99.
Andre Chailloux, Gus Gutoski, and Jamie Sikora. Optimal bounds for semi-honest quantum oblivious transfer. Chic. J. Theor. Compul. Sci. 2016, 2013. URL http:://pp...eemtttecccoeoororrs 2367049
Tyler A. Chang and Benjamin K. Bergen. Language model behavior: A comprehensive survey. Computational Linguistics, 50))) 350, 03 2024. ISSN 0891-2017. doi: 10.1162/coli 00492. CRL https://doi. org/10. 10. 11tp//o.1162/coli___0000000
Konstantinos Chatzikokolakis, Miguel E. Andrés, Nicolás Emilio Bordenabe, and Catuscia Palamidessi. Broadening the scope of differential privacy using metrics. In International Symposium on Privacy En-
hancing Technologies, 2013. URL htct::://ap..sennnticcch...... www.sticsccooll.aa... org/CorpusII: : 14750765
55

---

Published in Transactions on Machine Learning Research (01/2025)
Dingfan Chen, Raouf Kerkouche, and Mario Fritz. A unified view of differentially private deep gener-
alive modeling. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856. URL https: ///pperreveewnne///orrrf//tteeee wtw.:///foorrrriiecymmneoooooo Survey Certification.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for LLMs. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL htttps://openrewee net/
forum?id-GprveTwOok
Sai Chen, Fengran Mo, Yanhao Wang, Cen Chen, Jian-Yur Nie, Chengyu Wang, and Jamie Cui. A customized lext. sanilization mechanism with differnntial privacy. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 2023a. doi: 10.18653/vv /ttp///2.....nnnneeeeee
URL http://dx www.:ooo..ooo.oooooo org/1. 18653/v1/2023, wtt.:////22......nngg...... findings-acl. acl.
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. THE-X: seracc--reserving transformer inference with homomorphic encryption. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Lin- guistics: ACL 2022, pp. 3510-3520 Dublin, Ireland, May 2022a. Association for Computational Linguis- tics. doi: 10.18653 v1 22222222.2..m findings-acl.277 URL https::/aclaatthollgg/ wwwp:/ggg////222/ inndings-acl22111
Wei-Ning Chen, Ayfer Ozgur, and Peter Kairouz. The poisson binomial mechanism for unbiased federated learning with secure aggregation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba. Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 3490-3506. PMLR, 17-23 Jul 2022b. URL httpss:/procceedingg mlr. wttpppress/122che2222222 http://edddnssm..rrrrssiii/iei/d html
Yiyi Chen, Heather Lent, and Johannes Bjerva. Text embedding inversion security for multilingual language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 7808 7827, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10. /1t////1122222a/l//// URL https://accaaata httph/olacllatoooggoooooo org. 2224.2222...0 acl- 422
Yu Chen, Tingxin Li, Huiming Liu, and Yang Yu. Hide and soek (has): A lightweight framework for prompt privacy protection. Arxie, abs/2309.03055 2023b. URL httpss//ppp.ssmanntccccoooorrrrr www.mantticccoolaa.rrr/ooooooooo 261557670
Kai Yuen Cheong, Takeshi Koshiba, and Shohei Nishiyama. Strengthening the security of distributed oblivious transfer. In Austrolasian Conference Information Security and Privacy, 2009. URL
http::///pp..smaaiiiisssc wwwpaaatieecctooooooooooo//ooooo orgg/Crrpus1::118888.
Junjie Chu, Zeyang Sha, Michael Backes, and Yang Zhang. Reconstruct your previous conversations! com- prehensively investigating privacy leakage risks in conversations with GPT models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen ), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1584466000 Miami, Florida, USA, November 2024. Association for Com- putational Linguistics. doi:
URL https://aclanttooooy..rr.//
emnlp- main. 377/
Colin B. Clement, Matthew Bierbaum, Kevin P. O'Keeffe, and Alexander A. Alemi On the use of arxiv as a dataset, 2019.
Cameron Coles.
11% of data employees paste
into
ChatGPT is confidential https://www.cyberhavenr com/ cowp://g/cc/////////o//////
Cyberhaven
wwberhaven.com, 2023.
URL
Soham De, Leonard Berrada, Jamic Hayes, Samuel L. Smith, and Borja Ballo. Unlocking high-accuracy
differentially private image classification through scale. ArXiv, abs/2204.13636 2022. URL https://apt
seematttsccbbaarrrr///opppp2222n org8/Corpps1D::244477777
semanticscholar...
56

---

Published in Transactions on Machine Learning Research (01/2025)
Lea Demelius, Roman Kern, and Andreas Trügler. Recent advances of differential privacy in centralized deep learning: A systematic survey. ArXiu, abs/2309. 2023. URL https://api semanticscholar.
FTg/CorpuuSD::233345544
Jieren Deng, Yijue Wang, Ji Li, Chenghong Wang, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding. TAG: Gradient attack on transformer- measformer-baeed language models. In Mlarie-Francins Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (ods.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3600-3610, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1 //20//1///2/////// 20000...nniigg-eennp..... URL https: //aclanthology.org///000 findings-emnlp 305
Erik Derner, Kristina Batistic, Jan Zah' alka, and Robert Babuka. security risk taxonomy for large language models. ArXiv, abs/ 0bss/231..14115, 2023. URL https: http://pp..smmnnicsccooo.o.r wwwpchhoolrrrrr///Coeooo orggCorppsID:2e50 266009991 265294893
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapler of the Association for Compulational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Min- neapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/NT 1423. URL https://aclanntoooog.ocooo 101.///999..///111122333
Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. Collecting telemetry data privately. In Neural Information Processing Systems, 2017. URL. https://api. wwwanticscholattetteteteetoe http:::/ppp..ennntccccoooo...ro htt.aarorrrgggoopooa::22000
Youlong Ding, Xucyang Wu, Yining meng, Yonggang Luo, Hao Wang, and Weike Pan. Delving into dif-
ferentially private transformer. In Forty-first International Conference on Machine Learning, 2024. URL
https:://ppenreview. httpss//pperreeevv.e/////offffff
My-Hoa Nathalie Dinh and Ferdinando Fioretto. Context-aware differential privacy for language modeling, Arxis, abs/2301. 2bs/2301.12288, 2023. URL https: semanticscholas http:///.......aooooooooooooooo/ 256330
Tim Dockhorn, Tiansii Cao, Arash Vahdat, and Karsten Kreis. Differentially private diffusion models. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL httpps://ppenreview.. net/
forum?id=ZPpok7FFJXK
C. M. Downey, Wei Dai, Huseyin A. Inan, Kim Laine, Saurabh Naik, and Tomasz Religa. Planting and mitigating memorized content in predictivo-text language models. CoRR, abs/2212.086199, 2022. URL https://doi.org// http://rrrrro....5//////////
2212.08619)
Minxin Du, Xiang Yue, Sherman S. M. Chow, and Huan Sun. Sanitizing sentence embeddings (and labels) for local differential privacy. In Proceedings of the ACM Web Conference 2023, WWW '23, pp. 2349 2359, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 9781450394161. doi: 10.1145/ 3543507.35888 3583512. URL https://dobl /ttps/////o..r///.....o 1145/3543507 35835122
Minxin Du, Xiang Yue, Sherman S. M. Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. Dp-forward: Finc-tuning and inforence on language models with differential privacy in forward pass. In Proceedings of the 2023 A ACM SIGSAC Conference on Computer and Communications Securily, CCS '23. ACM, Novemn ber 2023b. doi: 10. 10.1145/3576915.3666
URL http://dxx.t.o//// http///......og/.....1111. http:///........oooo.../////////
Haonan Duan, Adam Dziedzic, Mohammad Yaghini, Nicolas Papernot, and Franziska Boenisch. On the privacy risk of in-context learning. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
Haonan Duan, Adam Dziedzie, Nicolas Papernot, and Franziska Bocnisch. Flocks of stochastic parrots: Dif- ferentially private prompt learning for large language models, Advances in Neural Information Processing Systems, 36, 2024a.
57

---

Published in Transactions on Machine Learning Research (01/2025)
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models?, 2024b. URL https::/arrxiv.org/bbss//// 1
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178 2024.
Christophe Dupuy, Radhika Arava, Rahul Gupta, and Anna Rumshisky. An efficient dp-sgd mechanism for large scale nlu models. In TCASSP 2022 2022 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4118 4122, 2022. doi: ALL109//CASSP449222222..222 202.297977797
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Compal. Sei., 9(3 4):211 4477, aug 2014. ISSN 1551-305X. doi: 10.
URL https://doi.
org/10. 15110..1561040000000
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptol- ogy (EUROCRYPT 2006), volume 4004 of Lecture Notes in Compuler Science, pp. 486 503. Springer Verlag, May 2006a. URL httpp:: //www. www.mooftttooooommmmmmmooooooo httpr/owwt..oo//ee///////eee/r/a
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in pri- vate data analysis. In Shai Halevi and Tal Rabin (eds.), Theory of Cryplography, pp. 265 284, Berlin, Heidelberg, 2006b. Springer Berlin Heidelberg. ISBN 978-3-540-32732-5.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning for LLMs, 2024. URL
Adel Elmahdy, Huseyin A. Inan, and Robert Sim. Privacy leakage in text classification a data extraction approach. In Oluwasoyi Feyisctan, Sepidch Ghanavati, Patricia Thaine, Ivan Habernal, and Fatemchsadat Mireshghallah (eds.), Proceedings of the Fourth Workshop on Privacy in Natural Language Processing, pp. 13 13-20, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022. 1022.privatenlp-1..3 URL
www.///gg/2.... pr 3
David Enthoven and Zaid Al-Ars. An Overview of Federated Deep Learning Privacy Attacks and Defensive Stralegies, pp. 173 196. Springer International Publishing, Cham, 2021. ISBN 978-3-330-70604-3. 10.1007/978-3-000 70600-3 8. URL http::://di..orggg/
doi:
11600998---00007700000 70604-388
Ulfrr Erlingsson, Aleksandra Korolova, and Vasyl Pihur. Rappor: Randomized aggregatable privacy- preserving ordinal response. Proceedings of the 2014 ACM SIGSAC Conference on Computer and Com- munications Securily, 2014. URL
Shimon Even, Oded Goldreich, and Abraham Lempel. A randomized protocol for signing contracts. Commun. ACM, 28(6) 637 647, June 1985. ISSN 0001-0782. doi: 10. 10.1145/3812.381) 1145/3812.3818
URL https://doi..rr//1)
Uriel Feige, Amos Fiat, and Adi Shamir. Zero-kmowledge proofs of identity. Journal of Cryptology, 1:77-94 94, 1988. URL htttps://pppi sesmmattissccolllarrrrrr/rrrrrpo
Vitaly Feldman. Does learning require memorization? short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium Theory of Computing, STOC 2020, pp. 954 959, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 45034999997 doi: 10.1145/3357713. 3384290. URL https::/doi.
1145/3357713, 3384290
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: discovering the long tail via influence estimation. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 0781713829544
58

---

Published in Transactions on Machine Learning Research (01/2025)
Shanglun Feng and Florian Tramèr. Privacy backdoors: stealing data with corrupted pretrained models. In Proceedings of the 41st International Conference on Machine Learning, ICMI'24. JMI.R.rrg 2025.
Oluwaseyi Feyisetan, Borja Balle, Thomas Drake, and Tom Diethe. Privacy- and analysis calibrated multivariate perturbations. Proceedings of the 13th International Conference on Web Search and Data Mining, 2019a. URL https://apl.
textual
Oluwascyi Feyisetan, Tom Diethe, and Thomas Drake. Leveraging hicrarchical representations for preserving privacy and utility in text. 2019 IEEE International Conference on Data Mining (ICDM), pp. 210-219, 2019b. URL https://api. wwwantiiscooooteeeeee http://oo...ooerrrorrsss::::::/a
Matthew Finlayson, Xiang Ren, and Swabha Swayamdipta. Logits of API-protected LLMs leak proprietary information. In First Conference Language Modeling, 2024. URL httpss///opnrreeeeennee//roomoio
ORCYFm8vyB
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1322-1333, 2015.
Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenntiess An end-to-end case study of personalized warfarin dosing. In Proceedings of the 23rd USENIX Conference on Security Symposium, SEC"14, 14, pp. 17-32, USA, 2014. USENIX Association. ISBN 9781931971157.
Arun Ganesh, Mahdi Haghifam, Milad Nasr, Sewoong Oh, Thomas Steinke, Om Thakkar, Abhradeep Guha Thakurta, and Lun Wang. Why is public pretraining necessary for private model training? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabalo, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of
Proceedings of Machine Learning Research, pp. 10611 10627. PMLR, 23 29 Jul 2023. URL https: //proceedings. mlr, wtt...prees//v22/gaees2222
html
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Gonzalo Munilla Garrido, Xiaoyuan Liu, Floria Matthes, and Dawn Song. Lessons learned: Surveying the practicality of differential privacy in the industry. Proceedings on Privacy Enhancing Technologies, 2023.
Jonas Geiping, Hartmut Bauermeister, Hamnah Dröge, and Michael Moeller. Inverting gradients- how easy is it to break privacy in lederated learning? In Proceedings of the 34th International Conference on Newral Information Processing Systems, NIPS '20 Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
General Data Protoction Regulation. Regulation (cu) 2016/679 of the european parliament and of the council, 4 2016. URL https://eur-lex euroee. http::/wwww.p.........o...tttcta 6R0679 Article 4(1): Definition of 'personal data'.
Craig Gentry. Fully homomorphic encrrpoorrrppieieeenneoioiiioooo using ideal lattices. In Symposium on the Theory of Computing, 2009. URL
http://pp...eannttt.cccooooorrrs 947660
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021.
Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Ira Ktena, Robert Stanforth, Jamie Hayes, Soham De, Samuel L. Smith, Olivia Wiles, and Borja Balle. Differentially private diffusion models generate use- ful synthetic images. Arxio, abs/2302. 138611, 2023. URL hhttps://api: http:////api...mmmtttccooo.ooooe www.teiccchooooaroorrooooooooo 257219853
59

---

Published in Transactions on Machine Learning Research (01/2025)
Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Nachrig, and John Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In Maria Flo- rina Balcan and Kilian Q Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 201-210, New York, New York, USA, 20 22 Jun 2016. PMLR. URL https:///rooceedings mwr. www..ppess//48//// hatp/////glllddddaaaee baw.adddabbhhh.....m htmll
Jennifer Gillenwater, Matthew Joseph, Andres Munoz, and Monica Ribero Diaz. A joint exponential mecha- nism for differentially private top-k. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learn- ing, volume 162 of Proceedings of Mochine Learning Research, pp. 7570-7582. PMLR, 17-23 Jul 2022. URL httpp:://proceedinngg mlr. wwt.e/preesssveeeeeeeee 2tte/ggileenatte222.haae
Aaron Gokaslan and Vanya Cohen.
Openwebtext
corpus.
http://kyyiion07.githe io/
OpenWebTextCorpus 2019.
Oded Goldreich and Yair Oren. Definitions and properties of zero-knowledge proof systems. Journal of Cryptology, 7:1-32, 1994. URL https://api.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, Cambridge, MA, 2016. ISBN 978-0262035613.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafford, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradcep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), pp. 15789-15809, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/ v1 v1/2024.acl-long.551 URL http:s:://cllanthoooo..orr.. 2204 /w0..///////2////// acl- long. 841/
Alejandro Cuerra-Manzanares, L. Julian Lochuga Lopez, Michail Maniatakos, and Farah E. Shamout. Presevy-rrservvig Machine Learning for Healtheare: Open Challenges and Future Perspectives, pp. 25 40. Springer Nature Switzerland, 2023. ISBN 9783031395390.
doi: 10.1007 1111070777-3-11 397779993300033933309 000503990333 3. URL
http://dkx www.x.doi.ooo oroi.org/10...
Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recovering private text in fodcrated Icarning of language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 8130- 8143. Curran Associates, Inc., 2022. URL https: pttppe://rroceedingg. www.nee....eeeeeeooooeee pourtpps.c//ppprr..llll///pppppp 75106667777711111222222227777100
Ivan Habernal. When differential privacy meets nlp: The devil is in the detail. In Conference on Empiri- cal Methods in Natural Language Processing, 2021. URL
237431523
Muhammad Usman Hadi, al tashi, Rizwan Qureshi, Abbas Shah Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Navced Akhtar, Jia Wu, Scyedali Mirjalili, Qasem Al-Tashi, Amgad Muncer, Mo- hammed Ali Al-garadi, Gru Cnn, and T5 RoBERTa. Large language models: A comprehensive survey of its applications, challenges, limitations, and future prospects. 2023. URL https://app semanticscholar: org//CorpuSDD:266378846
Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang. Tron: Pri- vate inference on transformers. In Neural Information Processing Systems, 2022. URL https://api.
60

---

Published in Transactions on Machine Learning Research (01/2025)
Andrew Hard, Kanishka Rao, Rajiv Mathews, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. ArXiv, 0021181110060 2018. URL https://api semanticscholar. org//orrssS11::522777 http////saaaaattoooooooooooooosa
Moritz Hardt and Guy N. Rothblum. A multiplicative weights mechanism for privacy-preserving data analysis. In 2010 IEEE 51st Annual Symposium on Foundalions of Computer Science, pp. 61 70, 2010. doi: 10.1 N0.1109/FOCS.20100
Jahid Hasan. Security and privacy issues of federated learning. ArXiv, abs/2307.1218, 2023. URL https:
//api. seaanticaant htt.ccchh......rrrrrruppso:DDD:. wwwp/oorpsssi:/20.... 2601 25925
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. neur Curran Associates, Inc., 2015. URL httts:://proceedings..eee neurips wtteurips.ccpppperree hwt.prrrrpp.....ppppppei///ppppp filees/papee///11///eel.
afdec7005cc9f 011077005cc91443222004774444 14802204744F00033355 Paper. pdd.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In T. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: htttss://ptteeeee /ptps////cocc.css...... pdf
Sorami Hisamoto, Matt Post, and Kevin Duh. Membership inference attacks on sequence-to-seqqeecc models: Is my data in your machine translation system? Transactions of the Association for Computational Linguistics, 8:49-66 63, 2020. doi: 10.1162/tacl a_ 00299. URL (BLLLttt:://aaccantoloooorr///// ta
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory, Neural Computationn (8): 1780, 11 1997. ISSN 0899-7667. doi: 10. wttp:///cc..19.........
URL httts:///ol..orr//1..116//eeee 1997.
9.8. 8.8. 1735
Naoise Holohan, Stefano Braghin, Pól Mac Aonghusa, and Killian Levacher. Diffprivlib: The ibm differen- tial privacy library. ArXiv, 3bs/1907.02444, 2019. URL. https://ppp.semmnticcco...... 195798910
Junyuan Hong, Jiachen T. Wang, Chenhui Zhang, Zhangheng LI, Li, and Zhangyang Wang. DP- OPT: Make large language model your seeecy-preeeriig prompt engineer. In The Twelfth International Conference Learning Represendations, 2024. URL htttps///ppeereeveee..t/torororo
Shlomo Hoory, Amir Feder, Avichai Tendler, Alon Cohen, Sofia Erell, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayclet Benjamini, Avinatan Hassidim, and Yossi Matias. Learning and evaluating differentially private pre-trained language model. In PRIVATENLP, 2021. URL httpp://app semanticschooar..rrg
CorpusID: 23:009765333
Xiaoyang Hou, Jian Liu, Jingyu Li, Yu hai Li, and Cheng Hong. Ciphergpt: Secure two-party gpt infer- ence. IACR Cryptol. ePrint Arch., 2023:1147, 2023. URL httpss//ppp.semanttcsschoo..... 260278194
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (cds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1790-2799 PMLR, 09 15 Jun 2019.
URL https::/proceeding/ mlm. wttpppprssss/////eeee http://b//hhoulbyyybeb/ ba..yyl99........ html
Dirk Hovy, Anders Johannson, and Anders Søgaard. User review sites as resource for largo-scale soci- olinguistic studies. Proceedings of the 24th International Conference on World Wide Web, 2015. URL
https://api. stt.appiisemmaticschooo..a http://iiiiiiaoo.ooooo/o//aaaaaa
61

---

Published in Transactions on Machine Learning Research (01/2025)
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL hnttps://ppenreview.. http://wwwww...ooooooo////////ee
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Asso- ciation for Computational Linguisties: EMNLP 2022, pp. 2038 2047,, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/vl /22.222.i2n2222...222 URL httpss///cccaanttoooooooor2/2222 findings-ennlp...... 148.
Timour Igamberdiev and Ivan Habernal. DP-BART for privatized text rewriting under local differential privacy. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13914 13934, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 2..1863//v1/222/ http:///00....nnnaaa.a...... URL httpss//aclanthology.ooo
1223.findings-ao www.ac....7 874/
Timour Igamberdiev, Doan Nam Long Vu, Felix Kuennecke, Zhuo Yu, Jannik Holmer, and Ivan Habernal. DP-NMT: Scalable differentially private machine translation. In Nikolaos Aletras and Orphee De Clereq (eds.). Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp. 94-105, St. Julians, Malta, March 2024. Association for Compu- tational Linguistics. URL hntess://cclanthologyy wtwp:/oyy...rr//2...ooo eacl-deo....l
Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. CoRR, abs/2309 11852, 2023. doi: (0.46550/ARXXI..200000
URL https://dol.org//1/ 48550/ arxiv. 2309.
11852
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Sco. Knowledge unlearning for mitigating privacy risks in language models. In Annual Meeting of the Association for Computational Linguistics, 2022. URL https///ap..mmmmticcccoooooooo/ wrrp:/orrrrrro/rouutooo.oo
252693065
Marija Jegorova, Chaitanya Kaul, Charlie Mayor, Alison Q. O'Neil, Alexander Weir, Roderick Murray- Smith, and Sotirios A. Tsaftaris. Survey: Leakage and privacy inference time. IEEE Transactions on Paltern Analysis and Machine Intelligence, 45:9090 9108, 2021. URL https://app. semanticscoolar.
(B6/CorppsID::335317711
Alistair E.W. Johnson, Tom Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. Mimic-ili, a freely accessible critical care database. Scientifiic Data, 3(1):160035, May 2016. ISSN 2052-4463. doi: 10.1038/sdata.200 2016.
35. URL https://doi..rr//10 1033///a.110///attta 2016. 35
Noah M. Johnson, Joseph P. Near, Joseph M. Hellerstein, and Dawn Xiaodong Song. Chorus: programming framework for building scalable differential privacy mechanisms. 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 535 551, 2018. URL https://api. www.seennniicccooooooooo/rrrrrr 226266222
Chihong Joo and Aaram Yun. Homomorphic authenticated encryption secure against chosen-ciphertext attack. IACR Cryptol. ePrint Arch., 2013:726, 2014. URL 17850861
Peter Kairous, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy, In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1376-1385, Lille, France, 07-09 Jul 2015. PMLR. URL
html
Peter Kairouz, Brendan Memahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Prac- tical and private (deep) learning without sampling or shuffling. In Marina Meila and Tong Zhang (eds.),
62

---

Published in Transactions on Machine Learning Research (01/2025)
Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Ma- PMLR, 18-24 Jul 2021a. Ut//htpp:://proceeiiign... wtwpiiig.........s.s...
chine Learning Research, pp.
v v139/kairouz211. html
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hu- bert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Koneený, Aleksandra Korolova, Farinaz Koushanfar, Samni Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mit- tal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Scbastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Fe- lix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Foundations and Trendso in Machine Learning, 14(1 2):1 210, 2021b. ISSN 1935-8237. doi: 10.1561/220000000 URL /ttp://dx..o dow.edoi.ooo org/ 10. 15111/1112222 /200//0000000083
Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 7403 7412,, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ 10/5533/1/2223.mm/m// 2wtp::3emmmmmmmii-5555555 main.458. URL http:ttt////catttollloee org/
2023. www.mp-manmmnn...mmm 458
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hy- percomplex adapter layers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 1022 1035, Curran Associates, Inc., 2021. URL https::/ppooeedingss.eeus neurips. httpr//ann..aerrroe..//////////a 0811ee9faFF077703cc8833356666671 parpaaae3...e00eeoooaaarrrrrerde Paper.
Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam D. Smith. What can we learn privately? 2008 49th Annual JEEE Symposium on Foundations of Computer Science, pp. 531 540, 2008. URL https://api, sennanniicccolla..oooo/oooooroni
Aly M. Kassem, Omar Mahmoud, and Sherif Saad. Preserving privacy through dememorization: An unlearn- ing technique for miligating memorization risks in language models. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https: //api. www.ssssscnnnooooaao..e.ooooo
httpps:/aap..smmmtttcchooooo wttp:olaaa..rg/ooossusooo 246164054
Gavin Kerrigan, Dylan Slack, and Jens Tuyls. Differentially private language models benefit from public pre-training, In Oluwaseyi Feyisetan, Sepideh Ghanavati, Shervin Malmasi, and Patricia Thaine (eds.), Proceedings of the Second Workshop on Privacy in NLP, pp. 39-45, Online, November 2020. Association for Computational Linguistics. doi: 10.18653 10.18653/v1/2000 http:/220200rvvaeelpp 1.5. URL https://aclamthhooyy.orrg 2020. 1.2..piiiateellp11.12
Eugene Kim. Amazon warns employees not to share confidential information with ChatGPT after secing cases where its answer 'closcly matches inside the company, 1 2023. URL
existing mate- com/
rial' from
Jeongsu Kim and Aaram Yun. Secure fully homomorphic entcenticaaeeddeccrttiioiio IEEE Access, 9:107279 107297, 2021. URL http::///ppp.ssmmmnnncccccorrrrr wtt.cc/o..a...rr//roppptpi:::::a
Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. Propile: Probing privacy leakage in large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (ods.), Advances in Neural Information Processing Systems, volume 36, pp. 20760-20762. Curran Associates, Inc., 2023. URL hngn: http:///eeeeeeesss......cccce/er 2200:228002221100000000777222221
63

---

Published in Transactions on Machine Learning Research (01/2025)
Bryan Klimt and Yiming Yang. The enron corpus: A new dataset for email classification rescarch. In Jean- Frangois Boulicaut, Floriana Esposito, Fosca Giannotti, and Dino Pedreschi (eds.), Machine Learning: ECML 2004, pp. 217 226, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg. ISBN 978-3-540-30115-8,
Oleksandra Klymenko, Stephen Meisenbacher, and Florian Matthes. Differential privacy in natural language processing the story so far. In Oluwaseyi Feyisetan, Sepideh Ghanavati, Patricia Thaine, Ivan Habernal, and Fatemehsadat Mireshghallah (eds.), Proceedings of the Fourth Workshop on Privacy in Natural Lan-
guage Processing, pp. 1-11, Seattle, United States, July 2022. Association for Computational Linguistics. doi:
URL
Pang Wei Koh and Percy Liang. Understanding black-box preduictioorciiiiionnnnn via influence functions. In International Conference on Machine Learning, 2017. URL
13199997
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twasawa. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference Newral Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 178778887.088.
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit lyyer. Thieves on sesame street! model extraction of bert-based apis. In International Conference Learning Representa- tions, 2020. URL https://openreeiee/ wttp://wwwwwteooo///e/.ya... DNREFDr
Satyapriya Krishna, Rahul Gupta, and Christophe Dupuy, Adept: Auto-encoder based differentially pri- vate text transformation. In Conference of the European Chapter of the Association for Computational Linguistics, 2021. http:: sewanticscholar
orgg/CorpuSD:23111 stt.aa/iisssolleeoooo///ooorosss 231750016
Alexcy Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, and Andreas Terzis. Harnessing large-language models to generate private synthetic text, 2024. URL http://s///pppeeeveeeeeee///e/ta TOE6N8dp4w
Tyadh Ben Cheikh Larbi, Aljoscha Burchardt, and Roland Roller. Which anonymization technique is best for which nlp task? it depends. a systematic study on clinical text processing, ArXiv, abs/2229.00062, 2022. URL hhttt:://ppp/
Jaewoo Lee and Daniel Kifer. Scaling up differentially private deep learning with fast per-example gradient clipping. Proceedings on Privacy Enhancing Technologies, 2021.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Buree and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 8424 8445, Dublin, Ire- acl land, May 2022. Association for Computational Linguisties. doi: 10.18653/v1 2/22://22222222oo....3.777 URL
www.://..gg/.2.....oeo acl-1on..l1 5777
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficeet prompt tuning, Tn Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tan Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ (0.18633/v1/211 http/2221emmmlyy main.243 URL httpps:/aallantholoyy.orrr htt.://pwwooorre...oommmmm/amaaa
emnlp-main. 243/
Dacheng Li, Hongyi Wang, Rulin Shao, Han Guo, Eric Xing, and Hao Zhang. MPCFORMER. FAST, PER- FORMANT AND PRIVATE TRANSFORMER INFERENCE WITH MPC. In The Eleventh International Conference on Learning Representations, 2023a. URL https::/
Haoran Li, Yulin Chen, Jinglong Luo, Yan Kang, Xiaojin Zhang, Qi Hu, Chunkit Chan, and Yangqiu Song. Privacy in large language models: Attacks, defenses and future directions, 2023b.
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-stop jailbreaking privacy attacks on ChatGPT. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 4138 4153, Singapore, December
64

---

Published in Transactions on Machine Learning Research (01/2025)
2023c. Association for Linguaaatttioallliinntittiiit doi: (0.18653/v1/2023 http://///22...mnssomm... findings-emmll.2222 URL https:
wwwpthooloog..o//222.....3 Findings-emnlp.. 272
Haoran Li, Mingshi Xu, and Yangqiu Song. Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (ods.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 14022 14040, Toronto, Canada, July 2023d. Association for Computational Linguistics. doi: 10.18653/v1 http:///......iinnsaa//aaa ac1.881. URL
org/2023. findings-acl.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference
Natural Language Processing (Volume 1: Long Papers), pp. 1992-4597, Online, August 2021. Association for Computational Tinguistics. doi: 10.18653/vv /1/00/11111222/2 /12021.aclloooog 3533. URL https://aclanttholog// org. wwwp:/ooy..rrrr2...llaa-ooo00 353/
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations, 2022. URL
Yansong Li, Zhixing Tan, and Yang Liu. Privacy-preserree prompt tuning for large language model services. ArXiv, abs/2305.06212,, 2023e. URL https://api..smmaniiccchh swwanticscoollar wotpattttssscooo.oooo/o/rroooooo org//orrpus1::25588111
Nikhil Dandckar Lili Jiang, Meg Risdal.
Quora question pairs, 2017. URL https://kaglle.com
Peng Liu, Xiangru Xu, and Wen Wang. Threats, allacks and defenses to federated learning: issues, laxonomy and perspectives. Cybersecurity, 5, 2022. URL https: oetaaaanniicccoooooooooo/rrrrrro 2444447799
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking machine unlearning for large language models, 2024a.
Xiao Xiao-Yang Liu, Rongyi Zhu, Daochen Zha, Jicchao Gao, Shan Zhong, Matt White, and Mcikang Qiu. Differentially private low-rank adaptation of large language model using federated learning. ACM Trans. Manage. Inf. Syst., August 2024b. ISSN 2158-656X. doi: (0.1145/368200 URL https:://oi.org: org//0000.../000.
1145/3682068 Just Accepted.
Ximeng Liu, Lehui Xie, Yaopeng Wang, Jian Zou, Jinbo Xiong, Zuobin Ying, and Athanasios V. Vasilakos. Privacy and security issues in deep learning: A survey. IEEE Access, 9:4566-4598, 2021. URL https:
//api. saapisssemaniccchol.as htt.s//mmnnnccccco....oooooo/o/p ortp:o///oopps1112212555555
Natasha Lomas. Italy orders ChatGPT blocked citing data protection concerns TechCrunch
|
techerunch.com.
2023.
Nils Lukas, A. Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-B Banella-B'eguelin. Analyzing leakage of personally identifiable information in language models. 2023 IEEE Symposium on Security and Privacy (SP), pp. 346-363, 2023. URL httpss///pp: www.ntscschoaaaaaaa
2564595544
Chuan Ma, Jun Li, Kang Wei, Bo Liu, Ming Ding, Long Yuan, Zhu Han, and Harold Vincent Poor Trusted ai in multiagent systems: An overview of privacy and security for distributed learning. Proceedings of the IEEE, 111:1097-1123 2022. URL
246996118
Peihua Mai, Ran Yan, Zhe Huang, Youjia Yang, and Yan Pang. Split-and-denoise: Protect large language model inference with local differential privacy. In Forty-first International Conference on Machine Learn- ing, 2024. URL
Nuno J. Mamede, J. Baptista, and Francisco Dias. Automated anonymization of text documents. 2016 IEEE Congress on Enolutionary Computation (CEC), pp. 1287-1294, 2016. URL https://apl: semanticscholar.. orrpi/oa.....rr///rppps//0000050
65

---

Published in Transactions on Machine Learning Research (01/2025)
Rahat Masood, Dinusha Vatsalan, Muhammad Ikram, and Mohamed Ali Kaafar. Incognito: A method for obfuscating web data. In Proceedings of the 2018 World Wide Web Conference, WWW '18, pp. 267-276, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steering Com- mittoe. ISEN 9781450356398. doi: 10.1145/3178876.9188008 3186093. 3186093
URL https: https://doi.oorg/100 /ttp://///ob.orr//0111////////9 1145/3178876.
Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, and Mrinmaya Sachan. Differ- entially private language models for secure data sharing. In Conference Empirical Methods in Natural
Language Processing, 2022a. URL http://app..emmnnicccooooaarr http:///...aaaaaa.oooooooooooooa
Justus Mattern, Benjamin Weggenmann, and Florian Kerschbaum. The limits of word level differential privacy. In NAACI-HITT, 2022b. URL htppss///pppp..eaatccccccoaaarrr
248 2485128488
Justus Mattern, Fatemehsadat. Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. Membership inference attacks against language models via neighbourhood com- parison. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 11330-1133 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1 hin23.iiniing-a..711. URL https://aclaattt httpss://cclannhoooy.orrr 2023. findings-acl. a19
R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN Transactions of the Association for Computational Linguistics, 11:652 670, 2023. doi: 10.1162/tacl 00567. URL https:///cclantoology 0rg/2023..acl-1.38
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, vol- ume 54 of Proceedings of Machine Learning Research, pp. 1273-1282. PMLR, 20-22 Apr 2017a. URL https::/procceedings. wel...prees/v5444 prress//54/mmmmaan1/55
Commmication
H. B. McMahan, Eider Moore, Daniel Ramage, and Blaise Agïera y Arcas. Federated learning of deep networks using model averaging. ArXiu, abs/ 11s/11602.05629, 2016. URL http:s//pppp..mmmnnncccooooorrr
CorpusID:16861557
H. B. McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private language models without losing accuracy. ArXiv, abs/17 abs/1710.06963, 2017b. URL https://apts httpss//appi.emmnticcchooo
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference Learning Representations, 2018. URL
Frank D. McSherry. Privacy integrated queries: an extensible platform for preeeccpreeervingg data analysis. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, SIGMOD '09, pp. 19 30, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585512. doi: 10.1145 10.1145/1559944 111111559945.55 1559880. URL https://doi. org/100. 1145/1559845, 1559850
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual as- sociations in gpt. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 17359 17372. Cur- ran Associates, Inc., 2022. URL https://proceedding: wwwpsss.eeiip.ooeomooe fitaaprrffllesspppe//////////// 11144//88277e8906663222222222822 Pwweaaa--aape--aer
pdf
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-oditing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. URL
66

---

Published in Transactions on Machine Learning Research (01/2025)
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learving Representations, 2017. URL http:///pppnnreeeeennetff/rorroe
Byj72udxe
Stephane Meystre, F Friedlin, Brett South, Shuying Shen, and Matthew Samore. Automatic de-identification of textual documents in the electronic health record: A review of recent research. BMC medical research methodology, 10:70, 08 2010. doi: 101186/1471-201 22-8--0-78-
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2024.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg- KirgKKikkaatrick, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. In Yoar Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pp. 8332 8347, Abu Dhabi, United Arab Emirates, De- cember 2022. Association for Computational Linguisties. doi: httas://aaclantholog..orrr org.e/22rrg/22222222 mmmlp-mmiin.55 570
h1/l/211222eeeee /ttp:2222222222222 emnlp-main 570. URL
Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. Can LLMs keep secret? testing privacy implications of language models via contextual integrity theory. In The Twelfth International Conference on Learning Represenlations, 2024. URL https:// openreview. httt/oorummid=gggttBBBB
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Securily Poundations Symposium (CSF). IEEE, August 2017. doi: 10.1109/csf.2017.. URL http::/dx. 1199//. 11
Fatemehsadat Mirshghallah, Mohammadkazem Taram, Praneeth Vepakomma, Abhishek Singh, Ramesh Raskar, and Hadi Esmaeilzadeh. Privacy in deep learning: A survey, Arxiv, abs/20044.12555 2020. URL
httpps//gg....eetttccccoooorrrrr 21155333
Robin Mitchell. Information
Samsung Fab Data
Leak:
How ChatGPT Exposed Sensitive
clectropages.com www.:hattptt-oooeeeeesssttteeeee
2023.
John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander Rush. Text embeddings reveal (al- most.) as much as text. In Houda. Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empérical Methods in Natural Language Processing, pp. 12448 12460, Singapore, December 2023. Association for Computational Linguistics. doi: http:///1//222.emlpmmmm/ 765. URL
https://aclanthologyy ottp:/acaaaatoooooooor//////// 2w0.2233......0.... ee:.:3mmmlppmmmmin main. 765
John Xavier Morris, Wenting Zhao, Justin T Chiu, Vitaly Shmatikov, and Alexander M Rush. Language model inversion. In The Twelfth International Conference on Learning Representations, 2024. URL
Ahmadreza Mosallanezhad, Ghazaleh Beigi, and Huan Liu. Deep reinforcement learning-based text anonymization against privato-attribute inference. In Conference Empirical Methods in Natural Lan- guage Processing, 2019. URL http://ppp...mmanncccchhoo...... htg.hooaar...rggooooopso12222212
Sasi Kumar Murakonda and R. Shokri. MI privacy meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning. ArXiv, abs/2007.09339, 2020. URL http::::/app.semanntisssses
TTt//CorpusDD:228473338
Mohammad Naseri, Jamie Hayes, and Emiliano De Cristofaro. Local and central differential privacy for robustness and privacy in federated learning. Proceedings 2022 Network and Distributed System Security Symposium, 2020. URL https::t/ http:://t....emttttcccoooooorrrp
Milad Nasr, Nicholas Carlini, Jonathan Hayasc, Matthew Jagiclski, Feder Cooper, Daphne Ippolito, Christopher A Choquette-Chor Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint ar Xrxiv:2311. 17155 2023.
67

---

Published in Transactions on Machine Learning Research (01/2025)
National Institute of Standards and Technology. Nist special publication 800-63-3: Digital identity guidelines. Special Publication 800-63-3, National Institute of Standards and Technology, 2017. URL httpt::ttptssstse nist. / Definition of Personally Identifiable Information (PII).
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee- Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning, arXiv preprint arxiv:2209. 02299, 2022.
Helen Nissenbaum. Privacy as contextual integrity. Wash. L. Rev., 79:119, 2004.
Helen Nissenbaum. Privacy in context: Technology, policy, and the integrity of social life. Bibliovault OAI Repository, the University of Chicago Press, 01 2010.
David Odera. Federated learning and differential privacy in clinical health: Extensive survey. World Jour- nal of Advanced Engineering Technology and Sciences, 2023. URL httpss//ppp..smmmnn..cccoooorrrr
20rpusID:258166500
OpenAl. March 20 chatgpt outage: Here's what happened: an update our findings, the ac-
tions we've taken, and technical details of the bug, 3 2023. URL https://oppma..commbbggg//
march-28-chatpp--oute
Team OpenDP. The opendp white paper. 2020. URL https://proojects. http://sw......aaeddddd///////// fills//penddp.wiittepppperr_ 1taaperr_11mmy222_ppppp
Ahmed El Ouadrhiri and Ahmed M Abdelhadi. Differential privacy for deep and federated learning: A survey,
IEEE Access, 10:22359-2288 22380, 2022. URL
httpp/ppp...mmttttcccooooooooor/ wwwp:rr///oooss:::...m (40TPP000ID:28899227 246890627
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe, Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730 27744. Curran Associates, Inc., 2022. URL https::/ wwwesss.euiipseeeeeeeee heuriiss.cc/ppprr_fllee 4ftp://111///edd5555666644949449
Mustafa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. Controlling the extraction of memorized data from large language models via prompt-tuning. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeling of the Associalion for Compulationall Linguislics (Volume 2: Short Papers), pp. 1512 1521, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2022 acl-short.129 URL https::/aclanthho/ http:/aannttoooooooooo/220/ org/ acl-short. 129
Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP), pp. 1314 1331, 2020. doi: 00095.
Rrubaa Panchendrarajan and Suman Bhoi. Dataset reconstruction attack against language models. In CEUR Workshop, 2021.
Qi Pang, Jinhao Zhu, Helen Möllering, Wenting Zheng, and Thomas Schneider. Bolt: Privacy-preservnng, accurale and efficient inference for transformers. 2024 IEEE Symposium on Securily and Privacy (SP), pp. 7753-4771, 2024. URL
266178755
Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. In International Conference on Learning Representations, 2017. URL htttps:/ http:://peeereeeeeeeeeeerornonca wtt.t//ooormm/.dddddoo/Psggg
Nicolas Papernot, Shuang Song, 1lya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with PATE. In 6th International Conference on Learning Representations, ICLR 2018, Vancowver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.nect 2018. URL
XbRZ
68

---

Published in Transactions on Machine Learning Research (01/2025)
Nicolas Papernot, Steve Chien; Shuang Song, Abhradeep Thakurta, and Ulfar Erlingsson. Making the shoe fit: Architectures, initialivations, and tuning for learning with privacy, 2020. URL https://openreview net/forum21drrrgg55rrww
Rahil Parikh, Christophe Dupuy, and Rahul Gupta. Canary extraction in natural language understanding models. In Smaranda Muresan, Preslav Nakóv, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 552 -60, Dublin, Treland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/20000 /2022.cl--hrrt..... URL
Vaidchi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from LLMs? objectives for defending against extraction atlacks. In The Twelfth International Conference on Learning Represen- tations, 2024. URL
Andrew J. Paverd and Andrew C. Martin. Modelling and automatically analysing privacy properties for honest-but-curious adversaries. 2014. URL http:////ppp...ee....o.oooooooo/ http://oooorrrs000011100666
Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners, 2024. URL
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirly-eight Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2024. URL
Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. In NeurIPS ML Safety Workshop, 2022. URL
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using di- vergence frontiers. Tn M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.), Advances in Neural Information Processing Systems, volume 34, pp. 4816-4828. Cur- ran Associates, Inc., 2021. URL httts::/proceedinng neurips.. wwwrips.cccpappr.rr httpriisscccppprri..eee///pppppp filles//paper//221//1ee 260c243288eecc2cce03c 1eaaeeedadddeaaaeeee
oww.a//eeeeeeeeeooooddddddeeaaaa Paper.
Natalia Ponomareva, Hussein Hazimeh, Alexey Kurakin, Zheng Xu, Carson E. Denison, H. B. McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Thakurta. How to dp-fy ml: A practical guide to machine learning with differential privacy. J. Artif. Intell. Res., 77:1113- 1201, 2023. URL https://api.
semanticscholar. wwwpllaarrrggggeoo orgg/CorpsII: 2572554511
Aman Priyanshu, Supriti Vijay, Ayush Kumar, Rakshit Naidu, and Fatemehsadat Mireshghallah. Are chatbots ready for privacy-sensitive applications? an investigation into input regurgitation and prompt- induced sanitization. arXio preprint arXiv:2305.15008 2023.
Dmitry Prokhorenkov and Yang Cao. Towards benchmarking privacy risk for differential privacy: A survey. In Proceedings of the 10th ACM International Conference on Systems for Energy-Etficte Efffciiett Buildings, Cities, and Transportation, BuildSys '23, pp. sociation for Computing Machinory. ISBN 9798400702303..
New York, NY, USA, 2023. As- doi: 10.145/33001003322373 URL
https://dol..rrg/e. 0145///00115/3600000 36001000066000333333333333333333
Chen Qu, Weize Kong, Liu Yang, Mingyang Zhang, Michael Bendersky, and Mare Najork. Natural language understanding with privacy-preserving bert. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM '21, pp. 1488 1497, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384466. doi:
URL https://doi.orr
10. 15.1115/3459637 1145/3459637.3482281
Michael O. Rabin. How to exchange secrets with oblivious transfer. IACR Cryptol. ePrint Arch., 2005:187, 2005. URL http::ttpt//apppe semaattissssollarroooooooossssos
69

---

Published in Transactions on Machine Learning Research (01/2025)
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H. Brendan McMahan, and Françoise Beaufays. Training production language models without memorizing user data, 2020.
Niranjan K. Ray, Deepak Puthal, and Dhruva Ghai. Federated learning. IEEE Consumer Electronics Magazine, 10(6))106 107, 2021. doi: 2..100//MCE.222133094777
Donghwan Rho, Taeseong Kim, Minje Park, Jung Woo Kim, Hyunsik Chae, Jung Hee Cheon, and Ernest K. 02486
Ryu. Encryption-friendly 11m architecture, 2024. URL
Maria Rigaki and Sebastián García. A survey of privacy attacks in machine learning. ACM Computing Surveys, 56:1 34, 2020. URL
Aleksandr Romanov Anna Kurtukova, Anastasia Fedotova, and Roman Meshcheryakov. Natural text anonymization using universal transformer with a self-attention. In Proceedings of the III International Conference on Language Engineering and Applied Linguistics (PRLEAL-2019), Saint Petersburg, Russia, pp. 22-37, 2019.
Pierangela Samarati and Latanya Sweeney. Protecting privacy when disclosing information: k--anonymity and its enforcement through generalization and suppression. 1998. URL. https://api semanticscholaa.
org/Corpus1D:2181334
Timo Schick and Hinrich Schütze. Few-shot text generation with natural language instructions. In Conference on Empirical Melhods in Natural Language Processing, 2020. URL http:///caa.smmmmiiiccchhooooooo
CorpusID:2382600
Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Xiaofeng Zhu, and Qing Li. Exploring the landscape of machine unlearning: A comprehensive survey and taxonomy. IEEE Transactions on Neural Networks and Learning Systems, pp. 1 1-21, 2024. doi: SALLL99/TNNLSS2024......
Adi Shamir. How to share secret. Commun. ACM, 22(11):612 613, nov 1979. ISSN 0001-0782. doi: 001115/3591688359177 URL httpp:://doi.org////// 111.:/..000111..11/5353331 1145835111//3311115511 359176
C. E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27(3):379--4 423, 1948. doi: https:/
URL https://onlinellirree iibrrrrrrrnerbrrr..eo wiley.
com/doi/abs/ 00000000011110 1538- 7305. 1948. tb01338.x
Hanyin Shao, Jie Huang, Shen Zheng, and Kevin Chang. Quantifying association capabilities of large lan- guage models and its implications on privacy leakage. In Yvette Graham and Matthew Purver (eds.), Find- ings of the Association for Computational Linguistics; EACL 2024, pp. 814 825, St. Julian's, 's, Malta, March 2024. Association for Computational Linguistics. URL https::/acllatthllpgg org/2024. findings-eacl. 54/
Or Sharir, Barak Peleg, and Yoav Shoham. The cost of training ulp models: A concise overview. anXiv preprint ar Xrxix:2004.00 08900, 2020.
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 1671-1685, 2024.
Weijia. Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevieew www.:neeeetttteeeeeeeee watrrrmmmd=zzaaM@@am0
Reza Shokri and Vitaly Shmatikov. preeeccppeeseviig deep learning. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS '15, pp. 1310-1321, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450338325. doi: 10.1145/2810103.28111 2813667 http::///oi..rr//1..1110.
70

---

Published in Transactions on Machine Learning Research (01/2025)
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 TEEE Symposium on Securily and Privacy (SP), pp. 3 3-18, 2017. doi: 10.1109/SSP.211..41
Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowledge unlearning for llus: Tasks, methods, and challenges, 2023.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Conference on Empirical Methods in Natural Language Processing, 2013. URL https://api. www.nticschoolar.orrrrr
CorpusID: 990233
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Warbert-Voss,, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffic, and Jasmine Wang. Release strategics and the social impacts of language models, 2019.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Tan Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafford, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannanch Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annuul Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp.
Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1 /2021// 840. URL
org org/ oor.a//o...r//2222.cc2-oooog acl-long. .oeloogg..0//
Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, CCS '20, pp. 377-390, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 450340037,. doi: 10.1145/ 3372297.34172700 URL https://doi wrtp//00.ooo../0..111 114.1..114/33329999 229177777777337770
Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Dala Mining, KDD '19, pp. 196-206, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362016. doi: 10.1145/3292500 322255003330088 URL https://doi. hwt./dii.ogggg//// org/10. 1145/3292500. 3330885
Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 245-248, 2013. doi: 2..109//lobalS1P22000
Alessandro Sordoni, Xingdi Yuan, Marc- Alexande Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosscini, Friederike Niedtner, and Nicolas Le Roux. Joint prompt optimization of stacked LLMs using variational inference. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
Samuel Sousa and Roman Kern. How to keep text private? a systematic review of deep learning methods for privacy-preserving natural language processing, Artificial Intelligence Review, 56(2):1427 1492, Feb 2023, ISSN 1573-7462. doi: 00.100//10442--00 10204-6.
Nishant Subramani, Sasha Luccioni, Jesse Dodge, and Margaret Mitchell. Detecting personal information in training corpora: an analysis. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNI.P 2023), pp. 208-220, 2023.
Xianghui Sun, Yunjic Ji, Baochang Ma, and Xiangang Li. comparative study between full- hall-parameter and lora-based fine-tuning on chinese instruction data for instruction following large language model. ArXiv, abs abs//2304.08109, 2023. URL. http://pp....mmmnicccccccclllaa. http:////roo...sseeiccocoooooorr 258179474
71

---

Published in Transactions on Machine Learning Research (01/2025)
Kunal Suri, Prakhar Mishra, Saumajit Saha, and Atul Singh. Suryakiran at mediqa-sum 2023: Leveraging lora. for clinical dialogue summarization, 2023.
Lalanya Sweeney. Guaranteeing anonymily when sharing medical data, the datafly system. Proceedings conference of the American Medical Informatics Association. AMIA Fall Sympostum, pp. 51-5, 1997. org//oorpuSD:22334777
URL https://api. semanticscholar 2ttp::/sn.n.t.t.ooooooooooooooo2
Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghalaah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. Privacy-preserving in-context learning with differ- entially private few-shot generation. In The Twelfth International Conference on Learning Representations, 2024. URL
Harry Chandra Tanuwidjaja, Rakyong Choi, and Kwangjo Kim. A survey deep learning techniques for privacy-preserving. In International Conference on Machine Learaing for Cyber Securily, 2019. URL
Harry Chandra Tamuwidjaja, Rakyong Choi, Seunggeun Baek, and Kwangjo Kim. Privacy-preserving deep lcarning on machine learning as a service- comprehensive survey. IEEE Access, 8:167455-167447, 2020.
URL ssoascccol. wttp:///rrruss0:2212555555
Muhammad Imran Tariq, Nisar Ahmed Memon, Shakeel Ahmed, Shahzadi Tayyaba, Muhammad Tahir Mushtag, Nalash Ali Mian, Muhammad Imran, and Muhammad Waseem Ashraf. A review of deep
learning security and privacy defensive techniques. Mob. Inf. Syst., 2020:6535834:1
6535834:18, 2020.
URL httppsss/api..smmatticshhhotoott sewantisccholllrr httpaaaiiisssaoooooo////pppsssaa org//orpuss0:215415544
Apple Differential Privacy Team. Learning with privacy at scale differential. 2017. URL https://api.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Anlonoglou, Julian Schrittweeser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Ruther- ford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawezyk, Cosmo Du, Ed Chi, Heng- Tze Cheng, Fric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdich, Mia Chen, Pci Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan,, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Bolly, Harish Ganapathy, Steven Zheng, HyunJeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emamuel Taropa, Majd Al Mcrey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Eurique Pi- queras, Maxim Krikun, Tain Barr, Nikolay Savinov, Tvo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Ko- cisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Tlya Kornakov, Nithya Attaluri, Iñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grim- stad, Ale Jakse Hartman, Xavier Garcia, Thanmmalyan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabricla Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisolto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wee Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew A Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Aln,
72

---

Published in Transactions on Machine Learning Research (01/2025)
Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanbur, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranay Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogozińska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Al- bin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Beluam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yiin Chang, Paul Komarek, Ross Mcllroy, Mario Lučié, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lene, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuchan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko limuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjic Miao, Mohamod Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashral, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih- Kuan Ych, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carric Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia. Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víc- tor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thorton,, Fangxiaoyu Feng, Wojeiech Stokowiee, Ce Zheng, Phoebe Thacker, Çağlar Unlü, Zhishuai Zhang, Mo- hammad Salch, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowsk, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Maric Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashomi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shiv- akumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz Kępa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor,
73

---

Published in Transactions on Machine Learning Research (01/2025)
Tina Chen, Valentin Anklin, Feiran Wang, Richic Feng, Milad Gholami, Kevin Ling, Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Sid- dhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeczul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Jamus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg, Yifan He, Oleksii Duzhyi, Anton Ålgmyr, Timothée Lot- taz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, Traaçois-Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixerra, Matthew Fritze, Francesco Bertolini, Liana- Eleonora Marinescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thict, Tulsoe Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominik Ra- biej, Vipul Ranjan, Krzysztof Styre, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Isracl, Francesco Pongetti, Chih-Was "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ó Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng- Chnu-Chu Lee, Wojeiech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yan- ping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya Pranavi Potharaju, Eileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader Andrew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichen- baum, Thomas Brovelli, Sahitya Potluri, Precthi Lahoti, Cip Bactu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi IV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Ähdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojeiech Rzadkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung. Kai Yang. Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kärrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathn, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamalcldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, GeolTrey Cideron, Edouard Leurent, Mahmoud Al- nahlawi, lomut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuczhi Wang, Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashckhar, Lara Tumch, Eyal DeviDavid, Rishub Jain, Jonathan Uesato, Romina Datla, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Chene Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, An- drew Lee, Nino Vicillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Ed- ward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikolaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeck- emeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek
74

---

Published in Transactions on Machine Learning Research (01/2025)
Chakladar, Ginger Perng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan le, Rama Pasumarthi, Nathan Lintz, Amitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnic Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lec, Shuyuan Zhang, Somer Greene, Due Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHosscin Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthicu Geist, Ser Ian Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp. Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nico- las Somnerat, Denis Vnukov, Rory Greig, Jemnifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Neje Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papa- makarios, Rupert Kemp, Sushant Kafle, Tanya Grumina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Santanaaria-eernandez, Fernandez, Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Riee
Sonam
Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Pöder, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosscini, Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette-Chon Yunjic Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rívière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Chiooro, Hanna Klimezak- Pluciíska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Manger, Alexcy Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yad- lowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang. Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Sacta, Tyler Licchty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Key- van Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripurancni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, War- ren Chen, Xiangtlai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emua Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M., Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka
75

---

Published in Transactions on Machine Learning Research (01/2025)
Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vit Listík, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirnschall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, lanna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Tu Vu, Alck Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models, 2024. URL httpp://arxxv..rr//bb///2
11805
Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays. Understanding unintended memorization in language models under federated learning. In Oluwaseyi Feyisetan, Sepideh Ghanavati, Shervin Malmasi, and Patricia Thaine (eds.). Proceedings of the Third Workshop on Prinacy in Natural Language Processing, pp. 1 10, Online, June 2021. Association for Computational Linguistics. /ttp://1111211111///11 privattpiittt URL https:://cllanthooegg http:///.2.....tattepp..1111
doi: 10.18653/v1/
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Chin Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier- Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. Lamda: Language models for dialog applications, 2022.
Zhiliang Tian, Yingxiu Zhao, Ziyue Huang, Yu-Xiang Wang, Nevin L. Zhang, and He He. Seqpate: Dif- ferentially private text generation via knowledge distillation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, Curran Associates, Inc., 2022. URL pttoce///rooceedings wwwripe. ccllp/ pdf
pp. 11117- 11130.
Meng Tong, Kejiang Chen, Jie Zhang, Yuang Qi, Weiming Zhang, Nenghai Yu, Tianwei Zhang, and Zhikun Zhang. Inferdpt: Privacy-preservnng inference for black-box large language model, 2024.
P. Torremans. Copyright Low: A Handbook of Contemporary Research. Research handbooks in intellec- tual property. Edward Elgar, 2007. ISBN 9781848447097. URL X7NEPgAACAAJ
it/boksi.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajfwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosscini, Rui Hou, Hakan Inan, Marcin Kardas, Viklor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Fllen Tan, Binh Tang, Ross Taylor, Adìna Williams, Jian Xiang Kuan, Puxin Xu, Zheng
76

---

Published in Transactions on Machine Learning Research (01/2025)
Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data), In International Conference on Learning Represenlations, 2021. URL http://pppnneeiie..ee///oorroe
Id=YIWGvPFOOD-
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction (APIs). In 25th USENIX security symposium (USENIX Security 16), pp. 801-618, 2016.
Florian Tramèr, Gautam Kamath, and Nicholas Carlini. Position: Considerations for differentially private learning with large-scale public pretraining. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 48453-48467 PMLR, 21-27 Jul 2024a. URL httpss:/pproceedings wwwpediinnn.m.....ppmmm wtt..r.peee//////ttreee 24a. 24mmm11
Florian Tramèr, Gautam Kamath, and Nicholas Carlini. Position: Considcrations for differentially private learning with large-scale public pretraining. In Ruslan Salakhuldinox, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 18453-48466 PMLR, 21-27 Jul 2024b. URL httpps://proceeddngg... http:/ddinn.mm...rree////////eee 24a. 24mm
Nguyen Truong, Kai Sun, Siyao Wang, Florian Guitton, and YiKe Guo. Privacy preservation in federated learning: An insightful survey from the gdpr perspective. Computers & Security, 110:102402, 2021. doi: https:/ /ttp://///o..o//11100000 cowp/S/eeee///tttlleee////////10
ISSN 0167-4048.
URL http///www..ieeeererettttt
United Nations. Universal Declaration of Human Rights. December 1948.
Ashish Vaswani, Noam Shazocr, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, http:://ppooeeeinss.eeer
volume 30. Curran Associates, Inc., 2017. URL Paperr pdff]
Arpita Vats, Zhe Liu, Peng Su, Debjyoti Paul, Yingyi Ma, Yutong Pang, Zeeshan Ahmod, and Ozlem Kalinli. Recovering from privacy-preserving masking with large language models. In ICASSP 2024 2024 IFEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10771- 10775, 2024. doi: ML.1109/CCASSP4S0 2024.10444244.
Michael Veale, Reuben Binns, and Lilian Edwards. Algorithms that remember: model inversion attacks and data protection law. Philosophical Transactions of the Royal Society A: Mathematical, Physical and 200188833,
Engineering Sciences, 376(2133) (166(133):20188)) 80083, 2018
Davide Venditti, Elena Sofia. Ruzzetti, Giancarlo A. Xompero, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, and Fabio Massimo Zanzotto. Enhancing data privacy in large language models through private association editing, 2024. URL
18221
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Manias Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. Decodingtrust: comprehensive assessment of trustworthiness in gpt models. In Proceedings of the 97th International Conference on Neural Information Processing Syslems, NIPS '23, Red Hook, NY, USA, 2024a. Curran Associates Inc.
Boxin Wang, Yibo Zhang, Yuan Cao, Bo Li, Hugh McMahan, Sewoong Oh, Zheng Xu, and Manzil Zaheer. Can public large language models help private cross-device federated learning? In Kevin Duh, Helena
77

---

Published in Transactions on Machine Learning Research (01/2025)
Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL
2024, pp- 934-949, Mexico City, Mexico, June 2024b. Association for Computational Linguisties. doi: 10.18653/v1/202/
2ttp:///www2222.mdnnnnnn-eeeeee findings-nnacl..9. URL htttss//accaantoolog.orrr/////// https///fccaaaoooooo/.o//////ffs hin.
Hua Wang, Sheng Gao, Huanyu Zhang, Milan Shen, and Weijie J Su. Analytical compossttin of differential privacy via the edgeworth accountant, 2023. URL ottpss///ooeeeeeveee.eeee//ooorp
Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. Selective forgetting: Advancing machine unlearning techniques and evaluation in language models, 2024c. URL https://
arxiv.org/abs//442..ce 05813
Benjamin Weggenmann, Valentin Rublack, Michael Andrejczuk, Justus Mattern, and Florian Kerschbaum. Dp-vac: Human-readable text anonymization for online reviews with differentially private variational autoencoders. Proceedings of the ACM Web Conference 2022, 2022. URL seaaanicoleemm /B//Corpus10::288677888
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Grifsin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint Xrxi2:21 000:2112.000 12.0499,, 2021.
Sean Welleck, Zixin Yao, Yu Gai, Jialin Mao, Zheng Zhang, and Kyunghyun Cho. Loss func- tions for multisct prediction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- Associates, Inc., 2018. URL https://proceediingg negrsssnerriigg neurips. ha/pppprrfillesspppr///1111 3e0bc 1190bccc07221 -07211-Pppr--.---- Paper pdf
fb3f768586b38e5bffdll 1etpe//fff11@bbcccccooccc
Sam Witteveen and Martin Andrews. Paraphrasing with large language models. In Alexandra Birch, Andrew Finch, Hiroaki Hayashi, Ioannis Konstas, Thang Luong, Graham Neubig, Yusuke Oda, and Katsuhito Sudoh (eds.), Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 215-220, Hong Kong, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-5622 URL https://aclaathologgy orggD19-56222
Fan Wu, Huseyin A Inan, Arturs Backurs, Varun Chandrasekaran, Janardhan Kulkarni, and Robert Sim. Privately aligning language models with reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024a. URL https:://penreview.nntttt wtwpe//eeee/ooommid-eeeeeee
Tong Wu, Ashwinee Panda, Jiachen T. Wang, and Prateek Mittal. trivacy-preserving in-context learning for large language models. In The Twelfth International Conference on Learning Representations, 2024b. wwt/ foutf
URL
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. DEPN: Detecting and editing privacy neurons in pretrained language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2875-2886, Singapore, December 2023. Association for Computational Linguistics. doi: 10.1 11118655111/00 aetp:3..emll-mai... 174. URL. https://aclanthoolggy org.:///ooooor/2223/3// wwtp:/.........mm.pmmmmm.... ewnle- 174/
Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S. Yu. Machine unlearning: A survey, ACM Comput. Sure., 56(1), aug 2023a. ISSN 03360-3300. doi: 05.144/3603620. URL httpss///oii.rr
10.1145/ 00.1155/363620
Zekun Xu, Abhinav Aggarwal, Oluwaseyi Feyisetan, and Nathanael Teissier. A differentially private text perturbation method using regularized mahalanobis metric. In Oluwaseyi Feyiselan, Sepideh Ghanavati, Shervin Malmasi, and Patricia Thaine (eds.), Proceedings of the Second Workshop Privacy in NLP, pp. 7-17, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. privateulp-1.2.
Zekun Xu, Abhinav Aggarwal, Oluwaseyi Feyisetan, and Nathanael Teissier. On a utilitarian approach privacy preserving text generation. In Oluwaseyi Feyisetan, Sepideh Ghanavati, Shervin Malmasi, and
78

---

Published in Transactions on Machine Learning Research (01/2025)
Patricia Thaine (eds.), Proceedings of the Third Workshop on Privacy in Natural Language Processing, pp. 11-20, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1 0058661120022 2ttp222001.1111en1ppppp privateenl-11 2. URL
hwtph//ooooo..o/221111.. wtt.:r////22..riiateen----111 1rriiaatenlp1.2//
Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher Choquette, Peter Kairouz, Brendan Memahan, Jesse Rosenstock, and Yuanbo Zhang. Federated learning of gboard language models with differential privacy. In Sunayana Silaram, Beala Beigman Klebanov, and Jason D Williams (eds.), ; Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties ( (Volume 5: Industry Track),
pp. 629 229-639, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/v1 vv1 2023. acl-industry.66. URL httpss//acclanthooogg.cooo wrt.://gg.rrr//2233//// http://aaa.oooooooo////222aaaaaa
Andrew C. Yao. Protocols for secure computations. In 23rd Annual Symposium on Foundations of Computer Science (sfes 1982), pp. 160-164, 1982. doi: 10.1109/SFCS.1900 S98.............
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 8403-8419, Bangkok, Thailand, August 2024a. Association for Computational URI.
Linguistics. doi: 10. (0.18653/v1 /v/ /2202.222--oong.....
wrg.ge..oor//.. org httgg/222.allloong... 457
Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (Il)) security and privacy: The good, the bad, and the ugly. High- Confidence Computing, 4(2): 100211, 2024b. ISSN 2667-2952. doi:
URL https://www
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Ana- lyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF). pp. 268-282. IEEF, 2018.
Hongxu Yin, Arun Mallya, Arash Vahdat, José Manuel Álvarez, Jan Kantz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp.
2021. URL
Ttgg/CorpusSDD233241007
Ying Yin and Tvan Habernal. Privacy-preservinn models for legal natural language processing. In Nikolaos Aletras, Ilias Chalkidis, Leslie Barrett, Cătälina Goantä, and Daniel Preotiuc-Pietre (eds.), Proceedings of the Natural Legal Language Processing Workshop 2022, pp. 172-183, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. doi: 10.18653, w1/2022..2lllp2p..// 1. 1.14. URL https://aclaantholggyyyoo
1l.p-1.14)
Yeojoon Youn, Zihao Hu, Juba Ziani, and Jacob Abernethy. Randomized quantization is all you need for differential privacy in federated learning. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunitics, 2023. URL http:://oeeeeeeeeeeeeee//ommmets
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opa- cus: User-friendly differential privacy library in pytorch. In NewrIPS 2021 Workshop Privacy in Machine Learning, 2021. URL
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Yan Liu. Large scale private learning via low- rank reparametrization. In Marina Meila and Tong Zhang (ods.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12208- 12218. PMLR, 18 24 Jul 2021. URL https::/proccedinggs... mlr.
http:///weecnnnno...rrrrrrs///// pptpss//139/yu2ffffff. htmll
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Leo, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differ- entially private fine-tuning of language models. In International Conference on Learning Representations, 2022. URL TtRLhttps:/// httper/wwwwweeeeeeooooooe///eore
79

---

Published in Transactions on Machine Learning Research (01/2025)
Da Yu, Sivakanth Gopi, Janardhan Kulkarni, Zinan Lin, Saurabh Naik, Tomasz Lukasz Religa, Jian Yin, and Huishuai Zhang. Selective pre-training for private fine-tuning. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL htttp::/opeereviiew http:s////peeeeeeeeeeeeee/rrrrrp
Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, and Sherman S. M. Chow. Differential privacy for text analytics via natural text sanitization. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli
(eds.). Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3853-3860 Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1 ht11///2...ffff findings-acl.337..
URL https://aclanttoooo.oorrr. http://022...iiinng--aaaaa findings-acl.
Xiang Yue, Huseyin Tnan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim. Synthetic text generation with differential privacy: A simple and practical recipe. In Anna Rogers, Jordan Boyd- Graber,, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Lánguisties (Volume Long Papers), pp. 1321-1342, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: https://aclaantoology org /ttg. acl-long. 74/
acl acl-long.744 URL
Santiago Kanella-Béguelle Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Mare Brockschmidt. Analyzing information leakage of updates to natural language models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications securily, pp. 363-375, 2020.
Santiago Zanella- Begillli,,, Shruti Tople, Andrew Paverd, and Boris Köpf. Grey-box extraction of natural language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12278-12286 PMLR, 18 24 Jul 2021. URL
mlr http:ings........rsssee wttla/////aeaaaaeeeeeeeeeee
html
Collin Zhang, John Xavier Morris, and Vitaly Shmatikov. Extracting prompts by inverting LLM outputs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14753 14777, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10. 10.18653/v1 12////3/11222..ee// /2204...222pppmmm 819.. URL https: /accanthology.orgg 002/ em12244..mppm...... 8199
Dan Zhang, Ryan McKenna, Ios Kotsogiannis, Michael Hay, Ashwin Machanavaijhala, and Gerome Mik- lau. Ektelo: A framework for defining differentially-private computations. In Proceedings of the 2018 International Conference on Management of Data, SIGMOD '18 pp. 115-130, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450347037. doi:
URL
https://doi..rg// 10, 111111111//1111... 3196921
Dawen Zhang, Pamela Finckenberg- Bromann Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. Right to be forgotten in the era of large language models: implications, challenges, and solutions. Al and Ethics, Sep 2024b. ISSN 2730-5961. doi: 101107//43888.024-0673-- URL https://doi..or// 10077744581---9440057--99
Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramer. Membership inference attacks cannot prove that a model was trained on your data, 2024c. URL http://aaxiiv..orr///////////
Rui Zhang, Song Guo, Junxiao Wang, Xin Xie, and Dacheng Tao. A survey on gradient inversion: Atlacks, defenses and future directions. In Lud De Raedt (ed.), Proceedings of the Thirty-Firss International Joint Conference on Artificial Intelligence, IJCAI-22, pp. 5678-5685. International Joint Conferences on Artificial Intelligence Organization, 2022a. doi: 20.29963//icaa.2222/9911 URL https://doi.onn org//000, Survey Track.
14963/ijcai.20227777
Rui Zhang, Song Guo, Junxiao Wang, Xin Xie, and Dacheng Tao. A survey on gradient inversion: Atlacks, defenses and future directions. In Lud De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAT-22, pp. 5678 5685. International Joint Conferences on Artificial Intelligence Organization, 2022b. doi: 224966/jjaai.202/ 00963/ijcaii.0022777 Survey Track.
URL https://doo org/10.
80

---

htw.:////.................. Published in Transactions on Machine Learning Research (01/2025)
Ruisi Zhang, Scira Hidano, and Farinaz Koushanfar. Text revealer: Private text reconstruction via model inversion attacks against transformers. arXiv preprinl ar arXiv:2209.106: 10505, 2022c.
Yi Zhang, Yunfan Lu, and Fengxia Liu. A systematic survey for differential privacy techniques in federated learning. Journal of Information Security, 14:111-135, 01 2023a. doi: 10.42366/jis 200....00....
Zhexin Zhang, Jiaxin Wen, and Minlie Huang. ETHICIST: Targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation. In Anna Rogers, Jordan Boyd-Graber. and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguislies (Volume 1: Long Papers), pp. 12674 12687, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10. 111885/111202233 hwtp:/22.....co....1. 709. URL hatess//aclaanthoooyy..g///22/
acl ocl-long.1. 709
Chuan Zhao, Shengnan Zhao, Minghao Zhao, Zhenxiang Chen, Chong-Zho Zhi Gao, Hongwei Li, and Yu an Tan. Secure multi-parly computation: Theory, practice and applications. Information Sciences, 476:357 372, 2019a. ISSN 0020-0255. doi:
URL https://www.
Jingwen Zhao, Yunfang Chen, and Wei Zhang. Differential privacy preservation in deep learning: Challenges, opportunities and solutions. IEEE Access, 7:48901 48911, 2019b. URL https://api. semanticscholar.
TTt//oorpusDD:13177777
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yur Nie, and Ji-Rong Wen. A survey of large language models, 2023.
Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. When docs pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, ICAIL '21, pp. 159-168, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450385268. doi:
00.1145/4462757.3466668 URL https:///oi.coo
11411811153344777344666566 3466888
Mengxin Zheng, Qian Lou, and Lei Jiang. Primer: Fast private transformer inference on encrypted data. 2023 60th ACM/IEEE Design Automation Conference (DAC), pp. 1-6, 2023. URL https:
Nina Zhou, Qiucheng Wu, Zewen Wu, Simeone Marino, and Tvo D. Dinov. Datasiftertext: Partially synthetic text generation for sensitive clinical notes. Journal of Medical Systems, 46(12):96, Nov 2022. ISSN 1573- 689X. doi: 10.1007 http///009002220088100
URL httpp:://doi.org////
(0007//10916-222/ 01880-6
Junyi Zhu and Matthew B. Blaschko. R-[gap): Recursive gradient attack on privacy. In International Conference on Learning Representations, 2021. URL htttps://openreviiww httpp://pppeeeeeieee.e///rrrrrrr
7UcKf
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In H. Wallach, H. Larochelle, A. Beygekiiee, F. d' Alché-Bue, F. Fox, and R. Garnett (eds.), Aduances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http::htttt//sseo/ /ttp://///ccceeieessoooo neurips. http:cccepppeeeeeeeeeeeee
Paper.
Yuqing Zhu and Yu-Xiang Wang. Adaptive private-k-seloction with adaptive k and application to multi- label pate. In Gustau Camps-Valle Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 5622 5635. PMLR, 28 30 Mar 2022. URL httph:t/pproceeedinngggo///// www.mmlrrrrrrrr.. ..l..press/v111/ zhu22e.html
Zheming Zuo, Matthew Watson, David Budgen, Robert Hall, Chris Kennelly, and Noura Al Moubayed. Data anonymization for pervasive health care: Systematic literature mapping study. JMIR Med Inform, 9(10): e29871, Oct 2021. ISSN 2291-9694, doi: 10.2196/298711 URL https:://oi. httg/11..196/2987
81

---

Published in Transactions on Machine Learning Research (01/2025)
A
Going Further
As shown throughout this work, the topics discussed here have been subject to extensive research in recent years. In our work, we have shown how privacy-preeering techniques like anonymization or Differential Privacy have been used to build private Language Models. For those readers interested in deepening their knowledge on a single subject, we present here a list of surveys specific to each single topic:
Data Leakage: Jegorova al. (2021) presents a comprehensive survey on privacy leakage inference time. They cover involuntary leakage and adversarial leakage and also touch on the available defence mechanisms. The main difference with our work is that we are focused on LLMs, which is also reflected in the techniques shown as solutions, which are much more recent.
Attacks: There are several surveys investigating attacks. Rigaki & Garcia (2020) investigate the privacy vulnerabilities of machine learning in general, while some other works are more focused on
specific topic. For example, Enthoven & Al-Ars (2021); Zhang et al. (2022b) Liu et al. (022)) focus on attacks against federated learning models or Derner et al. (2023): Li et al.
2023b) focus on
security risks of LLMs, including possible attacks and mitigations. While some of these works have
partial overlap with our section $ we mostly focus our investigation on privacy attacks against
LLMs and possible solutions.
Privacy Preserving Deep Learning: Literature yields many works on the topic of privacy- preserving deep learning. Some of them are generic, without focusing on any specific model. This is the case for Boulemtafes et al. (002)); Tanuwidjaja et al. (0020); Tariq et al. (2020); Liu et al. (2021); Tanuwidjaja et al. (2019); Mirshghallah et al. (2020) which all touch on privacy issues in deep learning and possible solutions. Although these works are relevant, they primarily focus on deep learning in general and do not address the recent advancements brought by LLMs. Recently,
Yao et al. (202bb) provided systematie review on security and privacy in language models divided into three sections: good applications, bad applications, and security risks and solutions. This work is complementary to our study since it covers a broader range of risks (not just privacy risks) while allocating less space to solutions. Ultimately, works focused on privacy in NLP, such as [Suus &
Kern 20233 concentrate on NLP and anonymization, while Stnerra-Manzanaree et al. 2023 targets
medical applications without necessarily focusing on textual data.
Machine Unlcarning: There are several surveys on the topic of Machine Unlearning: Xu ct al.
(2023a); Nguyen et al. (002 Shaik et al. (0024) are general, while Si et al. (2023 Liu et al. 2024a)
only focus on LLMs and the differences in application.
Differential Privacy: With the growing interest in Differential Privacy in recent years, numer- ous literature reviews have emerged. They all focus different aspects of Differential Privacy: Prokhorenkov & Cao (2023) is focused on the privacy risks of Differential Privacy, how to define it and measure it Several works invest igaled the intersection between Differential ial Privacy and deep
learning like Baraheem & Yao (2022); Zhao et al. (0019b); Ponomareva et al. (2023) Demelius et al.
(2023): Chen et al. (2024a) focus on generative models in particular; Klymenko et al. (2022) focus on DP the context of NLP: Garrido et al. (2023) present particularly interesting work as it focuses on the industry, the difficulties in the adoption and gives some suggestions to close the gap
between industry and academia; Ouadrhiri & Abdelhadi (2022) Zhang et al. (2023a): Odera (0223
focus on the applications in federated learning.
Federated Learning: Regarding federated learning, several surveys are available with different
perspectives. Kairouz al. (2021b) propose a general survey on federated learning, examining the
state of the art and open problems. Hasan (2023 Ma et al. (2022) Truong al. (2021) give
overview of the privacy and security issues, with Truong et al. [2021 also putting it in relation to
GDPR. Lastly, Brauneck et al. (2023) gives an overview of FL in medical rescarch.
Large Language Models: Chang Bergen (2024); Zhao et al. (023): Minace et al. (2024)
present comprehensive surveys large language models (LLMs), covering various aspects such
82

---

Published in Transactions on Machine Learning Research (01/2025)
their architecture, training techniques, applications, and the challenges they face. These surveys provide valuable insights into the current state of LLM research, highlight recent advancements, and discuss future directions for the development and deployment of these models.
Anonymization For text data anonymization, the most important works are related to the health-
care domain, which is the case for Larbi ct al. (2022); Zuo et al. (2021 Meystre et al. 2010
83

---

Published in Transactions on Machine Learning Research (01/2025)
B Summary Tables
In this section, we present summary tables for all the approaches discussed in the text. Table 2 and Table summarize the anonymization techniques respectively presented in $4.1 and $1.122 Table presents some information about the works that perform a fully private training, discussed $5.111 with information about the model, the datasets used both for pre-training and fine-tuning and the optimization method. Similarly, in Table there is an overview of the works that perform private fine-tuning, discussed in $5.1.2 Table @
summarizes the works found in the literature for private inference, which are discussed in 5.2 Lastly, Table
presents brief summary of the Unlearning methods presented in $5.4
Data Anonymization
Table 2: Classical methods for text anonymization
Method
Summary
Reference
Modular system that has pre-processing,
Modular
NER, CRR and Anonymization.
Mamede et al. (0016)
Performances dependent on NER module.
Altering data while preserving statistical significance. Effectively anonymizes data but may compromise the accuracy. Sensitive information masked and then imputed with BERT. Whitelist to mask sensitive terms, blacklist for mcaningful terms to retain. Retains high data utility.
Perturbation
Zuo et al. (2021)
DataSifterText
Zhou et al. (2022)
Text analyzed to extract identifying features, which are then smoothed. Universal Transformer is used to generate text incorporating the features. Integrates modern model with classical approaches. Specific to Russian.
Transformer
Romanov et al. (2019)
Attention-based text representation learner extract latent embeddings and a deep RL-based privacy preserver manipulates them. Effectively balances privacy and utility, but dependent on precise parameter tuning and has data-specific limitations.
RLTA
Mosallanczhad et al. (2019)
Use LLM to impute previously masked tokens. They find NEs with NER and then mask them. Performances are significantly better when compared to a model trained on the masked data.
LLMs
Vats et al. (2024)
Two local LLMs, the first one anonymizes the prompt before sending it to a black-box LLM, the sccond one de-anonymizes the output from the black-box LLM. They show good privacy-utlity tradeoff.
HaS
Chen al. (2023b
84

---

Published in Transactions on Machine Learning Research (01/2025)
Data Anonymization with DP
Table 3: Methods for text anonymization with DP
Method
Summary
Reference
Clipping and noising the embeddings before decoding to build DP latent representation.
ADePT
Krishna et al. (2021)
DP proof is problematic.
Apply isotropic Gaussian noise in the latent space. Good results for authorship anonymization.
DP-VAE
Weggenmann et al. (2022)
Clipping and noising the embeddings, new clipping and iterative pruning allows for a smaller encoder. Performances over the baseline. Purkayastha Mechanism used to take advantage of the angular distance as metric. Performances close non-private and increased MIAs robustness. Words mapped into embeddings, then compule distances and use them to sample replacement inversely proportional to the distance
DP-BART
Tgamberdiev & Habernal (2023)
BERT
Du et al. (2023b)
SANTEXT
Yue et al. (2021)
Replace every token: mapping function determines output space and sampling function chooses replacement tokens. Good privacy/utilty tradcoff.
CUSTEXT
Chen et al. (2023a)
Random adjacency lists to increase embedding inversion attacks robustness. Used with black- box LLMs and do-anonymization module for open-ended text generation, shows very strong performances New noise distribution specifically designed for DP. Epsilon calibration based geometric propertics of the word embedding space. Shows lavorable privacy srivacy/utilityy /utility tradeoff.
RANTEXT
Tong et al. (2024)
bILSTM
Feyisetan et al. (019bb
Truncated Exponential Mechanism to replace standard Exponential Mechanism, which not suitable for metric DP as it does not consider the specific structure the embedding space. Substantial improvements over baseline.
TEM
Carvalho et al. 2021
Regularized Mhalanobis Metric used for text perturbation to add elliptical noise to ensure that words in sparse regions have a sufficient likelihood of replacement without sacrificing overall utility.
RMM
Xu et al. (2020)
Include grammatical categories into privatization to preserve grammatical properties. Improves performances but slightly docreases privacy. Fine-tuned GPT2, introduce noise, temperature is sampled based on the exponential mechanism. Superior performance with respect to word-level DP.
Syntactic Approach
Arnold et al. 2023
Paraphrase
Mattern al. (202bb
85

---

Published in Transactions on Machine Learning Research (01/2025)
Models pre-trained with DP
Table 4: Models pre-trained with DP
Model
Method
Datasets
Reference
BERT - pre-training
BERT
DP-SGD
Hoory et al. (0011)
i2b2-2010 fine-tuning
DP-SGD
BookCorpus+ Wikipedia++ pre-training
BERT
Yin & Habernal (002
SGD
Overruling dataset/Caso CCascHODD fine-tuning
Brown corpus pre-training
Feedforward NN
DP-SGD
Kerrigan et al.
2020
Reddit comments dataset fine-tuning
Solected 15% of OpenWebText pro-trainig
GPT2
DP-SGD
Tramer & Boneh
(0211
Enron email dataset fine-tuning
Models fine-tuned with DP
Table 5: Models fine-tuned with DP
Model
Method
Dataset
PEFT Reference
GPT2
DP-SGD
Open Yelp Dataset
X
Yue al.
(2023)
RoBERTa
DP-ADAM
SNKL,,QP,ONLI,SSTT
X
ei al.
2022
GPT2
DP-ADAM
E2E dataset
V
Li et al.
(2022)
DialoGPT
DP-ADAM
Persona- Chat dataset
X
Li et al.
(2022)
GPT2
DP-SGD
IMDb, Amazon reviews dataset
X
Mattern al.
2022a
ROBERTA
DP-SGD
MNLI, SST-2, QQP, QNLI
Yu et al.
(2022)
GPT2
DP-SGD
E2E, DART
V
Yu et al.
(2022)
BERT
ADAM
SST, QQP, TP-UK
719
Li et al.
(023ee)
T5
ADAM
SST, QQP, TP-UK
Li et al.
(002bb)
GPT2
DP-SGD, DP-PPO Reddit TL;DR, IMDb
X
Wu et al.
[2024a]
"The authors used prompt-tuning and prefix-tuing after obfuscating the embeddings 'The authors used prompt-tuning and prefix-tuiigg after obfuscating the embeddings
DP-Inference methods
Table 6: DP-Infernnce methods
Model
Method
Datasct
Reference
Report-Noisy-Max-Er
SST-2, Amazon,
GPT-3
ESA
AGNews, TREC,
Wu et al.
(024bb)
KSA
PFI-DocVQA. SAMSum
GM,
AGNews, DBPedia, TREC
GPT-3
Tang et al.
2024
EM
MIT-G, MIT-D
SST-2, AGNews, TREC,
GPT-3
PromptPATE
Duan et al. 22024
DBPedia
Llama-2,
SST-2, TREC, Mpqa,
GPT-3,
DP-OPT
Hong et al. 2024
Disaster
Vicuna
T5, BERT,
Evae Offensive, Hate Speech 18, Healt Fact, Daily Dialogue and others
Seeio-and-Denoise
Mai et al. 2024
GPT-2
GPT-4,
CNN/Daily Mail, Wikitext-103-vl,
DPT
Tong et al.
2024
Vicuna
ArXiv Dataset
86

---

Published in Transactions on Machine Learning Research (01/2025)
Machine Unlearning Methods
Table Machine Unlearning Methods
Method
Summary
Reference
Allows to track factual predictions back
ROME
single neurons and edit them individually.
Meng al. (0022
Proven to be ineffective in some cases.
With a similar approach with respect to ROME, scale the editing multiple examples.
MEMIT
Meng et al. (002
The discussion is focused on factuality.
This method assumes that private information is encoded in specific neurons: their weights are set to zero to block information flow.
DEPN
Wu et al. (2023)
Model editing to break the association between user identity and its PIT. The edit method effective both in batch and in sequential editing,
PAE
Venditti et al. (2024)
Divide training data in shards, further divided in slices and then train instances of the model each shard by incrementally using slices and saving parameters at every step.
SISA
Bourtoule et al. (2019)
Requires limited re-training.
Negate the loss function used in training to maximize the loss on the target sequence (to unlearn)
KU for privacy
Jang et al. (2022)
Fine-tune model on target sequences, then it lower the likelihood sequences predicted by this model. Build Unlcarning layers and train them with a selective sludent-teacher objective based KL-divergence to maximize divergence from teacher model
Approximate UL in LLMs
Eldan & Russinovich (2024)
UL layers
Chen & Yang (2023)
Take samples from pre-training datasel, divide in prefix and suffix, use a negative
similarity metric to evaluate how different is the generated suffix from the real one.
DeMemorization
Kassem al. (2023)
This is used
reward signal to encourage the LLM to develop paraphrasing policy
Use LoRA to instruct the model with a sanitization phrase when a potentially harmful prompt is given Build a context where the datapoint you want to unlearn included but with a flipped label
Knowledge Sanitization
Tshibashi & Shimodaira (2023)
ICUL
Pawelczyk et al. (2024)
C
Discussion on Federated Learning
The central model of Differential Privacy (CDP) enforces privacy at the participant level rather than the record level. Record level Differential Privacy corresponds to the classical definition (see $2.2 for more details) and states that the outcome of differentially private computation should be the same (with probability c) regardless of whether particular record or is not in the datasct. As Nascri et al.) (2020) explain, participant-level DP ensures (with probability e) that the aggregation function produces the same result regardless of whether particular client participates in the training process.
87

---

Published in Transactions on Machine Learning Research (01/2025)
Although this method is more secure than sending plain data, it requires high level of trust in the central server and in the communication channel. Clients send unperturbed gradients, which can lead to training data reconstruction via gradient inversion (Zhang et al. 2022a), and even averaged gradients before perturbation can be exploited for reconstruction, as shown by Geiping et al. (2020), see $3.3 for more information. This means that an honest-uut-curous (or semi-honest (Paverd & Martin 2014 server could potentially access the training data. Given these risks, Central Differential Privacy is not widely used. However, Nascri ee al. (2000) demonstrates that CDP can reduce federated learning's 's vulnerability to backdoor and white- box Membership Inference Attacks ($3.2). For instance, on the CIFAR-10 dataset, they achieved a 23% reduction in the accuracy of MIAs without a significant loss in utility.
McMahan et al. 20118) introduce DP-FedAvg, a variation of the FedAVG algorithm by McMahan et al. (2016), incorporating the moments accountant (Abadi et al.) 2016) to calculate the privacy budget used during training. Typically, FedAVG performs part of the training locally on a subset of clients selected based on specific criteria. The clients then send their updated gradients to the server, which averages them, updates the global model, and sends the updated model back to the clients. In DP- gradients are clipped locally but not obfuscated. The central server adds noise after receiving and aggregating the gradients to prevent inversion, aligning the algorithm with central DP since local clipping alone does not meet the Local DP criteria. This approach was tested by training an LSTM for the next token prediction on a large dataset of Reddit posts described by [Al-Rfou et al. (2016), using various noise levels and clipping strategies. The model's performance was evaluated using Toppl, the probability that the predicted word is correct. The study demonstrated that, with sufficient data, it is possible to train a model with privacy guarantees without significantly sacrificing performance, provided the noise added is kept within reasonable limits.
Ramaswamy et al. (020)) apply DP-FedAVG to develop a private Next Word Prediction (NWP) model using an LSTM in a federated setting. Their goal is to train cnaluction-qualiiy NWP models with De-FedAvg in a real-world environment, spocifically on a diverse flect of mobile phones used for Gboard. Unlike DV-FedAVG's typical Poisson sampling, where each user is selected independently with a fixed probability for each round, this work employs fixed-size federated rounds. In this method, a fixed number of users is randomly sampled to participate in each round, ensuring a consistent number of participants and facilitating better management of computational and communication resources in production environment like Gboard. The model's performance is evaluated on a user-generated dataset from Gboard, using the same private dataset for training, and is measured in terms of Top-1 Recall and Top-3 Recall. The results surpass the non-private baseline N-gram FST (Finite State Transducer). Privacy attacks based on the work of Carlini et al. (2019) are conducted to test the model's s resistance to memorizing canaries. The findings show that the model does not memorize canaries unless they are repeated approximately 200 times or shared among roughly 16 users. The authors emphasize that the Differential Privacy approach relies on several assumptions mostly sound but challenging to verify- highlighhttgtnnnggggggg the importance of empirical measurements like canary tests to validate privacy guarantees.
Currently, large language models (LLMs) represent the state of the art due to their superior power, though they are much larger than the small LSTMs seen in previous studies. Wang et al. (2024b) focus on using LLMs to enhance the privacy/utility tradeoff of local LMs in differentially private federated learning. Similar to Yu et al. (2024), which samples pro-training data basod on its similarity to fino-tuning data, they propose a distribution matching algorithm to sample public data close to the private data distríbution, significantly improving sample efficiency. They utilize DP-FTRL (Kairouz et al. 2021a). an online optimization algorithm combining FTL (follow the leader) and regularization, adding noise during local updates. The local LM is
either a single-layer LSTM (Hochreiter & Schmidhuber 1997) or Transformer (Vaswani et al. 2017 due
to computational limits on edge devices, while the LLM used is LaMDA (Thoppilan et al.] 2022 Their
tests yielded several notable results. First, using a public tokenizer with a public vocabulary instead of a private one helps prevent privacy leakage. Second, pre-training local models on a public dataset (C4 dataset) for over week of single GPU time improved next-token prediction accuracy (tested on the private StackOverflow dev set). They also built distillation corpus by saving the top-k logits with k nonzero entries from the teacher LLM's next token predictions as a silver label dataset. Pre-training on this distillation corpus showed accuracy comparable to pre-training with the original dataset, even with just 1% distillation
88

---

Published in Transactions on Machine Learning Research (01/2025)
coverage. Furthermore, their new distribution matching technique allows using only 0.08% of the pre-training dataset to achieve performance equal to 1% distillation coverage, greatly enhancing sample efficiency.
Xu et al. (202b)) present and analyze twenty Gboard LMs trained for Next Word Prediction using DP-FTRL, asserting that all future Gboard models will have DP guarantees. In their DP-FTRL implementation, model updates are clipped, and noise is added during global model updates, ensuring Central Differential Privacy. They also use adaptive clipping (Andrew et al. 2021), which adjusts the clip norm each round by privately estimating the model's norm at targeted quantile. However, adaptive clipping showed no significant improvement and was less robust in experiments with large report goals, sometimes causing catastrophic failure or no progress in the first 1000 rounds. Despite this, adaptive clipping can reduce hyperparameter tuning when the privacy budget permits. Public pre-training on C4, as explained by WWang et al. (2024b), reduced the rounds needed to reach target utility by 1000 under the same noise multiplier, enhancing privacy guarantecs. They also combined DP-FTRI. with Secure Aggregation (Bonawitz et al., 2017), a aryptographic protocol ensuring the central server only accesses aggregated updates, not individual client updates. While SecAgg aids DP, il introduces significant computational slowdown and requires different system configurations for large report goals. The techniques for training language models with differential privacy in Gboard have produced models with stronger privacy guarantees than some established standards, such as those used by the US Census Bureau. In Europe, these differentially private neural network models outperform older N-gram models in utility. For English in the US and Portuguese in Brazil, the new models are on par with or slightly better than their non-private counterparts.
Basu et al. (021) provide another example of differentially private training of LMs within a central model in federated setting. They trained BERT, RoBERTa, DistilIBRTT, and ALBERT on tweets labelled to detect depression (Depression dataset) and sexual harassment (Sexual harassment dataset). Although the authors do not detail the training procedure, their results show that all models were trained with varions epsilon values. Interestingly, the smaller models performed more favourably under these conditions.
C.1 Federated Learning with Local DP
Initially formalized by Kasiviswanathan et al. (2008), local Differential Privacy (LDP) involves clipping and adding noise locally at the clients' level before sending gradients (see Figure 5.3). This method removes the need for a trusted central server and ensures safety even if an eavesdropper intercepts the gradients as they are noisy. It makes the process safer, but it comes with its own challenges. Though primarily used in networks handling image data, LDP approaches are worth mentioning for LLMs, as there is no thcoretical barrier to their application. The main obstacle is the computational burden coming from the high dimensionality of text data, which may be mitigated in the future.
Shokri & Shmatikov 2015) introduce DSSGD (Distributed Sclective Stochastic Gradient Descont) for private distributed training of ML.P and CNN models on MNIST and SVHN datasets for digit classification. While not directly related to language modeling, the algorithm is broadly applicable to various network types. DSSGD builds on SSGD, a local Selective SGD that selectively applics gradients based on certain criteria to reduce the amount of training data information used for updates, thereby enhancing privacy, though not guaranteeing it Tn DSSGD, each client trains a local model using SSGD's selection mechanism, and then shares selected parameters with a central server. The server aggregates these parameters to update the global model and redistributes it to clients for further local updates. This approach, however, has potential privacy risks from gradient selection and sharing. To address these, the authors use the Sparse Vector Technique (wwork et al., |006aa| Hardt & Rothblum 2010)) to randomly select gradicnts exccoding a threshold and add noise using the Laplacian mechanism. Their experiments show that models trained with DSSGD achieve accuracy close to non- con-iiivate methods. Specifically, on the MNIST dataset, the accuracy is almost indistinguishable from non-private training. Additionally, even with only 1% of parameters shared, the collaborative model significantly outperforms single local models.
Youn et al. (0023) introduce RQM (Randomized Quantization Mechanism), a novel approach to enforce DP in federated scttings with improved communication efficiency, used alongside Distributed SGD on a CNN for image classification on EMNIST. RQM enhances DP-SGD by eliminating the need for explicit noise addition; instead, RQM is performed after gradient clipping. Tn RQM, gradient updates are encoded in quantization
89

---

Published in Transactions on Machine Learning Research (01/2025)
levels, which are then randomly subsampled. A randomized rounding procedure maps continuous gradient values to the nearest subsampled quantization level. RQM falls under Renyi Differential Privacy (Mironov 2017), relaxation of the original differential privacy definition, placing the method under Local DP since RQM is performed on the clients. Tests show that RQM outperforms the Poisson Binomial Mechanism (Chen et al.] 2022b) (a traditional noise injection method) and noise-Tree clipped SGD, achieving better accuracy with stronger privacy guarantees.
Bebensee 2019 provide comprehensive overview of Local Differential Privacy (LDP) and its notable implementations, highlighting the increased variance and noise compared to the central model of DP. In LDP, perturbations are performed locally, leading to higher total variance, which directly depends on the number of participants. The lower error bound in the local setting is dependent on (the number of participants), necessitating a very high number of participants to inject sufficient noise. Additionally, since each client adds its own noise, the total noise much higher, requiring significantly more data for efficient learning or a higher
LDP is widely used in the industry to collect and aggregate telemetry data. While these approaches are not directly relevant to the research presented here, they demonstrate how high-profile companies use DP in daily operations. Google (Erlingsson et al. 2014), Microsoft (Ding et al. 2017), and Apple (Team] 2017) have developed their own LDP algorithms to collect and aggregate usage statistics for applications like Chrome and the Apple keyboard. This also goes to show once again how local DP is widely used, but for low dimensionality data (telemetry data are usually counters), while for high dimensionality data like text there is still no sound implementation.
90


Reviews:
[{"review_id": "sn5Ww7WYwe", "reviewer": "Reviewer_CECd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Extremely comprehensive survey that deals with an extremely important topic of privacy preserving ML. I think this is a timely survey too, as the issues raised in this work are some of the foundational issues that we are already facing with LLMs.\n2. The survey is painstaking well written and it makes reading it quite enjoyable. \n\nWeaknesses:\n\n1. Well, maybe it is too long -- but it is expected to be for such a broad area.\n2. For such a long article, missing out homomorphic encryption based ML is unacceptable. It may be well advised to cover at least the latest and greatest works in this area.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is a survey article on a broad range of privacy preserving techniques in AI, with a particular focus on Large Language models. The range of privacy techniques/ML topics covered include: classic anonymization, differential privacy, multi-party secure computations, federated learning, machine unlearning etc. \n\nThe survey starts with a detailed discussion on a wide range of privacy attack models: training data extraction (extract real examples that were used in the training), membership inference attacks (identify the training examples from a pool of examples) and model inversion (extracting the blackbox model using outputs and gradients.) \n\nSubsequently, they present the broad range of solutions available in literature, discuss their limitations and applicability in various scenarios. These techniques include: classical anonymization, anonymization using differential privacy, fine tuning using differential privacy approaches, federated learning approaches within the context of differential privacy. \n\nFinally, the survey discusses the topic of machine unlearning, where the goal is to \"delete\" some of the training examples from a trained model (especially, a large language model such as GPT2).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see a few paragraphs included for Fully Homomorphic encryption based machine learning. This will make the survey article really and truly complete.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2vfWGIEBHk", "reviewer": "Reviewer_iggB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \n- The survey focuses on privacy in the context of LLMs, which is an important research direction \n- The organization is simple and easy to follow\n\n## Weaknesses\n- The paper sometimes provides excessive detail on aspects not central to understanding it (e.g., machine learning, deep learning) while offering too little detail on important aspects (differential privacy mechanisms such as Laplace/Gaussian/Exponential mechanisms)\n- The informations presented in tabular format are redundant\n- The current presentation of threats focuses only on the LLM, ignoring privacy threats from user perspectives (i.e., input privacy at inference time)\n- LLMs can also be vulnerable to model stealing, which can also have privacy implications. I would like to see some discussion on that aspect\nSome solutions/tools do not seem to apply directly to LLMs. Given the survey's focus, it would be great to focus on what can apply directly to LLMs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey considers the problem of preserving privacy in large language models (LLMs) and presents existing privacy threats associated with LLMs, countermeasures, and tools for implementing them. The presentation of existing privacy attacks covers training data extraction, membership inference, and model inversion attacks. The solutions span privacy-preserving data publishing (k-anonymity and differential privacy-based), differential privacy at model level (pre-training with DP, public pretraining + private fine tuning with DP...), federated learning and machine unlearning. Finally, the presentation on existing tools covers open-source libraries for privacy-preserving data publication, differential privacy, and federated learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Provide some references for the definition of PII. I would suggest the NIST and GDPR definitions\n- The preliminaries section can be shortened by removing subsections on machine learning and deep learning (just provide references to existing resources like books)\n- Consider clarifying why input privacy at inference is not discussed\n- Section 2 introduces black-box and white-box access settings. However, grey-box setting, which is later used in the paper, was never properly defined\n- Consider defining sensitivity, Laplace/Gaussian/Exponential mechanisms in the background section, right after DP definition\n- The definition Rényi differential privacy introduces a mechanism \\mathcal{M} but another mechanism f is used in the inequality.\n- Since the survey focuses on LLMs, I think the discussion on the solution should focus on the methods tested for such models. For example, the discussion on traditional anonymization techniques mentions techniques that leverage LLM to sanitize datasets. However, the datasets mentioned are not used to train LLM. So, these techniques do not qualify as solutions to enhance privacy in the context of using LLMs. I have a similar remark regarding anonymization with DP. The discussion should focus on large language models. Some discussion is needed on how these methods can be applied to the training data of large language models. \n- For example, SANTEXT focuses on differentially private text analytics. Is it relevant to LLM's context?\n\n\n- The section on Federated learning is excessively long, with focus on models/tasks that are not relevant to LLM. Consider focusing directly on LLM. Same for Unlearning\n\n\n- Consider presenting tools that specifically focus on privacy-enhancing methods for LLM. For example this could be a table summarizing repositories to reproduce results, type of LLM supported, data modalities ...\n\n\n\n\n## Minor\n- I would suggest (Gentry 2009) as a reference for homomorphic encryption\n- For oblivious transfer, I would suggest (Michael O. Rabin, 1981), (Even, Goldreich, Lempel, 1985), or (Brassard, Crépeau, Robert 1986)\n- Page 19: \"Furthermore, referring to Figure 5\" => \"Furthermore, referring to Section 5\"", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4hwMo5HSuU", "reviewer": "Reviewer_Fx2K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Comprehensive detail of existing works in both the privacy-preserving and privacy-threatening field in LLMs.\n- Clear taxonomy and useful reference of existing tools and what they are used for (section 6).\n\n**Weaknesses**\n\n- The paper makes a distinction in section 2 that models such as BERT are not large enough to be considered LLMs, which are the focus of the survey paper. Yet the large portion of the mentioned literature are based on experiments around BERT-like masked language models with downstream classification tasks.\n- The paper claims that the main takeaway is that DP is the gold-standard for privacy protection due to its mathematical guarantee for privacy. However, the presentation of existing literature does not offer insight into how this gold-standard can be meaningfully met in the context of LLMs outside of a case by case approach given the many but scattered work of DP in language models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a comprehensive survey on the existing literature of privacy threats and potential solutions that exist in Large Language Models (LLMs). The authors start by introducing the threats to personal identifiable information in LLM training data and how it can be exposed via. data extraction, membership inference, and model inversion attacks. Then, the authors review the existing solutions in the field, and find they fall under two categories: directly applying privacy to the training data itself, or during the training methodology. Finally, the paper lists the many existing tools/frameworks and concludes that differential privacy (DP) is the benchmark/standard for privacy preserving in machine learning. The authors suggest that more work should be done applying DP to large scale start of the art LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- **(critical):** based on the large amounts of DP-related works in this space, are there any high-level insights or directions on how future research surrounding DP and LLMs can be practically implemented?\n- **(critical):** decrease amount of time/space spent on non-LLM models, in particular sections 3.2.1, 3.3.1, 5.3. Expand upon the works that deal with LLMs (like GPT-2) and use earlier works purely as guiding intuition.\n- (strengthen): Including paragraph headers in addition to sub-subsections. Some sub-sub-sections like 4.1.2 are dense collections of paragraphs without any pointers to the topic of each paragraph, which makes identifying specific points and insights harder for readers.\n- (strengthen): Since this survey is dedicated to large language models, sections 3.3, 5.3 seem a bit out of place in the context of its previous sections. As indicated in 5.3, federated learning is still in its infancy with large language models, which leads the survey to focus largely on transformers and BERT-like models. This detracts from the focus on the scale of LLMs. It might be beneficial to pull out these subsections into it’s own section in the end (or in the appendix), and mention that while distributed learning has not been widely studied with LLMs yet, this space is likely to face similar privacy concerns.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a survey article on broad areas of privacy preserving in LLMs. All reviewers lean towards acceptance, and think most of their concerns have been addressed. Some still have concerns around inadequate discussions of open questions and lack of scalability considerations. I agree with the reviewers that this submission can be published in TMLR with minor revision as follows.

* Add more discussions throughout the paper to highlight whether the mentioned tools/techniques/challenges/methods are unique with LLMs, or if they can be applicable to broad ML problems. Since this survey targets at LLMs, it’d be helpful to more clearly inform readers what is new with LLMs.

* Comment more on the scalability/efficiency properties of various measurements/approaches applied to large models.

* Figure 1: It is not clear why those papers are selectively highlighted in Figure 1 given that those areas have been studied extensively. The specific choices of papers seem random. I suggest changing paper citations to more fine-grained subareas or directions under those subtopics.

* It would be better to add a table of content to help with navigation.
Decision: Accept (poster)