Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning

Authors: lynn  chua
badih  ghazi
yangsibo  huang
pritish  kamath
ravi  kumar
daogao  liu
pasin  manurangsi
amer  sinha
chiyuan  zhang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 19:56:41

Abstract: Large language models (LLMs) have emerged as powerful tools for tackling complex tasks across diverse domains, but they also raise privacy concerns when fine-tuned on sensitive data due to potential memorization. While differential privacy (DP) offers a promising solution by ensuring models are “almost indistinguishable” with or without any particular privacy unit, current evaluations on LLMs mostly treat each example (text record) as the privacy unit. This leads to uneven user privacy guarantees when contributions per user vary. We therefore study user-level DP motivated by applications where it is necessary to ensure uniform privacy protection across users. We present a systematic evaluation of user-level DP for LLM fine-tuning on natural language generation tasks. Focusing on two mechanisms for achieving user-level DP guarantees, Group Privacy and User-wise DP-SGD, we investigate design choices like data selection strategies and parameter tuning for the best privacy-utility tradeoff.

Main paper: 
 introduction: 
 Modern large language models (LLMs) are shown to be capable of solving a wide range of complex understanding and reasoning tasks (Achiam et al., 2023; Anil et al., 2023; Bubeck et al., 2023) . Fueling their adoption, these pre-trained models can be further fine-tuned for specialized domains such as legal (Cui et al., 2023; Guha et al., 2023) , medical (Singhal et al., 2022; Luo et al., 2022; Moor et al., 2023; Lee et al., 2023) , and coding assistance (Chen et al., 2021; Li et al., 2023a; Roziere et al., 2023; Guo et al., 2024) . However, this proliferation of LLMs has introduced privacy concerns: Fine-tuning on domain data risks unintentionally retaining sensitive information, which could potentially be extracted later (Carlini et al., 2021) . Moreover, the trend towards using larger model sizes with more parameters (Kaplan et al., 2020) amplifies memorization risks (Carlini et al., 2022) .To address these privacy challenges, a growing body of work has focused on fine-tuning LLMs with differential privacy (DP) guarantees, a widely adopted notion to provide privacy guarantees in machine learning pipelines. At a high level, DP guarantees that a model trained with and without any particular data point is "almost indistinguishable". A training algorithm satisfying DP would ignore sensitive information of individual records but can still learn the distributional properties of the dataset. Typical DP training algorithms (e.g., DP-SGD, Abadi et al., 2016) provide such guarantees by limiting the contribution of each record to the trained model and adding calibrated noise.Existing studies have explored DP fine-tuning techniques for various applications, such as natural language understanding (Li et al., 2022; Yu et al., 2021; 2022) , generation (Yu et al., 2022; Anil et al., 2022; Yue et al., 2023) , and in-context learning (Tang et al., 2024; Wu et al., Figure 1 : The privacy unit is a crucial parameter for DP guarantees. Previous works commonly treat each training example or record as the unit of privacy (i.e., record-level DP, shown on the left). However, this leads to unequal privacy protection when users contribute varying numbers of records -users with more records unfortunately obtain weaker privacy guarantees under record-level DP. To address this discrepancy, we consider the stronger requirement of user-level DP (shown on the right) for use cases where it might be necessary or applicable. User-level DP ensures uniform privacy protection across all users, regardless of each individual's number of contributed records.2023; Duan et al., 2023) . Many training algorithms and evaluations in the literature make the (implicit) assumption that each user only contributes a single record to the training dataset. This notion is usually referred to as Example-level DP or Record-level DP. In the context of language models, a record is a sequence of (tokenized) text. However, in many realworld scenarios, each user may provide multiple records 1 . Under Example-level DP, users contributing more records inevitably obtain weaker privacy protection -an undesirable outcome (see Figure 1 ). To ensure uniform privacy protection regardless of the number of records per user, we consider the stronger requirement of User-level DP, which ensures that each user obtains the same privacy guarantee. This shift from Example-level DP to User-level DP forms the core focus of our study.We present a systematic empirical evaluation of User-level DP for fine-tuning LLMs on natural language generation tasks. Unlike existing relevant studies that mostly focus on the setting of (private) federated learning (McMahan et al., 2018; Kairouz et al., 2021; Xu et al., 2023) , we do not assume a distributed or federated setting. Instead, we consider the scenario where the language modeling task2 involves data from multiple users, and it is desirable to guarantee privacy at the user level, rather than the traditional record level. We identify and evaluate the two mainstream mechanisms for achieving user-level DP: Group Privacy and User-wise DP-SGD, as described in Section 2. We systematically propose and evaluate data selection strategies for both methods (Section 4 and Section 5) to improve the privacy-utility trade-off and compare their performance (Section 6). We hope the findings provide valuable empirical references for practitioners working on User-level DP for language modeling tasks.Furthermore, we provide a case study of applying the algorithm from Asi & Liu (2024) to our language model fine-tuning tasks. While many empirical studies focused on Examplelevel DP, the theory community has studied User-level DP extensively in the past few years, with a long line of algorithms with provable guarantees. Among those, Asi & Liu (2024) currently achieve the state-of-the-art excess rate (see Section 7 for more details). However, the theoretical results typically rely on assumptions such as that the data being drawn i.i.d. from an underlying distribution, and it has not been systematically evaluated whether those assumptions hold in practice. In this paper, we present the first such study, by focusing on the algorithm from Asi & Liu (2024) , which has the theoretically optimal excess rate. Our results show that the assumptions could be too restrictive for real world User-level DP applications, and call for the research community to consider more realistic assumptions.We note that an independent and concurrent work of Charles et al. (2024) also studies user-level DP for LLM fine-tuning. While their key observations are largely consistent with ours, we discuss some differences in Section 7.User-wise DP-SGD k records k records k records …Filtered data (size: # users x k)  
user-level differential privacy mechanisms: 
 Differential privacy (DP) (Dwork et al., 2006a; b) is a mathematical framework for ensuring the privacy of individuals in datasets. Formally, a randomized algorithm A satisfies (ε, δ)-DP if for any two neighboring datasets D and D ′ , and any subset S of outputs, it holds for privacy parameters ε ∈ R >0 and δ ∈ [0, 1) thatPr[A(D) ∈ S] ≤ e ε • Pr[A(D ′ ) ∈ S] + δ.The definition of "neighboring datasets" is determined by the scope of privacy. For Example-level DP, two datasets are neighboring if one can be obtained from the other by adding/removing one text record. For User-level DP, we assume each user has a (variable) number of records, and two datasets are neighboring if one can be obtained from the other by adding/removing all the text belonging to a particular user. Group Privacy. Thanks to the composition property of the DP guarantee, an Example-level DP algorithm can be directly turned into a User-level DP algorithm for the case where each user contributes at most k records. The simplest instantiation of such an approach says that if a mechanism M satisfies (ε, δ)-DP then for all k ≥ 1, the mechanism M ′ , that limits contribution from each user to be at most k (in an arbitrary manner), satisfies (kε, δ e kε-1 e ε -1 )-user-level-DP (see, e.g., Vadhan (2017)). However, tighter group privacy bounds are known for the specific case of DP-SGD (Ganesh, 2024) . 3 In Algorithm 1, we use GROUPPRIVACCOUNTING(k, ε, δ, M, B, T) to refer to any method that returns a σ such that DP-SGD with M records, batch size B, run for T iterations and with each user contributing at most k records, satisfies (ε, δ)-DP. We provide more details on the specific accounting method we use in Appendix B.3.User-wise DP-SGD. While Group Privacy can be applied to any underlying record-level DP algorithm, this flexibility comes at the potential cost of not utilizing the structure of the underlying algorithm directly. Here, we introduce a simple User-level DP algorithm, called User-wise DP-SGD, based on the classical DP-SGD algorithm (Abadi et al., 2016) for Examplelevel DP training. At each training step, User-wise DP-SGD samples a batch of users. For each user, it then samples several records, and computes the average gradients on those records4 . The (average) gradients from the batch of users are then norm-clipped, meanaggregated, and noised before passing to the optimizer. See Algorithm 2 for a formal algorithm description. We note the main modification of the original DP-SGD algorithm is user-wise sampling and clipping, which is similar to a combination of DP-SGD with FedAvg (McMahan et al., 2017) evaluated in some private federated learning tasks (McMahan et al., 2018; Kairouz et al., 2021) . In this paper, we do not assume a federated learning setting, and allow the learner to inspect all user data to devise more sophisticated sampling strategies. PRIVACCOUNTING(ε, δ, M, B, T) here refers to any method that returns a σ such that DP-SGD with M records, batch size B, run for T iterations, satisfies (ε, δ)-DP. Since Algorithm 2 treats "users" as "records", it suffices to simply replace M by N (total number of users) and B by n (number of users sampled in a batch); see Appendix B.3 for more details. Comparing to Group Privacy, User-wise DP-SGD does not need to preprocess the dataset to restrict that each user contributes at most k records. And we do not even need to assume each user's data consists of pre-defined records. For long form text, we can randomly sample contiguous chunks during training, creating more diverse records.We also evaluate the algorithm in Asi & Liu (2024) (see Algorithm 3) that is based on User-wise DP-SGD with a more involved gradient estimation sub-procedure. In the sub-procedure, the algorithm detects if the average gradients for the users in the batch are well-concentrated, in other words, if they are close enough to each other in ℓ 2 -distance. If most of the gradients are well-concentrated, the outliers are removed, and the remaining gradients are meanaggregated and noised. It is shown that less noise is needed in this case. In the other case, if there is no strong concentration, the algorithm halts and refuses to output anything meaningful. Under the assumption that all the records are i.i.d. from some underlying distribution, this algorithm can provably achieve the optimal privacy-utility trade-off. 
experimental setup: 
 Datasets. Our evaluation uses the Enron Email dataset (Klimt & Yang, 2004 ) and the BookSum dataset (Kryści ński et al., 2022) .• The Enron Email dataset (Klimt & Yang, 2004) consists of approximately 500,000 emails generated by employees of the Enron Corporation. As these emails originate from real-world users and cover diverse topics, they may also contain privacysensitive information such as personally identifiable data, making this dataset a Enron Email BookSum (Klimt & Yang, 2004) ( There is no established consensus on the optimal method for sampling records to train User-level DP mechanisms, whether using Group Privacy or User-wise DP-SGD. Therefore, in the following sections, we conduct a comprehensive evaluation exploring two key factors in this selection: (i) the record selection criteria, and (ii) the number of records to select for each privacy unit. Please see Appendix B for the detailed experimental setup and Appendix C.4 for the analysis of the computational overhead. 
group privacy: 
 In this section, we evaluate Group Privacy for User-level DP and explore design choices for selecting records for each user, including the data selection strategy (Section 4.1) and the number of records to be selected (Section 4.2).We compare the following five methods for selecting records for each privacy unit:• Random: Randomly select k records from each privacy unit. • Longest: Select k records with the longest record length from each privacy unit.ε = 1.0 ε = 3.0 ε = 8.• Shortest: Select k records with the shortest record length from each privacy unit.• Highest Perplexity (PPL): Select k records with the highest perplexity, i.e., records that are most surprising to the (initial pre-trained) model. • Lowest Perplexity (PPL): Select k records with the lowest perplexity, i.e., records that are least surprising to the (initial pre-trained) model.We fix k = 10 for both datasets and report the results in Table 2 .Selection strategy matters for Enron Emails, but not for BookSum. On the Enron Email dataset (Table 2a ), selecting the longest records from each user yields the best performance across all privacy levels. This suggests that longer email records tend to be more informative and representative of each user's data, leading to better utility after private fine-tuning. For the BookSum dataset (Table 2b ), the performance differences across selection methods are relatively small. Random selection performs best for ε = 1.0 and 8.0, while selecting records with the lowest perplexity (least surprising to the model) works slightly better for ε = 3.0.In general, the results indicate that simple heuristics like selecting the longest or shortest records can be effective strategies, sometimes outperforming more complex criteria like perplexity-based selection. The performance gap between different selection methods is more pronounced on the Enron Email dataset compared to BookSum. This could be due to the greater variation in record lengths and content in the email data versus book summary data (see Table 7 in Appendix B).The number of records (k) selected for each privacy unit is another important factor when applying group privacy. In group privacy, increasing k makes both the noise and signal stronger, with opposing impact on the model's performance. On the one hand, a larger k would result in higher noise due to the increased sensitivity of the privacy mechanism. On the other hand, using a larger k also results in more diverse data being used for training, which can potentially improve the model's utility. Therefore, determining the optimal value of k for utility is a trade-off between the increased noise and the diversity of the training data.ε = 1.0 ε = 3.0 ε = 8.0Optimal k balances noise from privacy and training data diversity. To investigate the impact of k, we fix the data selection strategy to the random selection method and vary the value of k. We observe in Table 3 that for lower privacy budgets (i.e., ε = 1.0 or 3.0), using a smaller k is generally better, as the undesirable increase in noise when k increases outweighs the benefits of diverse data. However, for larger ε, using a larger k is usually preferred, as the high privacy budget can potentially tolerate slightly more noise being introduced.In such cases, the diversity of the training data becomes more beneficial for larger k, as evidenced by lower perplexity for larger k's when ε = ∞ (non-private setting)6 . We also observe similar results on GPT-2 medium and large models, as detailed in Appendix C.3. 
user-wise dp-sgd: 
 In this section, we evaluate User-wise DP-SGD and explore design choices for selecting records for each user. We discuss the data selection strategy (Section 5.1) and the number of records to be selected (Section 5.2). We also analyze the applicability of the advanced User-wise DP-SGD method proposed by Asi & Liu (2024) based on a more advanced gradient estimation sub-procedure.Unlike Group Privacy that requires preprocessing to limit each user's contribution to k records, User-wise DP-SGD imposes no such constraint. This allows exploring data selection strategies beyond record boundaries. Therefore, in addition to the five selection strategies discussed in Section 4.1, we also explore a method called Random Chunk. Specifically, for each user, we concatenate all their records into a single document, and randomly fetch a chunk of the maximum record length (as defined by the model) from it.As shown in Table 4 , Random Chunk consistently outperforms other methods across datasets and privacy budgets ϵ. This benefit likely stems from increased data diversity during training and the ability to span record boundaries.User-wise DP-SGD favors larger k, with diminishing returns. In contrast to Group Privacy (Section 4.2), increasing k monotonically improves User-wise DP-SGD's utility by providing more diverse data per user. However, larger k also incurs slightly higher computation (Appendix C.5). The benefit diminishes as k grows, so for practical deployment, choosing a sufficiently large k will balance utility and efficiency. For instance, on the BookSum dataset, k = 10 appears to achieve the best trade-off between these two factors.k = 2 k = 5 k = 10 ε = 1.0As discussed in Section 2, Asi & Liu (2024) proposed a User-wise DP-SGD algorithm that achieves the state-of-the-art excess rate and requires injecting less noise if per-user averaged gradients in a batch exhibit low variance. Specifically, recall that User-wise DP-SGD has a clipping norm parameter C, and the noise added is proportional to C. Asi & Liu (2024) showed that when most of the gradients are within ℓ 2 -distance τ of each other, adding noise proportional to τ suffices for privacy purposes. Under the assumption that the records are drawn i.i.d. from some underlying distribution, Asi & Liu (2024) proved that this mechanism can achieve the optimal rate.However, it remains unclear how applicable this algorithm, and other such theoretically motivated ones, are to real-world problems where the i.i.d. assumption might not be true, and the number of records per user is relatively small (e.g., k ≤ 50). In order to answer this question, we measure the noise levels of standard User-wise DP-SGD and Asi & Liu (2024)'s method under different values of τ, controlling the required gradient concentration. Figure 3 shows that for most k and ϵ, Asi & Liu (2024)'s noise is lower than User-wise DP-SGD's only if τ/C < 0.1, making it highly unlikely to enter the advanced noising step (lines 10-15 in Algorithm 3). Despite its potential benefits, this method relies on high user gradient concentration, which does not hold in our settings. Asi & Liu (2024) to yield lower noise than standard User-wise DP-SGD, the ratio between the concentration factor τ and the clipping norm C must be smaller than 0.1. However, in the evaluated datasets, τ/C is around 1. Figure 7 presents results for different ε's. 
comparing group privacy and user-wise dp-sgd: 
 Finally, we compare the performance of Group Privacy and User-wise DP-SGD under the same privacy budget. For Group Privacy, we use Longest as its data selection strategy, and for each ε, we use its corresponding best k, as reported in Table 3 . For User-wise DP-SGD, we use Random Chunk for data selection, and use k = 10, as it achieves the best trade-off between utility and efficiency. We vary the other hyperparameters, including learning rate, batch size, number of training iterations, and the clipping norm, and report the best results for each mechanism in Table 6 .User-wise DP-SGD outperforms Group Privacy. At the mechanism level, User-wise DP-SGD consistently outperforms Group Privacy, especially for smaller privacy budgets. This is potentially due to User-wise DP-SGD allowing for more diverse data being selected in different minibatches during training, while Group Privacy already constrains the training data to be a subset of user records during the preprocessing stage. Additionally, for some users who have fewer than k records, our implementation of Group Privacy will perform resampling to obtain k records. This repetition may also negatively impact the model's performance.We acknowledge this as a potential limitation of our evaluation that could be addressed in future work. However, it is worth noting that Group Privacy requires minimal changes to implement on top of existing record-level private training, whereas User-wise DP-SGD may require modifications to the data loading pipeline. In this regard, practitioners may need to consider the implementation difficulty when choosing between these two methods.Sensitivity to clipping norms. We specifically investigate the sensitivity of both mechanisms to clipping norms, as reported in Figure 4 . As shown, Group Privacy is generally more sensitive to clipping norms compared to User-wise DP-SGD. This is because the clipping operation is performed on the gradient of each individual record for the Group Privacy method (see Algorithm 1). In contrast, for User-wise DP-SGD, the clipping is applied to the averaged gradients, which ideally exhibit less variance, making it less sensitive to clipping norms (see Algorithm 2). 
related work: 
 Record-level Differential Privacy. Differential privacy (DP) (Dwork et al., 2006b; a) is a widely adopted framework that formally bounds the privacy leakage of individual user information while still enabling the analysis of population-level patterns. When training deep neural networks with DP guarantees, the go-to algorithm is Differentially Private Stochastic Gradient Descent (DP-SGD), introduced in the seminal work of Abadi et al. (2016) . This algorithm operates by clipping the contributions of gradients on a per-record basis and injecting noise into the average gradient updates during each iteration of SGD.Previous research has applied DP-SGD for language models (Ponomareva et al., 2023) , primarily focusing on the record level. These applications include work in natural language understanding (Li et al., 2022; Yu et al., 2021; 2022) , natural language generation (Anil et al., 2022) , in-context learning (Tang et al., 2024; Wu et al., 2023; Duan et al., 2023) , prompt tuning (Li et al., 2023b) , synthetic text generation (Al Aziz et al., 2021; Yue et al., 2023) , and retrieval-augmented language models (Huang et al., 2023) .User-level Differential Privacy. McMahan et al. (2018) ; Kairouz et al. (2021) studied userlevel DP training of LSTM based language models in the federated learning setting. In contrast, we consider a general central training setting for more flexible samplers. Moreover, we studied modern transformer-based language models. More broadly, several fundamental and important problems have been actively studied under user-level DP in recent years, e.g., machine learning (Wang et al., 2019) , learning discrete distributions (Liu et al., 2020) , histogram estimation (Amin et al., 2019) , Structured Query Language (SQL) (Wilson et al., 2020) , number of users required with multiple items per user (Ghazi et al., 2021) , stochastic convex optimization (Levy et al., 2021; Bassily & Sun, 2023; Ghazi et al., 2023a; Asi & Liu, 2024) and reduction from item-level to user-level DP (Ghazi et al., 2023b) . As mentioned, a common assumption in the theoretical results above is that the data is drawn i.i.d. from an underlying distribution. This assumption is too restrictive, and it would be interesting to propose more realistic assumptions under which we can still gain some theoretical insights.We note that an independent and concurrent work of Charles et al. (2024) also examines the empirical performance of user-level DP methods for fine-tuning LLMs. The key findings that User-wise DP-SGD usually outperforms Group Privacy in terms of utility are generally consistent between the two studies. Notably, their work provides valuable insights by offering an explicit methodology for selecting the group size in both mechanisms, whereas ours mainly performs ablations of various sizes. Moreover, our work differs from theirs in that we explore a broader design space for data selection strategies under various User-level DP mechanisms. In this expanded exploration, we observe substantial performance gaps between suboptimal and better data selection methods. In addition, their evaluation focuses on the larger StackOverflow and news datasets using a single 350M-parameter model while our work utilizes books and email datasets and considers models of various sizes. We also provide case studies for the User-level DP algorithm with the state-of-the-art excess rate. 
conclusions: 
 In this work, we study language model fine-tuning with DP guarantees. Since each user generally contributes multiple text records to the training corpus, to ensure equal privacy guarantees across users, the privacy unit needs to be set at the user level. We identify and systematically evaluate two mechanisms, Group Privacy and User-wise DP-SGD, for achieving user-level DP when fine-tuning large language models on natural language generation tasks. For each mechanism, we investigate data selection strategies to improve the trade-off between privacy and utility.While our work provides valuable insights, we acknowledge a few limitations and suggest directions for future research. First, in our implementation of User-wise DP-SGD, we fixed the value of k (the number of gradients to sample per user) for simplicity. Exploring more dynamic choices of k for different users may yield better utility while preserving privacy guarantees. Second, since User-wise DP-SGD can perform data selection on the fly, it will be interesting to investigate how other data selection methods, such as those based on gradient information or active learning, might impact the utility. Finally, while we focus on language generation, user-level DP has broader applications motivating analysis in domains like recommendation systems and structured prediction. 
 
 Appendix: We describe the algorithm in Asi & Liu (2024) using our notation for completeness. They assume each user owns the same amount of record and use the full batch size (use all dataset in each step). We modify the algorithm by drawing k ′ i records for the ith user in the following Pseudocode. g t,i ← 1 k ′ i ∑ j∈[k ′ i ] ∇ θ t ℓ(θ t , (x j , y j ) 7:end for /* Compute the concentration scores */ 8:Compute EQUATIONData preprocessing for the Enron Email dataset. To prepare the Enron Email dataset (Klimt & Yang, 2004) for our evaluation of user-level DP implementations, we first parse each record, into DATE, FROM (sender), TO (receiver), SUBJECT, and BODY, using regular expressions. Note that for each of the "CC" or "BCC" addresses on the email, we treat it as a new receiver. Specifically, we will create a new record with the same (DATE, FROM, SUBJECT, BODY), but a different TO (i.e., receiver). We then perform in-email deduplication to attain a dataset with higher quality: We only keep records whose maximum repetition of any 20-gram is 1. Forwarded emails contain the email body of the forwarded messages; this complicates the privacy accounting process. Therefore, for simplicity, we remove forwarded emails. After preprocessing, the final dataset contains 240, 173 emails.Distribution of number of records per privacy unit. Figure 5 presents the distribution of number of records per privacy unit in both datasets. Statistics after data selection. For reference, Table 7 reports some statistics of the selected records after applying different data selection methods discussed in Section 4.1.Our experiments primarily use the pretrained GPT-2 small model (Radford et al., 2019) , which has 125 million trainable parameters. To assess the generalizability of our findings, we also present results with GPT-2 models of different sizes in Appendix C.3. For each evaluated dataset, we fine-tune the model for the language modeling task, using LoRA (Hu et al., 2022) with a rank of 32. As shown in Appendix C.1, LoRA does not significantly impact model performance. Model performance is measured using perplexity, a standard evaluation metric for language models.We perform privacy accounting assuming that the batches are constructed using Poisson subsampling, where each batch is independently sampled by including each record / user independently with a certain probability, even though our implementation involves constructing batches by going through users/records in a random order. We note that this assumption has been a common practice in the privacy accounting of DP-SGD, starting with the work of Abadi et al. (2016) . Recent work by Chua et al. (2024) points out that there can be substantial gaps between the privacy guarantees of DP-SGD when using Poisson subsampling vs. shuffling; however, since we make the Poisson subsampling assumption & Liu (2024) to yield lower noise than standard User-wise DP-SGD, the ratio between the concentration factor τ and the clipping norm C must be smaller than 0.1. However, in the evaluated datasets, τ/C is around 1.For instance, patients contributing various medical records to train healthcare AI assistants, or users submitting different prompts/conversations to an open-domain chatbot.We focus on natural language generation tasks, which underpin many key LLM applications.An earlier version of our work used the naive group privacy bound. However, followingCharles et al. (2024), we have updated our approach to use the tighter accounting; for details see Appendix B.3.The number of records sampled for each user does not need to be the same for different users. However, for ease of implementation, we keep them the same in our experiments.We note that the choice of DP fine-tuning recipe is orthogonal to the user-level DP algorithms, as the latter can be seamlessly combined with any DP fine-tuning pipeline. We show in Appendix C.2 that we observe consistent results with a different fine-tuning method(Li et al., 2022).Note that for both methods, we report results for ε = ∞ following their corresponding algorithms in Algorithm 1 and Algorithm 2 while bypassing the stage of aggregation and noise addition. 


Decision: Existing evaluations of privacy-preserving LLMs often treat each example (text sequence) as the privacy unit, leading to uneven privacy guarantees when contributions per user vary. To address this, the authors study user-level differential privacy (DP), driven by the need for uniform privacy protection across users. They conduct a systematic evaluation of user-level DP for LLM fine-tuning on natural language generation tasks under two mechanisms for achieving user-level DP guarantees—Group Privacy and User-wise DP-SGD. They investigate design choices such as data selection strategies and parameter tuning to optimize the privacy-utility tradeoff. Following the author response and author-reviewer discussions, this paper has received unanimous support from the reviewers. Therefore, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: In this work, the authors study language model fine-tuning with differential privacy guarantees. Since each user generally contributes multiple text sequences to the training corpus, to ensure equal privacy guarantees across users, the privacy unit needs to be set at the user level. The authors identify and systematically evaluate two mechanisms, Group Privacy and User-wise DP-SGD, for achieving user-level differential privacy when fine-tuning large language models on natural language generation tasks. For each mechanism, the authors investigate data selection strategies to improve the trade-off between privacy and utility.
strength_weakness: strength_weakness: 1. The technical novelty is limited. What is the challenge to adapt User-wise DP-SGD in LLM, there are many existing User-wise DP-SGD methods.

2. Lack of baselines. It is better to add some recent baseline to demonstrate the advantage of the proposed method.  For example: Large Language Models Can Be Strong Differentially Private Learners. 2022 ICLR
questions: questions: see weakness.
paper_summary: paper_summary: In this paper, the authors present a study on the application of user-level differential privacy (DP) in the context of fine-tuning large language models (LLMs) on natural language generation tasks.  It is a critical issue in the field of machine learning, specifically the privacy concerns associated with fine-tuning LLMs on sensitive data. The document is well-structured, with a clear abstract, introduction, methodology, experimental setup, results, and conclusions. Experimental results on Enron Email dataset and BookSum dataset were given to show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The paper mentions that User-wise DP-SGD may require modifications to the data loading pipeline, which could be a barrier to adoption in practice.

2. The computational overhead introduced by the differential privacy mechanisms is mentioned but not thoroughly analyzed, particularly for larger models or datasets.
paper_summary: paper_summary: The paper delves into the issue of privacy concerns when fine-tuning large language models (LLMs) on sensitive data. The paper introduces the concept of User-level Differential Privacy (DP) as a solution to ensure uniform privacy protection across users, especially in scenarios where contributions per user vary. By focusing on two mechanisms, Group Privacy and User-wise DP-SGD, the study evaluates the design choices such as data selection strategies and parameter tuning to strike a balance between privacy and utility.
Overall, the paper provides a new perspect of user-wise DP for LLM fine-tuning on natural language generation tasks, shedding light on the importance of considering user privacy guarantees in addition to example-level privacy. However, the solution is somehow lacking of novelty. Moreover, the comparison of proposed approach is not thorough.
strength_weakness: strength_weakness: 1). The solution is somehow straightforward building upon the existing groupwise DPSGD which limits the contribution of the submission.

2).  the comparison of proposed approach is not thorough. Many other DP approaches are not compared such as: LARGE LANGUAGE MODELS CAN BE STRONG DIFFERENTIALLY PRIVATE LEARNERS
https://openreview.net/pdf?id=bVuP3ltATMz
To name a few.

3). Typos: Page 3: a Example-level==>an Example-level

4). The method needs the present/availablility of user info, which might be missing in many real applications.
questions: questions: 1). More thorugh comparisons to other SOTAs of DP.

2). What if the user info is not available for different samples?

Comments: title: Thank you
Thanks for the response
Follow-up
Follow-up
Thank you for your response
comment: Thank you for taking the time to read our response and for raising your score. 

If you have any further questions or need additional clarification, please feel free to reach out.
Thanks for the informative response. I've changed my rating.
Dear Reviewer pxQP,

Thank you once again for taking the time to share your suggestions on our work. As the discussion period is nearing its end, we want to ensure that we have adequately addressed all your comments. Specifically, do you still have concerns regarding:
- Q1: The implementation of user-wise DP-SGD, where we clarify that it **can be implemented with minimal effort**.
- Q2: The computational overhead, where we report the training time for your reference and observe **minimal computational overhead** (4%–9%) compared to non-private training for the GPT-2 model of various sizes.

We look forward to your feedback.

Best regards, 

Authors of Submission933
Dear Reviewer sVEv,

Thank you once again for taking the time to share your suggestions on our work. As the discussion period is nearing its end, we want to ensure that we have adequately addressed all your comments. Specifically, do you still have concerns regarding:
- Q1: Our contribution, where we contribute a **comprehensive evaluation suite of user-level DP implementations for LLMs and well-motivated data selection strategies**.
- Q2: Performance when integrated with other DP fine-tuning methods, where we present **new results for our evaluation when incorporated into the DP fine-tuning pipeline you suggested**.
- Q3: The availability of user information in real applications, where we adhere to the **standard user-level DP notion** that can still function as long as there is a mechanism to group examples based on user ownership.

We look forward to your feedback.

Best regards,

Authors of Submission933
Thank you for taking the time to read our response and for raising your score. We're glad to hear that our response addressed most of your concerns. 

If you have any further questions or need additional clarification, please feel free to reach out.
Thanks for the response. It addressed most of my concerns. I will raise my score.