Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning
Authors: Lynn Chua
Badih Ghazi
Yangsibo Huang
Pritish Kamath
Ravi Kumar
Daogao Liu
Pasin Manurangsi
Amer Sinha
Chiyuan Zhang
Venue: COLM (2024)
Keywords: Privacy, User-level privacy, Differential privacy
TL;DR: We present a systematic evaluation of user-level differential privacy for LLM fine-tuning on natural language generation tasks

Abstract: Large language models (LLMs) have emerged as powerful tools for tackling complex tasks across diverse domains, but they also raise privacy concerns when fine-tuned on sensitive data due to potential memorization. While differential privacy (DP) offers a promising solution by ensuring models are “almost indistinguishable” with or without any particular privacy unit, current evaluations on LLMs mostly treat each example (text record) as the privacy unit. This leads to uneven user privacy guarantees when contributions per user vary. We therefore study user-level DP motivated by applications where it is necessary to ensure uniform privacy protection across users. We present a systematic evaluation of user-level DP for LLM fine-tuning on natural language generation tasks. Focusing on two mechanisms for achieving user-level DP guarantees, Group Privacy and User-wise DP-SGD, we investigate design choices like data selection strategies and parameter tuning for the best privacy-utility tradeoff.

Published as a conference paper at COLM 2024
Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning
Lynn Chua
Badih Ghazi
Yangsibo Huang Princeton University yangaibo@google.cmm
Google Research chualynn@google.com
Google Research
Mudihghazi@gmail.coo
Pritish Kamath
Ravi Kumar
Daogao Liu University Washington dgliu@uw.edu
Google Research
Google Research
pritishk@google.com
ravi.k33@ crnos30gmmalcoom com
Pasin Manurangsi
Amer Sinha
Chiyuan Zhang
Google Research
Google Research
Google Research
pasin@google.com
amersinha@google.com
chiyuan@google.com
Abstract
Large language models (LLMs) have emerged as powerful tools for tackling complex tasks across diverse domains, but they also raise privacy concerns when fine-tuned on sensitive data due to potential memorization. While differential privacy (DP) offers a promising solution by ensuring models are "almost indistinguishable" with or without any particular privacy unit, current evaluations on LLMs mostly treat each example (text record) as the privacy unit. This leads to uneven user privacy guarantees when con- tributions per user vary. We therefore study user-level DP motivated by applications where it is necessary to ensure uniform privacy protection across users. We present a systematic evaluation of user-level DP for LLM fine-tuning natural language generation tasks. Focusing two mecha- nisms for achieving user-level DP guarantees, Group Privacy and User-wise DP-SG, investigate design choices like data selection strategies and parameter tuning for the best privacy-utility tradeoff.
1 Introduction
Modern large language models (LLMs) are shown to capable of solving a wide range of complex understanding and reasoning tasks (Achiam et al., 2023; Anil et al., 2023; Bubeck et al., 2023). Fueling their adoption, these pre-trained models can further fine-tuned for specialized domains such legal (Cui et al., 2023; Guha et al., 2023), medical (Singhal et al., 2022; Luo et al., 2022; Moor et al., 2023; Lee et al., 2023), and coding assistance (Chen et al., 2021; Li et al., 2023a; Roziere et al., 2023; Guo et al., 2024). However, this proliferation of LLMs has introduced privacy concerns: Fine-tuning on domain data risks unintentionally retaining sensitive information, which could potentially be extracted later (Carlini al., 2021). Moreover, the trend towards using larger model sizes with more parameters (Kaplan et al. 2020) amplifies memorization risks (Carlini al., 2022).
To address these privacy challenges, growing body of work has focused on fine-tuning LLMs with differential privacy (DP) guarantees, widely adopted notion to provide privacy guarantees in machine learning pipelines. At a high level, DP guarantees that a model trained with and without any particular data point is "almost indistinguishable.. A training algorithm satisfying DP would ignore sensitive information of individual records but can still learn the distributional properties the dataset. Typical DP training algorithms (e.g., DP-SGD, Abadi et al., 2016) provide such guarantees by limiting the contribution of each record to the trained model and adding calibrated noise.
Existing studies have explored DP fine-tuning techniques for various applications, such as natural language understanding (Li et al., 2022; Yu et al., 2021; 2022), generation (Yu et al., 2022; Anil et al., 2022; Yue et al., 2023), and in-context learning (Tang et al., 2024; Wu et al.,

---

Published as a conference paper at COLM 2024
Users w/ more records have weaker privacy guarantees
Each user has the same privacy garrnttee
User-level DP mechanism
100 records
records
40 records
records
70 records
Model
records
Model
Data
Data
Record-evvl Differential Privacy (DP)
User-level Differential Privacy (DP)
The privacy unit is a crucial parameter for DP guarantees. Previous works commonly treat each training example or record as the unit of privacy (i.e., record-level DP, shown on the left). However, this leads to unequal privacy protection when users contribute varying numbers of records users with more records unfortunately obtain weaker privacy guarantees under record-level DP. To address this discrepancy, we consider the stronger requirement of user-level DP (shown the right) for use cases where it might be necessary or applicable. User-level DP ensures uniform privacy protection across all users, regardless of each individual's number of contributed records.
Figure 1:
2023; Duan et al., 2023). Many training algorithms and evaluations in the literature make the (implicit) assumption that each user only contributes a single record to the training dataset. This notion is usually referred to as Example-level DP or Record-level DP. In the context of language models, a record is a sequence of (tokenized) text. However, in many real- world scenarios, each user may provide multiple records Under Example-level DP, users contributing more records inevitably obtain weaker privacy protection an undesirable outcome (see Figure 1). To ensure uniform privacy protection regardless of the number of records per user, we consider the stronger requirement of User-level DP, which ensures that each user obtains the same privacy guarantee. This shift from Example-level DP to User-level DP forms the core focus of our study.
We present systematic empirical evaluation of User-level DP for fine-tuning LLMs on natural language generation tasks. Unlike existing relevant studies that mostly focus on the setting of (private) federated learning (McMahan et al., 2018; Kairouz et al., 2021; Xu et al., 2023), we do not assume a distributed or federated setting. Instead, we consider the scenario where the language modeling task2 involves data from multiple users, and it is desirable to guarantee privacy at the user level, rather than the traditional record level. We identify and evaluate the two mainstream mechanisms for achieving user-level DP: Group Privacy and User-wise DP-SGD, as described in Section 2. We systematically propose and evaluate data selection strategies for both methods (Section 4 and Section 5) to improve the privacy-utiling trade-off and compare their performance (Section 6). We hope the findings provide valuable empirical references for practitioners working on User-level DP for language modeling tasks.
Furthermore, we provide case study of applying the algorithm from Asi & Liu (2024) to our language model fine-tuning tasks. While many empirical studies focused on Example- level DP, the theory community has studied User-level DP extensively in the past few years, with long line of algorithms with provable guarantees. Among those, Asi & Liu (2024) currently achieve the state-of-the-art excess rate (see Section for more details). However, the theoretical results typically rely on assumptions such as that the data being drawn i..d. from an underlying distribution, and it has not been systematically evaluated whether those assumptions hold in practice. In this paper, we present the first such study, by focusing on the algorithm from Asi & Liu (2024), which has the theoretically optimal excess rate. Our results show that the assumptions could be too restrictive for real world User-level DP applications, and call for the research community to consider more realistic assumptions.
We note that an independent and concurrent work of Charles et al. (2024) also studies user-level DP for LLM fine-tuning. While their key observations are largely consistent with ours, we discuss some differences in Section? 7.
For instance, patients contributing various medical records train healthcare Al assistants, or users submitting different prompts/ conversations to an We focus natural language generation tasks, which underpin many key I.I.M applications.
chatbot.
2

---

Published as a conference paper at COLM 2024
Training data (size:all)
Filtered data (size: compute gradients
Sample records
Training data (size: all)
Sample K' records per user & compute
User-wise gradient averaging
records
Rrssores
Gradient clipping add noise
N, records
K records
Gradient clipping add noise
N
Rg records di R
N, records
Group Privacy
User-wise DP-SGD
Illustration of how Group Privacy (left) and User-wise DP-SGD (right) preprocess training data, sample records, and calculate gradients at each training step.
Figure 2:
User-level Differential Privacy Mechanisms
Differential privacy (DP) (Dwork et al., 2006a;b) is mathematical framework for ensuring the privacy of individuals in datasets. Formally, a randomized algorithm A satisfies (e, DDDP if for any two neighboring datasets D and D and any subset S of outputs, it holds for privacy parameters e IRso and d E [0, 1) that
Pr(A(D) E S) < d E E +6
The definition of "neighboring datasets" is determined by the scope privacy. For Example-level DP, two datasets neighboring if odding/removee one text record. For User-level DP, we assume each user has a (variable) number of records, and two datasets are neighboring if can be obtained from the other by adding/ feiinn/removvinggg all the text belonging to particular user.
be obtained from the other by
Algorithm 1: Croup Privacy
Algorithm 2: User-wise DP-SGD
1: Input:
Initial model weights d, training of N users and M records, learning rate Y iterations Т, batch size B, privacy budget number of records per user gradient norm bound C
1: Input:
Initial model weights 6. training D of users records, learning rate y iterations T, user batch size n, privacy budget number of records per user (K) gradient norm bound
Stage Compute noise multiplier
/* Stage Compute noise multiplier
2: 0 + GROUPPRIVACCOUNIINGG
e, 6,M, 2: + PRIVACCOUNTINGIE, 6, N,
/* Stage 2. Prepare the dataset */
/* Stage 2. Prepare the dataset */
3: D' <- Ø
3: (User-wise DP-SGD directly uses D)
4: 1 1,..., N do
/* Stage Training */
Sample { records for the ith
4: for 1 do
user
5: Randomly draw 24, batch of n users
6: D' < D' (xy
6: for = 1, 1,..., do
7: end for
7:
Sample (zy.V; K/ records for the ith user in 24;
/- Stage 3. Training */
8: 1,.., do
/* Compute aver-ave gradient */
Randomly draw from D', a batch of records
8:
Bi + K (EjER1 (o,e(0, (xy.v))
/+ Clip gradient +/
10:
lorj=1,......
ê. + -//// max (1, Bulll)
11:
81,j - Vg, (0, (x,31)
9:
/ Clip gradient
10:
end for
Br, /Bt// max (1, (
/* Aggregate and add noise +/
12:
11:
8. + [Ej|m] + (
13:
end for
/* Aggregate and add noise +/
/+ Model update +/
14:
& 8 [ Bij +N(1,
0,41 <- 9, 1/:
13: end for
/* Model update +/
15:
0.11 + 9: - 1/8:
16: end for
3

---

Published as a conference paper at COLM 2024
Group Privacy. Thanks to the composition property of the DP guarantee, Example-level DP algorithm can be directly turned into a User-level DP algorithm for the case where each user contributes at most k records. The simplest instantiation of such an approach says that if a mechanism M satisfies (e, (e,P) then for all k 2 1, the mechanism M', that limits contribution from each user to be at most l (in an arbitrary manner), satis- fies (ke,ō" k T -user-level-DP (see, e.g., Vadhan (2017)). However, tighter group privacy bounds are known for the specific case of SP (Ganesh, 2024).3 In Algorithm 1, we use 6, M, B, T) to refer to any method that returns such that DP-SGD with M records, batch size B, run for T iterattoons and with each user contributing at most k records, satisfies (e, - We provide more details on the specific accounting method we use in Appendix B.3.
User-wise DP-SGD. While Group Privacy can be applied to any underlying record-level DP algorithm, this flexibility comes at the potential cost of not utilizing the structure of the underlying algorithm directly. Here, we introduce a simple User-level DP algorithm, called User-wise DP-SCD, based on the classical DP-SGD algorithm (Abadi et al., 2016) for Example- level DP training. At each training step, User-wise DP-SGD samples a batch of users. For each user, it then samples several records, and computes the average gradients on those records The (average) gradients from the batch of users are then norm-clipped, mean- aggregated, and noised before passing to the optimizer. See Algorithm for a formal algorithm description. We note the main modification of the original DP-SGD algorithm is user-wise sampling and clipping, which is similar to a combination of DP-SCD with FedAvg (McMahan et al., 2017) evaluated in some private federated learning tasks (McMahan et al., 2018; Kairouz et al., 2021). In this paper, we do not assume a federated learning setting, and allow the learner to inspect all user data to devise more sophisticated sampling strategies. PRIVACCOUNTING that DP-SGD with M records, batch size B, run for T iterations, satisfies (e, (eP Since Algorithm 2 treats "users" as "records", it suffices to simply replace M by N (total number of users) and B by (number of users sampled in a batch); see Appendix B.3 for more details. Comparing to Group Privacy, User-wise DP-SCD does not need to preprocess the dataset to restrict that each user contributes at most records. And we do not even need to assume each user's 's data consists of pre-defined records. For long form text, we can randomly sample contiguous chunks during training, creating more diverse records.
8, M, B, T) here refers to any method that returns 0 such
We also evaluate the algorithm in Asi & Liu (2024) (see Algorithm 3) that is based on User-wise DPD with a more involved gradient estimation sub-procedure. the sub-procedure, the algorithm detects if the average gradients for the users in the batch are well-concentrrated, in other words, they are close enough to each other in di-itttncc If most of the gradients are well-concentratee, the outliers are removed, and the remaining gradients aggregated and noised. It is shown that less noise is needed in this case. In the other case, if there is no strong concentration, the algorithm halts and refuses to output anything meaningful. Under the assumption that all the records are i.i.d. from some underlying distribution, this algorithm can provably achieve the optimal privacy-utility trade-off.
mean-
3 Experimental Setup
Datasets.
Our evaluation uses the Enron Email dataset (Klimt & Yang, 2004) and the BookSum dataset (Kryšcifiski et al., 2022).
The Enron Email dataset (Klimt& & Yang, 2004) consists of approximately 500,000 emails generated by employees of the Enron Corporation. As these emails originate from real-world users and cover diverse topics, they may also contain privacy- sensitive information such as personally identifiable data, making this dataset
AA earlier version of our work used the naive group privacy bound. However, following Charles al. (2024), we have updated our approach to use the tighter accounting: for details see Appendix B.3. "The number of records sampled for each user does not need to be the same for different users. However, for ease of implementation, we keep them the same in our experiments.
4

---

Published as a conference paper at COLM 2024
Enron Email (Klimt & Yang, 2004)
BookSum (Kryšciriski et al., 2022)
# records for training # records for evaluation
240, 173 39,086
9, 6,0 1,431
Avg. # records per privacy unit
13.2
52.2
Max # records per privacy unit Min # records per privacy unit Median records per privacy unit Choice of k
5, 0.94
266
1
2.0
28.5
{2,5,
52,
Table 1: Sequence-level and privacy unit level dataset statistics for the Enron Email and BookSum datasets.
suitable approximation of real-world user-level DP applications. We preprocess the dataset following the details provided in Appendix B.
The BookSum dataset (Kryšciñskki al., 2022) contains human-written summaries of various books. Books and other literary works are typically protected by copy- right laws, and training models on copyrighted material without permission from the copyright holders could raise legal issues (Henderson et al., 2023). While the summaries in the BookSum dataset may be considered transformative fair use of copyrighted books, applying DP training can further obfuscate the original data, potentially reducing the risk of memorizing and regurgitating copyrighted content from the source material.
Privacy units. For the Enron Email dataset, consider each unique email sender single privacy unit. For BookSum, because the author name is not available in the dataset, consider the book-level privacy instead. Example-level and privacy unit-level statistics for these two datasets are provided in Table
Models. We use the pretrained GPT-2 small (125M) model (Radford et al., 2019) and LoRA (Hu et al., 2022) fine-tuning, following Yu et al. (2022). Please see Appendix C.3 for results on larger GPT-2 models, and our analysis in Appendix C.1 showing that LoRA does not significantly impact model performance.
There is no established consensus on the optimal method for sampling records to train User-level DP mechanisms, whether using Group Privacy User-wise DP-SGD. Therefore, in the following sections, we conduct a comprehensive evaluation exploring two key factors in this selection: (i) the record selection criteria, and (ii) the number of records to select for each privacy unit. Please see Appendix for the detailed experimental setup and Appendix C.4 for the analysis of the computational overhead.
Group Privacy
In this section, we evaluate Group Privacy for User-level DP and explore design choices for selecting records for each user, including the data selection strategy (Section 4.1) and the number of records to be selected (Section 4.2).
4.1 Data selection strategy
We compare the following five methods for selecting records for each privacy unit:
Random: Randomly select k records from each privacy unit.
We note that the choice of DP fine-tuning recipe is orthogonal to the user-level DP algorithmss, as the latter can be seamlessly combined with any DP fine-tuning pipeline. We show in Appendix C.2 that observe consistent results with different fine-tuning method et al., 2022).

---

Published as a conference paper at COLM 2024
1.0
3.0
8.0
1.0
3.0
8.0
Random
35.23
31.45
28.51
Random
27.31
27.15
27.00
Longest
34.95
31.32
28.23
Longest
27.34
27.19
27.02
Shortest
35.24
31.94
28.70
Shortest
27.40
27.18
27.03
Highest PPL
35.28
31.53
28.40
Highest PPL
27.40
27.16
27.06
Lowest PPL.
35.28
31.62
28.58
Lowest PPL
27.37
27.14
27.03
(a) Enron Email
(b) BookSum
Table 2: Perplexity of Group Privacy with different sequence selection methods on the Enron Email dataset (a) and BookSum dataset (b) under varying privacy budget e's. Selecting the longest sequence consistently performs best for Enron Email. For BookSum, no single strategy dominates across all privacy budgets.
=
2
k
5
10
k
k 5
10
k
20
50
1.0
35.67
34.26
35,23
1.0
27.30
27.34
27.31
27.54
27.46
32.01
30.92
31.45
8
27.13
27.15
27.15
27.22
29.67
8.0
30.13
28.74
28.51
8
27.08
27.05
27.00
27.07
27.22
= 00
23.00
19.88
19.53
8 00
26.07
25.75
25.62
25.53
25.43
(a) Enron Email
(b) BookSum
Table 3: Perplexity of Group Privacy with different number of sequences per privacy unit (i.e., k) on the Enron Email dataset (a) and BookSum dataset (b) under varying privacy budget g's. Smaller values of generally achieve better results for smaller e, while larger & favors larger k.
Longest: Select & records with the longest record length from each privacy unit. Shortest: Select k records with the shortest record length from each privacy unit.
Highest Perplexity (PPL): Select & records with the highest perplexity, i.e, records that are most surprising to the (initial pre-trained) model.
Lowest Perplexity (PPL): Select k records with the lowest perplexity, i.e., records that are least surprising to the (initial pre-trained) model.
We fix = 10 for both datasets and report the results in Table 2.
Selection strategy matters for Enron Emails, but not for BookSum. On the Enron Email dataset (Table 2a), selecting the longest records from each user yields the best performance across all privacy levels. This suggests that longer email records tend to be more informative and representative of each user 's data, leading to better utility after private fine-tuning, For the BookSum dataset (Table 2b), the performance differences across selection methods relatively small. Random selection performs best for & = 1.0 and 8.0, while selecting records with the lowest perplexity (least surprising to the model) works slightly better for 3.0.
In general, the results indicate that simple heuristics like selecting the longest or shortest records be effective strategies, sometimes outperforming more complex criteria like perplexity-basee selection. The performance gap between different selection methods is more pronounced on the Enron Email dataset compared to BookSum. This could be due to the greater variation in record lengths and content in the email data versus book summary data (see Table 7 in Appendix B).
4.2 Number selected records (k) per unit
The number of records (k) selected for each privacy unit is another important factor when applying group privacy. In group privacy, increasing makes both the noise and signal stronger, with opposing impact would result higher noise due to the increased sensitivity of the privacy mechanism. On the other hand, using larger k also results in more diverse data being used for training,
the model's performance. On the one hand, larger k

---

Published as a conference paper at COLM 2024
1.0
3.0
8.0
1.0
3.0
8 8.0
Random chunk
34.01
31.32
29.87
Random chunk
27.31
27.10
26.93
Random
34.22
31.85
30.22
Random
27.38
27.16
27.01
Longest
34.47
31.76
30.54
Longest
27.38
27.17
27.02
Shortest
35.68
32,91
31.19
Shortest
27.42
27.16
27.02
Highest PPL
34.36
31.65
30.16
Highest PPL
27.41
27.18
27.04
Lowest PPL
35.02
31.53
30.71
Lowest PPL
27.39
27.16
27.02
(a) Enron Email
(b) BookSum
Table 4: Perplexity of User-wise DP-SGD with different sequence selection methods on the Enron Email dataset (a) and BookSum dataset (b) under varying privacy budgets (i.e., E). Selecting the random chunks consistently performs best.
which can potentially improve the model's utility. Therefore, determining the optimal value of k for utiliity is trade-off between the increased noise and the diversity of the training data.
Optimal k balances noise from privacy and training data diversity. To investigate the impact of k, we fix the data selection strategy to the random selection method and vary the value of k. We observe in Table that for lower privacy budgets (i.e., & 1.0 3.0), using smaller k is generally better, as the undesirable increase in noise when k increases outweighs usually preferred,
the benefits of diverse data. However, for larger E, using larger as the high privacy budget can potentially tolerate slightly more noise being introd uced. In such cases, the diversity the training data becomes more beneficial for larger k, as evidenced by lower perplexity for larger k's when = co (non-private setting) We also observe similar results on GPT-2 medium and large models, as detailed in Appendix C.3.
5 User-wise DP-SGD
In this section, we evaluate User-wise DP-SGD and explore design choices for selecting records for each user. We discuss the data selection strategy (Section 5.1) and the number of records to be selected (Section 5.2). We also analyze the applicability of the advanced User-wise DP- SGD method proposed by Asi & Liu (2024) based more advanced gradient estimation sub-procedure.
5.1 Data selection strategy
Unlike Group Privacy that requires preprocessing to limit each user's contribution to records, User-wise DP-SGD imposes no such constraint. This allows exploring data selection strategies beyond record boundaries. Therefore, in addition to the five selection strategies discussed in Section 4.1, we also explore method called Random Chunk. Specifically, for each user, we concatenate all their records into a single document, and randomly fetch a chunk of the maximum record length (as defined by the model) from it.
As shown in Table 4, Random Chunk consistently outperforms other methods across datasets and privacy budgets e. This benefit likely stems from increased data diversity during training and the ability to span record boundaries.
5.2 Number of selected records (k) per unit
User-wise DP-SGD favors larger k, with diminishing returns. In contrast to Group Privacy (Section 4.2), increasing & monotonically improves User-wise DP-SGD's utility by providing more diverse data per user. However, larger also incurs slightly higher computation (Appendix C.5). The benefit diminishes as k grows, for practical deployment, choosing
Note that for both methods, report results for following their corresponding algorithms in Algorithm and Algorithm while bypassing the stage of aggregation and noise addition.
7

---

Published as a conference paper at COLM 2024
5
10
k
10
K 20
k 50
1.0
34.37
35.02
34.01
1.0
27.62
27.38
27.38
27.35
27.33
3.0
31.31
31.92
31.32
3.0
27.29
27.19
27.18
27.17
27.17
= 8.0
30.12
30.29
29.87
8 8.0
27.11
27.02
27.01
27.00
27.01
8
00
22.87
19.67
18.64
=
00
24.38
24.22
24.12
24.11
24.09
(a) Enron Email
(b) BookSum
Table 5: Perplexity of User-wise DP-SGD with different number of sequences per privacy unit (i.e., k) on the Enron Email dataset (a) and BookSum dataset (b) under varying privacy budget e's. Using a larger value of consistently improves performance.
sufficiently large will balance utility and efficiency. For instance, on the BookSum dataset, k = 10 appears to achieve the best trade-off between these two factors.
5.3 Analysis of the advanced User-wise DP-SGD
As discussed in Section 2, Asi & Liu (2024) proposed a User-wise DP-SGD algorithm that achieves the state-of-the-art excess rate and requires injecting less noise if per-user averaged gradients in a batch exhibit low variance. Specifically, recall that User-wise DP-SGD has clipping norm parameter C, and the noise added proportional showed that when most of the gradients are within lz-distance of each other, adding noise proportional to suffices for privacy purposes. Under the assumption that the records are drawn i.idd from some underlying distribution, Asi & Liu (2024) proved that this mechanism can achieve the optimal rate.
Asi & Liu (2024)
However, it remains unclear how applicable this algorithm, and other such theoretically motivated ones, are to real-world problems where the i.i.d. assumption might not be true, and the number of records per user is relatively small (e.g., k < 50). In order to answer this question, we measure the noise levels of standard User-wise DP-SGD and Asi & Liu (2024)'s method under different values of t, controlling the required gradient concentration. Figure shows that for most and Asi & Liu (2024)'s noise is lower than User-wise DP-SGD's ' only if T/C 0.1, making it highly unlikely to enter the advanced noising step (lines 10-15 in Algorithm 3). Despite its potential benefits, this method relies on high user gradient concentration, which does not hold in our settings.
k = c, 3.0
k c 3.0
10,
User-vris DPSCD
Unitorine DPSGD
Usersoiee DPSGD
0.02
Asi Lio. T/C
Noise
Asi Liu. 7/C 0.05
As Lio, Asi & iin, 7/C 0.1
Noise
Asi & Liu 0.05
0.01
Asi ii, 7/0 0.1
0.1
Asi & in 7/C
0.00
0.0
240
211
2
21
21
2/2
245
2
2
2
2.
2
2
Batch size
Batch size
Batch size
Figure 3: Effective noise of User-wise DR-SGD and the advanced method proposed by Asi & Liu (2024) under different numbers of records per user (k), with ε = 3.0. As shown, for Asi & Liu (2024) to yield lower noise than standard User-wise DP-SGD, the ratio between the concentration factor T and the clipping norm C must be smaller than 0.1. However, in the evaluated datasets, T/C around 1. Figure 7 presents results for different e's.
6 Comparing Group Privacy and User-wise DP-SGD
Finally, we compare the performance of Group Privacy and User-wise DP-SGD under the same privacy budget. For Group Privacy, we use Longest as its data selection strategy, and for each E, use its corresponding best as reported in Table 3. For User-wise DP-SGD, use Random Chunk for data selection, and use 10, as achieves the best trade-off between utility and efficiency. We vary the other hyperparameters, including learning rate, batch
8

---

Published as a conference paper at COLM 2024
Group Privacy
User-wise DP-SGD
Group Privacy
User-wise DP-SGD
0.5
35.28
35.08
0.5
27.50
27.43
1.0
33.50
33.36
1.0
27.41
27.29
g 3.0
31.01
30.98
8 3.0
27.17
27.15
8 8.0
28.51
28.32
8.0
27.00
27.02
(a) Enron Email
(b) BookSum
Table 6: Perplexity of Group Privacy and User-wise DP-SGD under varying privacy budgets (e).
User-wise DP-SGD generally outperforms Group Privacy, especially for smaller privacy budgets.
0.01
27.58
27.55
27.54
0.01
27.36
27.31 37
Clipping norm
Clipping norm
0.1
27.49
27.44
27.42
0.1
27.32
27.28
27.29
27.47
27.42
27.41
27.31
27.28
27.29
10.0
10.0
27.50
27.42
27.33
27.29
27.20
256
512 Batch size
1024
250
510
1020
Batch size
(a) Group Privacy
(b) User-wise DP-SGD
Figure 4: Perplexity of Group Privacy (a) and User-wise DP-SGD (b) the BookSum dataset with a privacy budget & 1.0, under varying clipping norms and batch sizes, Using larger batch sizes generally improves the performance for both methods. However, Group Privacy exhibits higher sensitivity to clipping norm variations compared to User-wise DP-SCD.
size, number of training iterations, and the clipping norm, and report the best results for each mechanism in Table 6.
User-wise DP-SGD outperforms Group Privacy. At the mechanism level, User-wise DP-SCD consistently outperforms Group Privacy, especially for smaller privacy budgets. This is potentially due to User-wise DP-SGD allowing for more diverse data being selected in different minibatches during training, while Group Privacy already constrains the training data be subset of user records during the preprocessing stage. Additionally, for some users who have fewer than k records, our implementation of Group Privacy will perform resampling to obtain k records. This repetition may also negatively impact the model's 's performance, We acknowledge this as a potential limitation of our evaluation that could be addressed in future work. However, it is worth noting that Group Privacy requires minimal changes to implement on top of existing record-level private training, whereas User-wise DP-SGD may require modifications to the data loading pipeline. In this regard, practitioners may need to consider the implementation difficulty when choosing between these two methods.
Sensitivity to clipping norms. We specificllyy investigate the sensitivity of both mechanisms to clipping norms, as reported in Figure 4. As shown, Croup Privacy is generally more sensitive to clipping norms compared to User-wise DP-SGD. This is because the clipping operation is performed on the gradient of each individual record for the Group Privacy method (see Algorithm 1). In contrast, for User-wise DP-SGD, the clipping is applied to the averaged gradients, which ideally exhibit less variance, making it less sensitive to clipping norms (see Algorithm 2).
Related Work
Record-level Differential Privacy. Differential privacy (DP) (Dwork et al., 2006b;a) is widely adopted framework that formally bounds the privacy leakage of individual user information while still enabling the analysis population-level patterns. When training deep neural networks with DP guarantees, the go-to algorithm is Differentially Private

---

Published as a conference paper at COLM 2024
Stochastic Gradient Descent (DP-SGD), introduced in the seminal work of Abadi et al. (2016). This algorithm operates by clipping the contributions of gradients on a per-record basis and injecting noise into the average gradient updates during each iteration of SGD.
Previous research has applied DP-SGD for language models (Ponomareva et al., 2023), primarily focusing on the record level. These applications include work in natural language understanding (Li et al., 2022; Yu et al., 2021; 2022), natural language generation (Anil et al., 2022), in-context learning (Tang al., 2024; Wu et al., 2023; Duan et al., 2023), prompt tuning (Li et al., 2023b), synthetic text generation (Al Aziz et al., 2021; Yue et al., 2023), and retrievalauugmened language models (Huang al., 2023).
User-level Differential Privacy. McMahan et al. (2018); Kairouz et al. (2021) studied user- level DP training of LSTM based language models in the federated learning setting. In contrast, we consider general central training setting for more flexible samplers. Moreover, we studied modern transformer-base based language models. More broadly, several fundamental and important problems have been actively studied under user-level DP in recent years, e.g., machine learning (Wang et al., 2019), learning discrete distributions (Liu et al., 2020), histogram estimation (Amin et al., 2019), Structured Query Language (SQL) (Wilson al., 2020), number of users required with multiple items per user (Ghazi et al., 2021), stochastic convex optimization (Levy et al., 2021; Bassily & Sun, 2023; Ghazi et al., 2023a; Asi & Liu, 2024) and reduction from item-level to user-level DP (Ghazi et al., 2023b). As mentioned, common assumption the theoretical results above is that the data is drawn i.i.d. from an underlying distribution. This assumption is too restrictive, and it would be interesting to propose more realistic assumptions under which we can still gain some theoretical insights.
We note that independent and concurrent work of Charles et al. (2024) also examines the empirical performance of user-level DP methods for fine-tuning LLMs. The key findings that User-wise DP-SGD usually outperforms Group Privacy in terms of utility are generally consistent between the two studies. Notably, their work provides valuable insights by offering an explicit methodology for selecting the group size in both mechanisms, whereas ours mainly performs ablations of various sizes. Moreover, our work differs from theirs in that we explore broader design space for data selection strategies under various User-level DP mechanisms. In this expanded exploration, we observe substantial performance gaps between suboptimal and better data selection methods. In addition, their evaluation focuses on the larger StackOverflow and news datasets using single 350M-parameter model while our work utilizes books and email datasets and considers models of various sizes. We also provide case studies for the User-level DP algorithm with the ttate-of-the-al excess rate.
Conclusions
In this work, we study language model fine-tuning with DP guarantees. Since each user generally contributes multiple text records to the training corpus, to ensure equal privacy guarantées across users, the privacy unit needs to be set at the user level. We identify and systematically evaluate two mechanisms, Group Privacy and User-wise DP-SGD, for achieving user-level DP when fine-tuning large language models on natural language generation tasks. For each mechanism, we investigate data selection strategies to improve the trade-off between privacy and utility.
While our work provides valuable insights, we acknowledge few limitations and suggest directions for future research. First, in our implementation of User-wise DP-SGD, we fixed the value of & (the number of gradients to sample per user) for simplicity. Exploring more dynamic choices of for different users may yield better utility while preserving privacy guarantees. Second, since User-wise DP-SGD can perform data selection on the fly, it will be interesting to investigate how other data selection methods, such as those based on gradient information active learning, might impact the utility. Finally, while we focus on language generation, user-level DP has broader applications motivating analysis in domains like recommendation systems and structured prediction.
10

---

Published as a conference paper at COLM 2024
References
Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS, pp. 308-318, 2016.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arxiv preprint arXiv:2303.08774, 2023.
Md Momin Al Aziz, Tanbir Ahmed, Tasnia Faequa, Xiaoqian Jiang, Yiyu Yao, and Noman Mohammed. Differentially private medical texts generation using generative neural networks. ACM HEALTH, 3(1):1-27, 2021.
Kareem Amin, Alex Kulesza, Andres Munoz, and Sergei Vassilvtiskii. Bounding user contributions: bias-variance trade-off in differential privacy. In ICML, pp. 263-271, 2019.
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private BERT. In EMNLP (Findings), pp. 1481-6491, 2022.
Rohan Anil, Andrew Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. aiv preprint arXiv:2305.10403 2023.
Hilal Asi and Daogao Liu. User-level differentially private stochastic convex optimization: Efficient algorithms with optimal rates. In AISTATS, 2024.
Raef Bassily and Ziteng Sun. User-level private stochastic convex optimization with optimal rates. In ICML, pp. 1838-1851, 2023.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. ar preprint ar Xrx::22303.122 12711,, 2023.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX, pp. 2633-26500 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In ICLR, 2022.
Zachary Charles, Arun Ganesh, Ryan McKenna, H Brendan McMahan, Nicole Mitchell, Krishna Pillutla, and Keith Rush. Fine-tuning large language models with user-level differential privacy. arXiv preprint ar X0io::207000777, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint 1xXiv:2077.0377,, 2021.
Lynn Chua, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, and Chiyuan Zhang. How private are DP-SGD implementations? In International Conference on Machine Learning (ICML), 2024.
Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases. aiv preprint ar arXiv:2306.16092 2023.
Vadym Doroshenko, Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Connect the dots: Tighter discrete approximations of privacy loss distributions. PoPETS, 4:552-570, 2022.
Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: Differentially private prompt learning for large language models. In NeurIPS, 2023.
11

---

Published as a conference paper at COLM 2024
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT, PP- 486-503, 2006a.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to sensitivity in private data analysis. In TCC, pp. 265-284, 2006b.
Arun Ganesh. Tight group-level DP guarantees for DP-SGD with sampling via mixture of Gaussians mechanisms. arXiv preprint ar Xrio:2401.100944 2024.
Badih Ghazi, Ravi Kumar, and Pasin Manurangsi. User-level differentially private learning via correlated sampling. In NewrlPS, pp. 20172-20184, 2021.
Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Raghu Meka, and Chiyuan Zhang. On user-level private convex optimization. In ICML, pp. 11283-11299, 2023a.
Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Raghu Meka, and Chiyuan Zhang. User-level differential privacy with few examples per user. In NeurlPS, 2023b.
Google's DP Library. DP Accounting Library.
https://ggtuub.oommmge/ 2020.
differential- privacy tatrraccyytee/mmiinn http:///e/emmmm/p///////////eep/ parthon/ddpaccounne
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential
privacy. NeurIPS, pp. 11631-11642, 2021.
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. In NeurlPS, 2023.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder, When the large language model meets programming the rise of code intelligence. arXiv preprint wrXio:2401.1419, 2024.
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. arXio preprint urXio:2303.15715, 2023.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
Yangsibo Huang, Samyak Gupta, Zexuan Zhong, Kai Li, and Danqi Chen. Privacy implica- tions of retrieval-based language models. In EMNLP, pp- 14887-14902, 2023.
Peter Kairouz, Brendan McMahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling or shuffling. In ICML, pp. 22.5213--222 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. Xr preprint ar Xiv:2000 .08361,, 2020.
Bryan Klimt and Yiming Yang. The Enron corpus: A new dataset for email classification research. In ECML, pp. 217-226, 2004.
Antti Koskela, Joonas Jälkö, Lukas Prediger, and Antti Honkela. Tight differential privacy for discrete-valued mechanisms and for the subsampled Gaussian mechanism using FFT. In AISTATS, pp. 3358-3366, 2021.
Wojciech Kryšciñiski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. Booksum: A collection of datasets for long-form narrative summarization. In EMNLP (Findings, pp. 6536-6555, 2022.
Peter Lee, Sebastien Bubeck, and Joseph Petro. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. NEJM, 3833 2023.
12

---

Published as a conference paper at COLM 2024
Daniel Levy, Ziteng Sun, Kareem Amin, Satyen Kale, Alex Kulesza, Mehryar Mohri, and Ananda Theertha Suresh. Learning with user-level privacy. In NewrlPS, pp. 12466-12479, 2021.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! TMLR, 2023, 2023a.
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In ICLR, 2022.
Yansong Li, Zhixing Ian, and Yang Liu. Privacy-preserving prompt tuning for large lan- guage model services. ariiv preprint ar arXi::3305.062 06212, 2023b.
Yuhan Liu, Ananda Theertha Suresh, Felix Xinnan X Yu, Sanjiv Kumar, and Michael Riley. Learning discrete distributions: user vs item-level privacy. In NeurIPS, pp. 20965-20976, 2020.
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie- Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining, Briefings in Bioinformatics, 23(6)bbbcc499 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiien learning of deep networks from decentralized data. In AiSTATS, pp. 1273-1282, 2017.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In ICLR, 2018.
Microsoft. A fast algorithm to optimally compose guarantees of differentially private (DP) mechanisms to arbitrary accuracy. https://github.co wwmmmmcoooottt..oooomoooomom/moe swtpssotfttrvvveeeeeeo pttp:pvv___ooaa accountann,,,, 2021.
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, ((667956)225922222 2023.
Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H Bren- dan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta. How to DP-fy ML: A practical guide to machine learning with differential privacy, JAIR, 77: 1113-1201, 2023.
Lukas Prediger and Antti Koskela. Code for computing tight guarantees for differential privacy. https:// http://gtthubccoo htw.bb.cco////oo wtt..commmDDPaa.. DPBaya/PaDDDD Accountant, 2020.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever. Language models unsupervised multitask learners. 2019. https: //cdn.openai.commmmmmmm www.ppppn.ii.caoo.ooooooooooocet wwt.c/ttooeeeeeeeeeeeeeeeeeeee laage-llanggggmdeelllle laatelllllangugg__aee models are unsupervised Seciittask_learrers pdf..
modells
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Llama: Open foundation models for code. Xiv preprint ar arXir:2308.119955 2023.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Xrv preprint arXiv:2212.13313. 2022.
Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. Privacy preccc-preseriingg in-context learning with differentially private few-shot generation. In ICLR, 2024.
Salil P. Vadhan. The complexity of differential privacy. In Tutorials on the Foundations of Cryplography, pp. 357-450. Springer International Publishing, 2017.
13

---

Published as a conference paper at COLM 2024
Zhibo Wang, Mengkai Song, Zhifei Zhang, Yang Song, Qian Wang, and Hairong Qi. Beyond inferring class representatives: User-level privacy leakage from federated learning. In INFOCÓM, pp. 2512-2520, 2019.
Royce Wilson, Celia Yuxin Zhang, William Lam, Damien Desfontaines, Daniel Simmons- Marengo, and Bryant Gipson. Differentially private SQL with bounded user contribution. PoPETS, 2:230-250, 2020.
Tong Wu, Ashwinee Panda, Jiachen Wang, and Prateek Mittal. Privacy-preserving in- context learning for large language models. In ICLR, 2023.
Zheng Xu, Maxwell Collins, Yuxiao Wang, Liviu Panait, Sewoong Oh, Sean Augenstein, Ting Liu, Florian Schroff, and H Brendan McMahan. Learning to generate image embeddings with user-level differential privacy. In CVPR, pp. pp. 7969-7980, 2023.
embeeddnngg
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Yan Liu. Large scale private learning via low-rank reparametrization. In ICML, pp. 12208-12218, 2021.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In ICLR, 2022.
Xiang Yue, Huseyin A Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim. Synthetic text generation with differential privacy: A simple and practical recipe. In ACI,, 2023.
14

---

Published as a conference paper at COLM 2024
Appendices
A User-Level Algorithm of Asi & Liu (2024)
16
Experimental details
17
B.1 Dataset details
17
B.2 Model details
17
B.3 Privacy Accounting
17
More results
19
C.1 LoRA v.s. Full fine-tuning
19
C.2 Compatibility with different DP fine-tuning method
19
C.3 Results under different models
19
C4 Computational overhead
C.5 Cost of k in User-wise DP-SGD
21
C.6 More results for comparing User-wise DP-SGD and Asi & Liu (2024)
21
15

---

Published as a conference paper at COLM 2024
A
User-Level Algorithm of Asi & Liu (2024)
We describe the algorithm in Asi & Liu (2024) using our notation for completeness. They assume each user owns the same amount of record and use the full batch size (use all dataset in each step). We modify the algorithm by drawing K records for the ith user in the following Pseudocode.
Algorithm User-wise DP-SGD with filter
1: Input: Initial model weights 6,, training set D N users and M records, learning rate H), iterations T, privacy budget e, number of records per user {K] concentration
threshold т
/+ Stage 1. Compute noise multiplier */
2: 0 + CANVACCOUUNTNNG(E222//......
/+ Stage 2. Training */
3: for 1,.., , do
4: for 1..,,,..,,Na do
5: Sample jele) K records for the ith user in D
/* Compute user- averagee gradient */
6: St b
(Ejen)) Vo, {(,, (x))
7: end for
/* Compute the concentration scores +/
8: Compute
ccnnc (D, = E 8rrll S T).
(1)
Run the AboveThreshold with the concentration scores. it passes, remove the outliers */
9:
T if Threshold (s -,,4N/3 (44 T then
10:
Set 13 Ø
11:
Set fú, Ei (1) -rr S 2T)
0
fúi < N/2
12:
Add 1th user to B, with probability Pt, for Paj
1
fúi 2 2N/3
f/-N/22
N/6
O.W.
/* Aggregate and add noise for the remaining users +/
13:
Let gi B. Lies, gú, if B, is not empty, and 0 otherwise
14:
g: - & + vı, where Vi ~ (0 8T 1og(eT/8)02//N7 L /* Model update */ n
15:
0:11
16:
else
17:
Halt. /* Halt the algorithm if it does not pass. +/
18:
end if
end for
AboveThreshole Threshold is a classic algorithm used in DP literature, See Algorithm in Asi Liu (2024) for more details and references.
16

---

Published as a conference paper at COLM 2024
B Experimental details
B.1 Dataset details
Data preprocessing for the Enron Email dataset. To prepare the Enron Email dataset (Klimt & Yang, 2004) for our evaluation of user-level DP implementations, first parse each record, into DATE, FROM (sender), To (receiver), SUBJECT, and BODY, using regular expressions. Note that for each of the "CC" "BCC" addresses the email, we treat it as new receiver. Specifically, we will create a new record with the same (DATE, FROM, SUBJECT, BODY), but different To (i.e., receiver). We then perform in-email dedu- plication to attain dataset with higher quality: We only keep records whose maximum repetition of any 20-gram is 1. Forwarded emails contain the email body of the forwarded messages; this complicates the privacy accounting process. Therefore, for simplicity, we remove forwarded emails. After preprocessing, the final dataset contains 240, 173 emails.
Distribution of number of records per privacy unit. Figure 5 presents the distribution of number of records per privacy unit in both datasets.
10000
Number senders
Number of books
6000
1000
2000
10
10a
10²
50
100
1.50
2000
250
Number of emails per sender
Number of summaries per book
Figure 5: Distribution of number of records per privacy unit in Enron Email (left) and BookSum (right).
Statistics after data selection. For reference, Table 7 reports some statistics of the selected records after applying different data selection methods discussed in Section 4.1.
B.2 Model details
Our experiments primarily the pretrained GPT-2 small model (Radford et al., 2019), which has 125 million trainable parameters. To assess the generalizability of our findings, also present results with GPT-2 models of different sizes in Appendix C.3. For each evaluated dataset, we fine-tune the model for the language modeling task, using LoRA (Hu et al., 2022) with rank of 32. As shown in Appendix C.1, LoRA does not significantly impact model performance. Model performance is measured using perplexity, a standard evaluation metric for language models.
B.3 Privacy Accounting
We perform privacy accounting assuming that the batches are constructed using Poisson subsampling, where each batch is independently sampled by including each record / user independently with a certain probability, even though our implementation involves constructing batches by going through users/recon records in random order. We note that this assumption has been a common practice in the privacy accounting of DP-SGD, starting with the work of Abadi et al. (2016). Recent work by Chua et al. (2024) points out that there can be substantial gaps between the privacy guarantees of DP-SGD when using Poisson subsampling shuffling: however, since make the Poisson subsampling assumption
17

---

Published as a conference paper at COLM 2024
Avg. length
Avg. PPL
Fraction of selected sequences
w/ length < max_seq_len
Random
240.67
214.67
0.73
Longest
274.91
201.14
0.69
Shortest
222.28
278.75
0.76
Highest PPL.
233.27
1123.77
0.74
Lowest PPL
247.18
188.92
0.72
(a) Enron Email
Avg. length
Avg. PPL
Fraction of selected sequences w/ length < max_seq_len
Random
557.49
34.37
0.61
Longest
817.93
33.79
0.30
Shortest
338.43
41.99
0.80
Highest PPL
483.79
50.32
0.66
Lowest PPL
539.52
25.94
0.65
(b) BookSum
Table 7: Comparison of the average sequence length, perplexity, and fraction of selected sequences with length smaller than the maximum allowed sequence length, after applying different data selection techniques to the (a) Enron Email dataset and (b) BookSum dataset.
for both Group Privacy and User-wise DP-SGD, we consider our evaluation to still be informative in a relative sense even in the presence of this gap.
The privacy analysis of DP-SGD using Poisson subsampling with M records and pected batch size B, run for T steps is performed by analyzing the T-fold composition of the Poisson subsampled Gaussian mechanism with subsampling probability of B/M. It is possible to numerically compute the smallest 0r, up to some tolerance, such that this mechanism satisfies (-DD using algorithms based on privacy loss distributions (PLD) (Koskela et al., 2021; Gopi et al., 2021; Doroshenko et al., 2022), using various open source libraries Microsoft. (2021); Prediger & Koskela (2020). In particular, we use the implementation in the dp uppaccuuninng library (Google's s DP Library,, 2020). We use PRIVACCOUNTINC(E. M, B, T) to refer this smallest v.
PRIVACCOUNTING for Algorithm Under the assumption of Poisson subsampling of users, the same privacy analysis is applicable with "number of records" M replaced by "number of users" N and "batch size" B replaced by "user batch size" n, and thus, the optimal noise parameter is given as PRIVACCOUNTINC((. N,,, T).
GROUFPRIVACCOUNING for Algorithm 1. Ganesh (2024) showed that the privacy anal- ysis of a single step of DP-SGD under addition reriitioonremovalll removal of k records under Poisson subsam- pling is given by considering the Gaussian mechanism with sensitivity that is randomly drawn from the Binomial distribution Bin(k, B /M) referred to as the "Mixture of Gaus- sians (MoG)" mechanism. The privacy analysis of the entire DP-SGD can thus be performed by analyzing the T-fold composition of this MoG mechanism, which can be done in a tight manner up numerical accuracy using privacy loss distributions. We use the implementation available in the dp_ accounting library.
18

---

Published as a conference paper at COLM 2024
C
More results
C.1
LoRA v.s. Full fine-tuning
Table presents the perplexity scores for full fine-tuning and Low-Rank Adaptation (LoRA) (Hu et al., 2022) fine-tuning on the Enron Email and BookSum datasets. As shown, applying LoRA fine-tuning has a relatively minor impact on the model's utility.
Enron Email
BookSum
Full fine-tuning
18.53
23.79
LoRA, r = 64
18.83
24.07
LoRA, 32
19.05
24.18
LoRA, r = 16
19.39
24.31
Table 8: Comparison of perplexity scores for full fine-tuning and Low-Rank Adaptation
(LoRA) fine-tuning with different rank values on the Enron Email and BookSum datasets.
C.2 Compatibility with different DP fine-tuning method
Note that the choice of DP fine-tuning recipe is orthogonal to the user-level DP algorithms, as the latter can be seamlessly combined with any DP fine-tuning pipelines.
We also rerun our evaluation using the pipeline from Li et al. (2022) on the Enron Email dataset. Consistent with our previous findings, random chunking works best for User-wise DP-SGD, and longest sequence selection works best for Group Privacy. The performance of both mechanisms, using their respective optimal selection strategies, is shown in Table and aligns with our findings in the main paper.
Group Privacy
User-wise DP-SGD
é 1.0
33.44
33.21 21
3.0
31.08
30.77 76.76
6 8.0
28.34
28.07
Table 9: Perplexity of Group Privacy and User-wise DP-SGD under different privacy budgets (e) on the Enron Email dataset, using the DP--ine-tuning method from Li et al. (2022). User-wise DP-SGD generally outperforms Group Privacy.
C.3 Results under different models
We further report the performance of varying the number of records per privacy unit (k) for Croup Privacy and User-wise DP-SGD under different sizes of GPT-2 models in Table 10 and Table 11, respectively. The observed trends are consistent across different model sizes.
19

---

Published as a conference paper at COLM 2024
2
10
20
50
8 c 3.0
27.31
27.34
27.45
27.56
27.49
27.14
27.15
27.17
27.24
29.68
8 8.0
27.08
27.05
27.02
27.07
27.22
(a) GPT2-small
8 2
k=
10
20
50
21.91
21.99
22.09
22.87
8 8 8.0
21.70
21.73
21.68
21.72
22.05
21.61
21.60
21.52
21.53
21.68
(b) GPT2-medium
2
10
20
50
1.0
19.28
19.28
19.32
19.39
20.04
8
19.21
19.18
19.18
19.22
19.74
8 8.0
19.12
19.03
18.99
19.00
19.31
(c) GPT2-large
Table 10: Perplexity of Group Privacy with different number of sequences per privacy unit (i.e., k) on the Enron Email dataset (a) and BookSum dataset (b), under varying privacy budget e's, using different sizes of GPT-2 models. Smaller values of k generally achieve better results for smaller e, while larger favors larger k.
2
k
k 10
k 20
k 50
8 27.62
27.38
27.38
27.35
27.33
8 3.0
27.29
27.19
27.16
27.17
27.17
8 8.0
27.11
27.02
27.01
27.00
27.01
(a) GPT2-small
2
k=
10
20
50
21.88
21.80
21.77
21.75
21.75
8
21.61
21.52
21.54
21.52
21.51
8 8.0
21.50
21.47
21.44
21.43
21.43
(b) GPT2-mediung medium
2
k
k 10
20
50
1.0
19.22
19.21
19.19
19.19
19.19
8
19.13
19.08
19.06
19.05
19.05
8 8.0
18.99
18.93
18.93
18.92
18.92
(c) GPT2-large
Table 11: Perplexity of User-wise DP-SGD with different number of sequences per privacy unit (i.e., k) on the Enron Email dataset (a) and BookSum dataset (b), under varying privacy budget e's, using different sizes of GPT-2 models. Using larger value of consistently improves performance.
20

---

Published as a conference paper at COLM 2024
C.4
Computational overhead
The main overhead of Group Privacy training methods introduced by the per-example gradient clipping and noise addition (lines 11-13 in Algorithm 1). For the User-wise DP- SGD method, the primary overhead stems from the user-wise data sampling (line in Algorithm 2), as well as per-user gradient clipping and noise addition (lines 8-9 Algorithm Al Algorithm 2).
We record the training time (in seconds) per step for different models and training methods with k = 10 on the Enron Email dataset in Table 12. The results are obtained using LoRA fine-tuning with 32 and a batch size 1024, averaged across 100 training steps, on single NVIDIA A100-80G GPU. As shown, both methods introduce very little computational overhead (4%-9%) compared to non-private training, regardless of the model size.
Model
Non-private training
Group privacy
User-wise DP-SGD
GPT2--mall
11.95
12.70 (1.06x)
12.82 (1.07 x)
GPT2-medium
30.90
32.00 (1 (1.04x)
32.85 (1.06x)
GPT2-large
62.57
64.80 (1.04x)
68.45 (1.09
Table 12: Comparison of per-step runtime (seconds) for non-private training, and training with Croup Privacy and User-wise DP-SGD on GPT-2 models of varying sizes.
C.5 Cost in User-wise DP-SGD
Increasing in Group Privacy automatically results in larger datasets, and thus higher training time. We also show in Figure that increasing k in User-wise DP-SGD also incurs longer running time, due the cost of sampling more records per training step.
Runtime [seconds)] per training epoch
3000
2500
2000
1500
1000
500
k=
Figure Runtime per training epoch under different K's for BookSum dataset with User-wise DP-SGD.
C.6 More results for comparing User-wise DP-SGD and Asi & Liu (2024)
Figure 7 compares the noise of User-wise DP-SGD and the advanced method proposed by Asi & Liu (2024) under different e's. Still, for Asi & Liu (2024) to yield lower noise than standard User-wise DP-SGD, the ratio between the concentration factor and the clipping norm C must be smaller than 0.1. However, in the evaluated datasets, T/C around 1.
21

---

Published as a conference paper at COLM 2024
== 2,
1.0
5,
10,
User wise DP5GD
User DPSGD
0.3
User wise DPSGD
Asi & Lis r/C 0.02
0.10
Asi & Liu, +/0 = 0.02
Asi & Liu, r/C 0.02
Noise
0.02
z/C7
Noise
7/C 0.05
Noise
0.2
Lin, v/C
Asi Liu. T/C
Li. 0.1
0.05
0.01
0.1
0.00
0.0
2
24
2/4
2
2
212 Batch size
2'
Batch size
Batch size
k 2, 3.0
k 5, 3.0
10,
3.0
User wize DPSGD
0.05
User visa DPSCD
0.2
User wise DPSGD
& Lis r/C 0.02
Asi Liu, +/C = 0.02
Li., r/C 0.02
Noise
Noise
0.06
0.05
Noise
Lin,
Asi & Li +/C
Asi Lin, 1/C 0.1
0.1
As & Lin, r/C 0.1
0.02
0.00
0.00
0.0
2
21
2'
214
2
2
21
2
2
2
2'
2
2
2/
214
Batch size
Batch size
Batch size
2, c 8.0
5,
B.0
10,
8.0
0.015
User-uise DPSGD Asi & Lis.
0.06
Usersoiie DPSCD) T/C
0.15
Usenowie DPSCO Asi Uu, Asi & Liu, r/C 0.1
r/C
Noise
0.010
Noise
Asi Liu, T/C 0.05
Noise
Asi & Lis. 7/C 0.1
Asi & Liu, r/C = 0.1
0.005
0.02
0.05
0.000
0.00
0.00
211
2/4
228
211
212
Batch size
Batch size
Batch size
Figure 7: Effective noise of User-wise DP-SGD and the advanced method proposed by Asi & Liu (2024) under different e's and numbers of records per user (k). As shown, for Asi
& Liu (2024) to yield lower noise than standard User-wise DP-SGD, the ratio between the concentration factor T and the clipping norm C must be smaller than 0.1. However, in the evaluated datasets, T/C is around 1.
22


Reviews:
[{"review_id": "1SrXm0C4UI", "reviewer": "Reviewer_mJeH", "summary": "In this work, the authors study language model fine-tuning with differential privacy guarantees. Since each user generally contributes multiple text sequences to the training corpus, to ensure equal privacy guarantees across users, the privacy unit needs to be set at the user level. The authors identify and systematically evaluate two mechanisms, Group Privacy and User-wise DP-SGD, for achieving user-level differential privacy when fine-tuning large language models on natural language generation tasks. For each mechanism, the authors investigate data selection strategies to improve the trade-off between privacy and utility.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 Limited technical novelty\n> What is the challenge to adapt User-wise DP-SGD in LLM given there are many existing methods?\n\n**A**:  We outline three challenges in adapting existing methods and place our contributions in this context.\n\n1. **Lack of a systematic evaluation**: Prior work (McMahan'18; Kairouz'21; Xu'23) has neither systematically studied user-level DP implementation choices nor their performance in LLMs. Many applications used naive group privacy (Amin'19; Levy'21) which leads to worse utility for smaller ε's. (see Sec.6).\n\n- Our contribution: We systematically evaluated the two mainstream mechanisms for user-level DP in LLMs. We also empirically show the overly restrictive assumptions of Asi'23, an algorithm with a theoretically optimal excess rate.\n\n2. **Lack of reliable testbeds** for user-level DP methods in LLM applications for effective comparison\n\n- Our contribution: We derived two testbed datasets, Enron Email and BookSum, from existing non-DP-purpose datasets to simulate real-world use cases and benefit future research.\n\n3. **Overlooked data selection strategy** when adapting user-level DP to LLMs\n\n- Our contribution: We proposed and evaluated data selection strategies for both group privacy and user-wise DP-SGD, showing significant performance gap based on the strategy used - e.g., on Enron Email w/ ε=1.0, perplexity ranges from 34.01 for the best strategy to 35.68 for the worst (Table 4a).\n\n###  Q2 Lack of baselines\n>e.g. Large Language Models Can Be Strong Differentially Private Learners. ICLR’22\n\n**A**: Please note that the choice of DP fine-tuning recipe is **orthogonal** to the user-level DP algorithms, as the latter can be seamlessly combined with any DP fine-tuning pipelines. Our evaluation used the strong DP fine-tuning baseline from [1], and showed consistent results across models (GPT2 small, medium, large) and datasets (BookSum & Enron Email).\n\nIn response to the suggestion, we reran our evaluation using the pipeline from the referenced ICLR'22 work on Enron Email. Consistent with our previous findings, random chunk works best for user-wise DP-SGD, and longest sequence selection for group privacy.  The performance of both mechanisms, using their respective best data selection strategies, is shown below and aligns with our submission results.\n\n|ε|Group privacy|User-wise DP-SGD|\n|-|-|-|\n|1.0|33.48|33.21|\n|3.0|31.09|30.76|\n|8.0|28.34|28.07|\n\n[1] Yu, et al. Differentially Private Fine-tuning of Language Models. ICLR’21", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors present a systematic empirical evaluation of User-level DP for fine-tuning LLMs on natural language generation tasks.\n\n2. The authors therefore study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users.", "reasons_to_reject": "1. The technical novelty is limited. What is the challenge to adapt User-wise DP-SGD in LLM, there are many existing User-wise DP-SGD methods.\n\n2. Lack of baselines. It is better to add some recent baseline to demonstrate the advantage of the proposed method.  For example: Large Language Models Can Be Strong Differentially Private Learners. 2022 ICLR", "questions_to_authors": "see weakness.", "extra_scores": {}}, {"review_id": "4WmV3djpFD", "reviewer": "Reviewer_pxQP", "summary": "In this paper, the authors present a study on the application of user-level differential privacy (DP) in the context of fine-tuning large language models (LLMs) on natural language generation tasks.  It is a critical issue in the field of machine learning, specifically the privacy concerns associated with fine-tuning LLMs on sensitive data. The document is well-structured, with a clear abstract, introduction, methodology, experimental setup, results, and conclusions. Experimental results on Enron Email dataset and BookSum dataset were given to show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 Implementation of User-wise DP-SGD\n> The paper mentions that User-wise DP-SGD may require modifications to the data loading pipeline, which could be a barrier to adoption in practice.\n\n**A**: Thanks for the comment. Please note that an efficient implementation of user-wise DP-SGD can be achieved with only **minor changes** to the data loading pipeline. Specifically, it entails:\n- Modifying the per-example dataloader to a per-user dataloader (line 5 in Algorithm 2), which only requires implementing a customized dataloader with a few additional lines of code.\n- Fetching training examples for each user and taking the gradient average per user on the fly during each gradient step (lines 6~10 in Algorithm 2).\n\nFortunately, since both steps can be implemented with minimal effort, user-wise DP-SGD can be easily adopted in practice.\n\n### Q2 Computational overhead\n> The computational overhead introduced by the differential privacy mechanisms is mentioned but not thoroughly analyzed, particularly for larger models or datasets.\n\n**A**: Thanks for the comment. The main overhead of group privacy training methods is introduced by the per-example gradient clipping and noise addition (lines 11-13 in Algorithm 1). For the user-wise DP-SGD method, the primary overhead stems from the user-wise data sampling (line 5 in Algorithm 2), as well as per-user gradient clipping and noise addition (lines 8-9 in Algorithm 2).\n\nTo provide a more quantitative analysis, we present the following table showcasing the training time (in seconds) per step for different models and training methods with $k=10$ on the Enron Email dataset. The results are obtained using LoRA fine-tuning with rank being 32 and a batch size of 1024, averaged across 100 training steps, on a single NVIDIA A100-80G GPU.\n\n| Model | Non-private training | Group privacy | User-wise DP-SGD |\n|:---:|:---:|:---:|:---:|\n| GPT2 | 11.95 | 12.70 (1.06x) | 12.82 (1.07x) |\n| GPT2-medium | 30.90 | 32.00 (1.04x) | 32.85 (1.06x) |\n| GPT2-large | 62.57 | 64.80 (1.04x) | 68.45 (1.09x) |\n\nAs shown in the table, both methods introduce very little computational overhead (4%–9%) compared to non-private training for the GPT-2 model, regardless of the size.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper tackles the concept of user-level DP in the fine-tuning of LLMs, which is a departure from the traditional example-level DP.\n\n2. The authors provide a systematic empirical evaluation of two user-level DP mechanisms.", "reasons_to_reject": "1. The paper mentions that User-wise DP-SGD may require modifications to the data loading pipeline, which could be a barrier to adoption in practice.\n\n2. The computational overhead introduced by the differential privacy mechanisms is mentioned but not thoroughly analyzed, particularly for larger models or datasets.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3njoe4qofq", "reviewer": "Reviewer_sVEv", "summary": "The paper delves into the issue of privacy concerns when fine-tuning large language models (LLMs) on sensitive data. The paper introduces the concept of User-level Differential Privacy (DP) as a solution to ensure uniform privacy protection across users, especially in scenarios where contributions per user vary. By focusing on two mechanisms, Group Privacy and User-wise DP-SGD, the study evaluates the design choices such as data selection strategies and parameter tuning to strike a balance between privacy and utility.\nOverall, the paper provides a new perspect of user-wise DP for LLM fine-tuning on natural language generation tasks, shedding light on the importance of considering user privacy guarantees in addition to example-level privacy. However, the solution is somehow lacking of novelty. Moreover, the comparison of proposed approach is not thorough.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 Limited contribution: straightforward solution building upon existing method\n\n**A**: While our work builds on existing methods, our main contribution is to provide a comprehensive understanding of the pros and cons of group privacy and user-wise DP-SGD through systematic evaluation. We tested two datasets, three models, various DP budgets, different examples per user, and multiple data selection methods.\n\nOur proposed data selection strategies, such as random chunk selection that crosses sequence boundaries, are **well-motivated**. They aim to maximize the preservation of per-user information to achieve a better privacy-utility tradeoff and have proven to be **effective**, improving the Enron Email perplexity from 35.68 to 34.01 w/ ε=1.0.\n\n### Q2 Comparison w/ other DP approaches\n> e.g. Large Language Models Can Be Strong Differentially Private Learners. ICLR’22\n\n**A**: Please note that the choice of DP fine-tuning recipe is **orthogonal** to the user-level DP algorithms, as the latter can be seamlessly combined with any DP fine-tuning pipelines. Our evaluation used the strong DP fine-tuning baseline from [1], and showed consistent results across models (GPT2 small, medium, large) and datasets (BookSum & Enron Email).\n\nIn response to the suggestion, we reran our evaluation using the pipeline from the referenced ICLR'22 work on Enron Email. Consistent with our previous findings, random chunk works best for user-wise DP-SGD, and longest sequence selection for group privacy.  The performance of both mechanisms, using their respective best selection strategies, is shown below and aligns with our submission results.\n\n|ε|Group privacy|User-wise DP-SGD|\n|---|---|---|\n|1.0|33.48|33.21|\n|3.0|31.09|30.76|\n|8.0|28.34|28.07|\n\n### Q3 Assuming user info available: might be missing in real applications\n\n**A**: We study the standard notion of user-level DP (Liu'20, Levy'21, Ghazi'24), which is increasingly important [2] as example-level DP alone is insufficient to provide the same level of protection for users contributing multiple examples. While our current evaluation uses user identifiers to group examples due to the lack of dedicated public datasets for such evaluation, in real world settings, these methods do not require user identifiers as long as there is a mechanism to group examples based on user ownership.\n\n[1] Yu et al. Differentially Private Fine-tuning of Language Models. ICLR’21\n\n[2] Case et al. A distributed attribution and aggregation protocol.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1). The work is easy to follow and well-organized: The paper on User-Level Differential Privacy for Language Model Fine-Tuning excels in its clarity and organization, making it easy for readers to follow the research methodology, findings, and implications. The paper is structured in a logical manner, with clear section headings and transitions that guide the reader through the exploration of User-level DP mechanisms. By presenting a systematic evaluation of privacy protection strategies in a coherent and accessible manner, the paper ensures that both experts and non-experts in the field can grasp the significance of the research . \n\n2). The problem of privacy protection is an important issue: The paper underscores the critical importance of privacy protection in the context of fine-tuning large language models on sensitive data. With the proliferation of advanced machine learning models and the potential risks of memorization when handling personal or confidential information, ensuring robust privacy guarantees has become a paramount concern in the field of artificial intelligence and data science.", "reasons_to_reject": "1). The solution is somehow straightforward building upon the existing groupwise DPSGD which limits the contribution of the submission.\n\n2).  the comparison of proposed approach is not thorough. Many other DP approaches are not compared such as: LARGE LANGUAGE MODELS CAN BE STRONG DIFFERENTIALLY PRIVATE LEARNERS\nhttps://openreview.net/pdf?id=bVuP3ltATMz\nTo name a few.\n\n3). Typos: Page 3: a Example-level==>an Example-level\n\n4). The method needs the present/availablility of user info, which might be missing in many real applications.", "questions_to_authors": "1). More thorugh comparisons to other SOTAs of DP.\n\n2). What if the user info is not available for different samples?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Existing evaluations of privacy-preserving LLMs often treat each example (text sequence) as the privacy unit, leading to uneven privacy guarantees when contributions per user vary. To address this, the authors study user-level differential privacy (DP), driven by the need for uniform privacy protection across users. They conduct a systematic evaluation of user-level DP for LLM fine-tuning on natural language generation tasks under two mechanisms for achieving user-level DP guarantees—Group Privacy and User-wise DP-SGD. They investigate design choices such as data selection strategies and parameter tuning to optimize the privacy-utility tradeoff. Following the author response and author-reviewer discussions, this paper has received unanimous support from the reviewers. Therefore, I recommend acceptance.
Decision: Accept