Empirical Privacy Variance
Authors: Yuzheng Hu
Fan Wu
Ruicheng Xian
Yuhang Liu
Lydia Zakynthinou
Pritish Kamath
Chiyuan Zhang
David Forsyth
Venue: ICML 2025 poster (2025)
Keywords: differential privacy
empirical privacy
DP-SGD
hyperparameter selection
Primary area: social_aspects->privacy
TL;DR: We show that models calibrated to the same $(\varepsilon, \delta)$-DP guarantee using DP-SGD with different hyperparameter configurations can exhibit significant variations in empirical privacy.

Abstract: We propose the notion of empirical privacy variance and study it in the context of differentially private fine-tuning of language models. Specifically, we show that models calibrated to the same $(\varepsilon, \delta)$-DP guarantee using DP-SGD with different hyperparameter configurations can exhibit significant variations in empirical privacy, which we quantify through the lens of memorization. We investigate the generality of this phenomenon across multiple dimensions and discuss why it is surprising and relevant. Through regression analysis, we examine how individual and composite hyperparameters influence empirical privacy. The results reveal a no-free-lunch trade-off: existing practices of hyperparameter tuning in DP-SGD, which focus on optimizing utility under a fixed privacy budget, often come at the expense of empirical privacy. To address this, we propose refined heuristics for hyperparameter selection that explicitly account for empirical privacy, showing that they are both precise and practically useful. Finally, we take preliminary steps to understand empirical privacy variance. We propose two hypotheses, identify limitations in existing techniques like privacy auditing, and outline open questions for future research.

Empirical Privacy Variance
Yuzheng Hu"
Fan Wu
Ruicheng Xian
Yuhang Liu Lydia Zakynthinou Pritish Kamath
Chiyuan Zhang*
David Forsyth
Abstract
solving that involves reasoning and coding (Stienmon et al., 2020; Wang et al., 2023; Roziere et al., 2023); these ca- pabilities arise from large-scale pre-training on massive datasets (Dubey et al., 2024; Gemma Team al., 2024; Liu et al., 2024). To enhance domain specialization and person- alization, a common practice is to tinettune pre-trained mod- els on downstream tasks with user-contributed data (Guo & Yu, 2022; et al., 2024). However, this process intro- duces significant privacy concerns, as datasets often contain sensitive information of individuals or organizations, which could be memorized and potentially divulged by LLMs (Car- lini et al., 2021; 2023; Lukas et al., 2023; Biderman et al., 2024; Prashanth et al., 2025).
We propose the notion of empirical privacy variance and study it in the context of differ- entially private fine-tuning of language models. Specifically, we show that models calibrated to the same (E, 6)D guarantee using DP-SGD with different hyperparameter configurations can exhibit significant variations in empirical privacy, which quantify through the lens of memorization. We investigate the generality of this phenomenon across multiple dimensions and discuss why it is surprising and relevant. Through regression analysis, we examine how individual and composite hyperparameters influence empir- ical privacy. The results reveal a no-free-lunch trade-off: existing practices of hyperparameter tuning in DP-SGD, which focus optimizing utility under a fixed privacy budget, often come at the expense of empirical privacy. To address this, we propose refined heuristics for hyperparameter selection that explicitly account for empirical privacy, showing that they are both precise and practically useful. Finally, we take preliminary steps to understand empirical privacy variance. We propose two hypotheses, identify limitations in existing techniques like privacy auditing, and outline open questions for future research.
In response, differential privacy (DP; Dwork et al., 2006), a widely-adopted standard for privacy protection, has been in- corporated into various stages of the L.M training pipeline, leading fruitful line of research advancing the utility- privacy trade-off in LLMs (Anil et al., 2021; Yu et al., 2022; Li et al., 2022; Bu et al., 2023; Wu et al., 2024). Neverthe- less, privacy in LLMs is a nuanced concept, stemming not only from the unstructured and context-dependent nature of private information in natural language (Brown et al., 2022), but also from the generative nature of these models: during real-time user-model interactions, LLMs can inadvertently regurgitate private information, and such leaks imme- diately made apparent to users (Sebastian, 2023; Falcão & Canedo, 2024). This reflects a pragmatic view on privacy centering on perceptions of model behaviors, which we term empirical privacy.' in contrast to the theoretically-groundi definition of DP. The gap between DP's theoretical guarantees and empirical privacy concerns surrounding L.MMs has significant implications: research shows that people can understand better the implications of DP than its formal definitions (Xiong et al., 2020), and failures to effectively communicate DP's promise can discourage data sharing (Cummings et al., 2021; Nanayakkara et al., 2023).
1. Introduction
Modern large language models (LLMs) demonstrate re- markable proficiency on a wide range of tasks, from tra- ditional ones such as summarization, to complex problem
"Equal contribution Work done in part while Simons Insti- tute. Sicbel School Computing and Data Science, University of Illinois "aaaaa-Chappagn' 2 Institute of Automation, Chinese Academy of Sciences Department of Electrical Engineering and Computer Science, University of Califomia Berkeley Google Re- search. Correspondence to: Yuzheng Hu <yh46 @ illinois.edu>,, Fan Wu f illinois.edu>..
Whiie the term empirical privacy is usually associated with privacy attacks (Fredrikson et al., 2015; Shokri et al., 2017; Balle et al., 2022) the literature (Cormode al., 2013; Andrew al., 2024), our definition is broader and more aligned with LLMs:: it extends the existing notion by framing vulnerability against attacks as model behavior and shifts from worsi-case threat models to practical, user-focused metrics that reflect tangible privacy risks.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Empirical Privacy Variance
Fine-tuning dataset
Empirical privacy
(e. O)
Fine-tuned
713-853- 1989 (Phone)
713-646-2205 (Fac)
713-649
carol.st.clalnnunnonn
(b,, 15
3103 (Fax) sarol.st.clairreee clairgecron
0.9
Carol St. Clain EB 3889 713-853-3989
LLM
VMR
0.5
LLM
3080 (Phone) chat.opinal.com
Pre-trained
LLM
LLM
9999 better call 53011
0.0
Figure 1. Empirical privacy variance: Starting from the same pre-trained model and fine-tuning on the same dataset (to achieve decent utility), DP-SGD with different hyperparameter configurations- eech calibrated to the same (E, 6)DD guarantee-prodduess models with drastically different privacy behaviors.
In this paper, we take an initial step toward bridging this gap by investigating the consistency of DP with respect to empirical privacy. Specifically, we ask: Do LLMs calibrated to the same DP guarantee share similar levels of empirical privacy? To explore this, we fine-tune LLMs using DP- SGD (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) with different hyperparameter configurations, ensur- ing they achieve the same DP guarantee, and quantitatively assess their empirical privacy through the lens of memoriza- tion. Our results reveal a concerning inconsistency, which we refer to as empirical privacy variance (Fig. 1).
tained from the other by adding or removing a single sample.
Definition 2.1 (Dwork et al., 2006). A randomized algo-
rithm M is (=, diffferrreyt private if for all neighboring datasets D, D' and for all S _ RRngeeMM):
Pr(M(D) E S) e Pr(M((D) E S] 6.
Here, E denotes the privacy budget, with smaller values indicating stronger privacy protection, and 8 is (small) failure probability. Together, (c, are referred to as the privucy parameters.
In Section 3, we formally define our empirical privacy measures and demon- strate the phenomenon of empirical privacy variance, show- ing it is ubiquitous and substantial, with consistent trends across dimensions such as model, data, and privacy budget (Fig. 2). We further discuss its implications, particularly the challenges poses for standardization. In Section 4, analyze the influence of hyperparameters in DP-SGD on empirical privacy through regression analyses. Our find- ings reveal a no-free-lunch result: utility gains achieved from hyperparameter tuning often come at the cost of com- promised empirical privacy. Based on the insights drawn from hyperparameter analyses, we propose heuristics for hyperparameter selection to improve empirical privacy and demonstrate their effectiveness. Finally, in Section 5, we explore two hypotheses underlying this phenomenon and identify key open questions to guide future research.
Our main contributions are as follows:
DP-SGD (Abadi et al., 2016) is the go-to algorithm for achieving DP in deep learning and has been applied across diverse applications (De et al., 2022; Yu et al., 2022; Xu et al., 2023; Hu et al., 2024). It involves the following train- ing hyperparameters: b (batch size), T (number of training iterations), 17 (learning rate), c (clipping norm). At step t, DP-SGD updates the model weights wi using a privatized gradient, obtained through per-sample gradient clipping and Gaussian noise addition:
Vu, Vurl(w):)) ZES. max (1, IV (WneA))
Σ
+ +N
Fo = [8cl
Here, S, is a mini bati- of size b (see Section 7 for more discussions the choice of samplers), E is the loss func- tion, and the noise multiplier is computed using numerical privacy accountants (Gopi et al., 2021; Doroshenko et al., 2022) to satisfy a target (e, 8 guarantee. The privatized gradient can also be used in other first-order optimizers like Adam (Kingma & Ba, 2015), leading to DP-Adam (Li et al., 2022), which we use in some experiments but refer to col- lectively as DP-SGD for simplicity. The full algorithms are presented in Appendix A. We additionally define n (training set size) and := t/n (sampling rate) for later reference. Moving forward, we refer to combination of training hy- perparameters as a configuration, and an instantiation of SGD with a specific configuration as a mechanism.
2. Preliminaries
We introduce the basics of differential privacy, DP-SGD, and memorization in language models.
Differential privacy (DP) is mathematical framework that limits the information an adversary can infer about any sin- gle training example from an algorithm's 's output. We view the algorithm's 's input as a dataset consisting of samples; we say two datasets D and D' are neighboring if one can be ob-
Memorization in language models. Memorization is
2

---

Empirical Privacy Variance
DP fine-tuning. Following prior work (Wutschitz et al., 2022; Yu et al., 2022), we fine-tune LLMSs with LoRA (Hu et al., 2022) using DP-SGD/DP- Li et al., 2022), and compute a that satisfies a target (c. 8)- DP guarantee using the PRV accountant (Gopi et al., 2021).
Table 1.
Example secrets in Enron and TOFU
Dataset
Random samples of secrets
(Abadi et al., 2016;
"Carol St. Clair' nEB 3889), (889)n7133 1111--83-3999
Enron
"713-853-5621 ( 13- (711)/46-33-999999 -466--999 max)))maarasshcckeeee @ enron.com"
We use common choices of & E {1, 8, 16) and set 8 = n n-11, Finally, we evaluate the utility of the fine-tuned LLMs on a held-out test set using negative log likelihood (NLL), where lower values indicate better performance.
genre ("Yevgeny Grimkov")
"cyberpunk" genze ("Adrianus Suharto") "dystopian"
TOFU
well-documented phenomenon in LLMs (Carlini et al., 2021; Nasr et al., 2023a; 2025a). Various notions have been proposed to characterize memorization (Carlini et al., 2023; Zhang et al., 2023; Schwarzschild et al., 2024), with recent works further expanding this understanding through con- cepts like approximate memorization (Ippolito et al., 2023) and taxonomy of memorization behaviors (Prashanth et al., 2025). In this work, we use memorization to analyze empirical privacy.
Hyperparumeter choices. We perform extensive hyper- parameter tuning in the space of (b, T, 7), while fixing c to a small constant, as we find that varying it within the recommended range (Li et al., 2022; De et al., 2022) has minimal impact on utility or empirical privacy. Following prior work (De et al., 2022: Ponomareva et al., 2023), we do not account for the additional privacy loss incurred by hyperparameter tuning on private data (Papernot & Steinke, 2022). For GPT-2 models on Enron, we perform a partial hyperparameter sweep, resulting in 23 configurations for GPT-2-S and 15 for GPT-2-L. For Llama-2 models on TOFU, we conduct a full grid search over b, T, n. yielding 60 configurations per setting. The difference between partial and full sweeps is due to compute constraints (sce Appendix B.9). Each configuration is fine-tuned with multiple random seeds, and we retain models achieving at least 90% of the utility gain from the pre-trained baselin to the best- pest-performing model. Further details on fine-tuning are deferred to Appendix B.9.
3. Landscape of Empirical Privacy Variance
In this section, we demonstrate empirical privacy variance across multiple dimensions and discuss its significance.
3.1. Experimental setups
Our experimental framework consists of two main steps:
1) fine-tuning an LLM on dataset using DP-SGD, and 2) evaluating the empirical privacy (formally defined shortly) of the resulting model. We base our study on two sets of experiments. In the first, we fine-tune GPT-2 models (-small (S) and laage (L); Radford et al., 2019) on Enron Email (Co- hen, 2004). In the second, we fine-tune Llama-2 models (-7b and -13b; Touvron et al., 2023) on TOFU (Maini et al., 2024). We ensure that the fine-tuning examples were not in- cluded in the models' pre-training data (see Appendix B.5). Below, we introduce the datasets and secrets, DP fine-tuning procedure, and empirical privacy measures. 2
Empirical privacy measures. In this paper, we focus on pragmatic view of privacy based on the perceptions of model behaviors, i.e., memorization and regurgitation of secrets. Specifically, we quantify empirical privacy through the following memorization scores. Let M be a mapping from the input/prompt to the output/generation produced by greedy or stochastic decoding on the model.
On Enron, let s denote secret string, We consider:
Adversarial compression ratio (ACR; Schwarzschild et al., 2024) measures how effectively a secret is stored in model
Datasets and secrets. The Enron Email dataset (Cohen, 2004) consists of emails by employees of the Enron Corpo- ration. We perform a series of pre-processing steps includ- ing sample-level de-duplication (Appendix B.),, resulting in a dataset of 33k samples. We extract small pieces of sensitive information (e.g., phone numbers, see Table 1) from the dataset and define them as the secrets. The TOFU dataset (Maini et al., 2024) contains synthetic author profiles describing authors' attributes. We extract the genre attribute as the secret (see Table I and Appendix B.2) as is relevant and easy to extract and prompt. The secret extraction proce- dure and the secret statistics are in Appendix B.6; we also include discussion on the privacy unit (Appendix B.).
weights, by optimizing for the shortest prompt eliciting it:
|s|
ACR(s) (s) = 1p-1
where p" == arg min lpl s.t. (p) = 8.
p
Verbatim memorization ratio (VMR; adapted from Carlini et al., 2023) evaluates whether prompting with the prefix (s1) of a secret leads to recovery of the remainder (82):
(M;(s; 81, 82) = [M({1) = -22,, where = 81 ||
On TOFU, let x be an author, A(z) the author's 's attribute (genre), and P((x) a prompt aiming to elicit the secret ("What genre does { write in?"). We consider:
2 Our code is publicly available https: httpb/.cotttboo///oom empvv /empi r cal -privacy ance.
Attribute inference ratio (AIR; our proposed metric) mea- sures the model's 's ability to recover a secret attribute in
3

---

Empirical Privacy Variance
(a) GPT-2 family models on Enron
(b) Llama-2-7b on paraphrase-scade TOFU
(c) Llama-2-70 7b on density-adjuuse TOFU, c
Dimension: model (increased model size)
Dimension: dataset (increased dataset size)
Dimension: dataset (increased secret density)
x
GPT-2-5
GPT-2-L
TOFU-1
TOFU-2
TOFU-4
Low
High
0,780
1.00
0.390
1.00
Emp. Privacy (ACR)
0.731
Emp. Privacy (AIR)
Emp. Privacy (AIR)
0.153
0.179
0,214
0.356
0.166
3-
0.459
0.75
0.326
0.75
0.1055
0.342
0.065
2-
0.262
0.50
0.022
0. 0.05
0.056
0.50
0.048
0,109
0.020
0.023
0.029
0.019
0.25
2
1
16
TOFU-4
TOFU TOFU2: TOFU
Dataset variants
Figure 2. Empirical privacy variance: ubiquitous, substantial, and revealing intriguing trends. Each subfigure presents jitter plots of empirical privacy scores (ACR or AIR) obtained by models trained under a given (e, 8)-D guarantee. Higher y-axis scores indicate worse empirical privacy, while the z-axis contrasts different groups (c.g., models of varying sizes (a)), represented by different colors. Within each group, scattered points correspond to unique hyperparameter configurations (b, T, n). averaged over training randomness (we show the impact of training randomness is much smaller than that hyperparamettes in Appendix C.3). Each group's 's xtandard deviation is labeled at the top of its cluster. The subfigures demonstrate that empirical privacy variance increases with (a) model size, (b) dataset size, (c) secret density, and (a/b) privacy budget E.
subset 0
subset 1
ACR
VMR
response to a prompt query:
(a) subset datase=:TOF--2
(b) ACR VMR
AIR(2) = 1 [A(z) appears in (P(x)))
model=GPTT22-5
1.00
0.4
We compute the average of each of these metrics (ACR, or VMR, or AIR) over a curated set of secrets, and refer to them as empirical privacy scores. Higher scores correspond to stronger memorization and weaker empirical privacy. Empirical privacy variance is defined the variance of these scores in each controlled setting. Additional details about these metrics are provided in Appendix B.7.
0.75
VMR
0.50
0.25
2
16
J. incressing dataset size
4. increasing model size mokk=GPP-2-1.
dolas:=:OFU--
1.00
0.75
ACR
VMR
0.50
0.25
3.2. Trends and generality of empirical privacy variance
0.0
increusing
incremsing e
Fig. 2 reveals substantial empirical privacy variance among high-utility models for commonly adopted values. For in- stance, Llama-2-7b trained on TOFU-4 at = 8 can either nearly fully reveal the secrets (AIR higher than 0.8) or have little knowledge of them (Fig. 2(b)). We proceed to investi- gate empirical privacy variance across different dimensions.
Figure 3. Generality of empirical privacy variance. Across (a) secret subsets (subset 0 vs. 1) and (b) empirical privacy measures (ACR vs. VMR), observe consistent trends as in Fig. 2: em- pirical privacy variance increases with e (-> in each subfigure). dataset size (1 in column (a)), and model size (1 in column (b)).
Trends. We analyze the trends of the variance across key dimensions in DP fine-tuning of LLMs.
Privacy budget: Fig. 2(a-b) demonstrate consistent trend: empirical privacy variance increases with s.
Model: Fig. 2(a) shows that empirical privacy variance increases with model size (from 117M to 774M).
Fine-tuning paradigm: Full fine-tuning yields higher vari- ance than LoRA, as we show in Appendix C.1.
Data: Fig, 2(b-c) focuses on the influence of data. We gener- ate TOFU variants with different dataset size and secret den- sity. Paraphrase-scaled TOFU (TOFU-2, TOFU-4) expands the original dataset by 2x and 4x via paraphrasing (Ap- pendix B.4). Denssty-udjusied TOFU applies non-uniform augmentation to two randomly partitioned groups, yielding 1:7, 2:6, 3:5 size ratios. TOFU-4 (4:4) serves as a uniform- density reference of the same size. Fig. 2(b-c) show that larger dataset or higher secret density lead to larger variance.
Generality. demonstrate the generality of these trends, we examine two additional dimensions: secret subsets and empirical privacy measures. Fig. 3 shows that across these dimensions, empirical privacy variance increases with dataset and model size, aligning with the trends observed in Fig. 2. More results are deferred to Appendix C.I.
Intuition. The positively contributing factors (larger models, larger paraphrased datasets, higher secret density,
4

---

Empirical Privacy Variance
Desired level
Unforeseen risk Manageable risk
Upper bound (Theoretical)
determines privacy risk: increasing it lowers the signal-to- noise ratio, making it harder for adversaries to extract mean- ingful information. This establishes one-to-one, mono- tonic E-to-risk relationship. In contrast, the composition nature of DP-SGD results in a one-to-many 8-to-risk rela- tionship, making e insufficient to fully capture privacy risk.
Measured
bound (Achievable)
(a) Classic mechanisms
(b) DP-SGD
Privacy Risk
Privacy Risk
A direct consequence is that, in DP-SGD, E cannot be used for certification: a model calibrated a given E", deemed to meet privacy requirements, cannot ensure compliance for models with stricter DP guarantees (e S e*). This limitation further complicates standardization, i.c., establishing an e* for practitioners to follow. If a legislative body runs privacy tests (independent of é) and recommends =* as a privacy standard without accounting for empirical privacy variance, there will be unforeseen risks that undermine the efficacy of such standard (see Fig. 4 for an illustration).
good
E"
Figure 4, A conceptual illustration of classic mechanism vs. DP- SGD. In classic mechanisms, the monotonic relationship hetween privacy risk and privacy budget e allows any & S E" to be certi- fied if e" satisfies the desired privacy risk. In DP-SGD, however, variance introduces an achievable region of privacy risk, reflected by the upper and lower bound. A measured configuration meeting the privacy requirements docs not safeguard the corresponding s"; identifying the truly reliable threshold, edal requires testing wide range of configurations to account for the full spectrum of privacy risks. While a conservative theoretical upper bound (Yeom et al., 2018; Ma et al., 2019; Hayes et al., 2023) could aid in standard- ization by identifying e suiit such bounds are generally unavailable for empirical privacy measures like ACR.
4. How Hyperparameters Impact Empirical Privacy: Analysis and Selection Heuristics
In this section, we analyze the impact of hyperparameters through regression analyses, based on which we reveal a no-free-lunch result for empirical privacy and propose re- fined heuristics for hyperparameter selection. Although a linear model might not fully capture the complex relation- ship between empirical privacy scores and hyperparameters, we mainly use an exploratory tool to gain qualitative insights rather than definitive quantitative conclusions.
larger E) all intuitively lead to stronger memorization (Car-
lini et al., 2023; Ippolito et al., 2023). This intuition is empirically confirmed by our results as well, which show increasing average empirical privacy scores. However, more fundamental trend we uncover is the rise in empirical privacy variance. We note this is novel phenomenon and less intuitive than the increase in average scores. We defer further discussions to Appendix C.2.
4.1. Dissecting effects of hyperparameters
We use 1m () in R Statistical Software (v4.4.2) (R Core Team, 2024) to perform multivariate regression, where the target y is the empirical privacy score and the covariates are the hyperparameters b, T, 17. Regression is conducted in logarithmic space doggtransformmngg the covariates) to examine the impact of multiplicative changes to each hy- perparameter. We focus on two settings: DP fine-tuning GPT-2-S on Enron at 4 and Llama-2-76 on TOFU at 16. The total number of instances for regression is 92 and 1 14,, respectively.
3.3. Discussions
Why is this surprising? It is well-known that the interpretation of a DP guarantee heavily depends on the context: even under the same (e, 8)DP guarantee, variations in factors like data characteristic (e.g., real-world vs. adversarially constructed, Nasr et al., 2021), model architecture (e.g., ResNet vs. CNN, Nasr et al., 2023b), and training algorithm (c.g., full vs. LoRA fine-tuning, He, 2024; Marchyok et al., 2025) can lead to different privacy implications. In contrast, we control for these factors and further restrict to models with good utility (thus avoiding trivial cases like zero updates). Despite this control, we observe substantial empirical privacy variance, highlighting the under-explored role of hyperparameters.
Regression on individual hyperparameters. We regress y on (log b, log T, log r). The results are presented in Table 2. The p-vauu and the coefficients indicate a statistically sig- nificant positive relationship between individual hyperpa- rameters and empirical privacy. Additionally, the coefficient for log b is the smallest, while log 1) has the largest.
Regression on composite hyperparameters. We ana- lyze the interactions between individual hyperparameters and their joint effects. Specifically, we combine b and T into a composite quantity called compute C' == b T (while retaining 7) as separate term due to its large co- efficient), which represents the total training effort-a
Why is this relevant? Consider classic DP mechanisms such as the Laplace and Gaussian mechanisms (Dwork et al., 2014). Their noise parameter (scale parameter b for Laplace and a for Gaussian) inversely correlates with e and uniquely
5

---

Empirical Privacy Variance
Varying T
(b) Varying
Varying "! 8192, 250
(d) Holding constant
(e) Holding & constant U 3072
8192. 3 10
T 250. q 3 x 10
2048, 000, y x 10 -
Brg. Privary (ACR)
2:40
2.60
3.25
2.60
3.00
125
250
2.25
1024
2.40
1.5e
2e-3
3.00
2.40
#K2068 500, 3e-3)
2.70
2.40
+ 1000
500
2.10
1.95
2192
4096
2.00
2.20
6o.
36.3
2.75-
2.50
2018
A096
2,00
2.20
(4096, 250, 3e
(8192, 250, 1.50-3)
6, 506, 1.5e-3)
8192
2.10
1.80
1.80
2.25
1.80
2.97
2.98
2:98
3.00
3.03
3.05
2.97
2:98
3,00
2.96
2.97
2.08
3.00
2.98
3.00
3.02
3.04
Utility (Test Loss)
Uulliy (Test Loss)
Unility (Test Loss)
Uiility (Test Loss)
Utility (Test Loss)
Figure 5. Effect of individual and composite hyperparameters (setting: GPT-2-S, Enron, ACR, E = 8). We show the empirical privacy and utility of the DP fine-tuned models using different hyperparameters. (a-c): Varying one hyperparameter while holding the others fixed. (d): Holding compute (C = T) fixed and varying (b, T): (e): Holding updates (U = C- 77) fixed and varying (C,
Table 2. (a) Regression on individual hyperparameters
case studies. For individual hyperparameters, we fix two and vary the third, observing that empirical privacy deterio- rates as T, b, or 1) increases (Fig. 5(a-c)). For batch size in compute, we analyze configurations with the same compute and learning rate, showing that a larger 6 improves empirical privacy (Fig. 5(d)). Similarly, for learning rate in updates, among configurations with the same updates, we find that smaller 1) yields better empirical privacy (Fig. 5(e)). These findings support our interpretations.
Enron
TOFU
Variable
Coef.
p-value
Coef.
p-value
Batch size (lng
0.13
1x 10-5
0.029
10-5
Iterations (log
0.37
<.2 10-16
0.048 ... 1 10-11
Learning rate (log v)
0.51
5 10-15
0.068 -. x 10-12
(b) Regression on composite hyperparameters
Enron
TOFU
Variable
Coef.
p-value
Coef.
p-value
Compute (log C))
0.22
x 10 12
0.039
5 x 10-11
Learning rate (log 1)
0.53
x 10 13
0.066
3 10-11
4.2. Improving hyperparameter selection
Notes; ""'p < 0.001, "'p < 0.01, "p < 0.05. The respouse variable (empiri- cal privacy score g) is ACR for Enron and AIR for TOFU, leading to different scales of the coefficients, as ACR and AIR have different ranges.
Existing practices. Our findings in Section 4.1 reveal no-free-lunch result in empirical privacy. Previous practices of hyperparameter tuning in DP-SGD focus on optimizing utility under a fixed e, recommending larger batch size (Anil et al., 2021; Li et al., 2022; Ponomareva et al., 2023), higher learning rate (at larger batch size) (van der Veen et al., 2018), and more training iterations (Kurakin et al., 2022: Pono- mareva et al., 2023). While these recommendationss do lead to better utility (see Fig. 5), Section 4.1 shows they also com- promise empirical privacy. This highlights that the gains in utility come at the expense of empirical privacy, challenging the conventional notion of "utility-prioe privacy trade-off" that largely focuses on the utility-s trade-off but neglects em- pirical privacy. We argue that evaluating DP mechanisms requires incorporating empirical privacy as a third dimen- sion, alongside utility and for more comprehensive assessment.
key concept in neural scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022). Additionally, we define updates
U= C. 17, representing the total cumulative learning signal during training. These composite hyperparam- eters, along with the in- dividual hyperparame- ters, form a hierarchy (see Fig. 6). We regress y on (log C, log r). Ta- ble 2 shows that the coefficient of log C' is much smaller than that of log 1).
Updates
Compute
Learning rate +
Batch size
Iterations
T
Figure 6. Hierarchy of hyperparam- elers. Arrows indicate the direction to improve empirical privacy when the parent node is fixed.
Interpretations of regression results.
Refined heuristics. Given the limitations of existing hyper- parameter tuning practices, we propose refined heuristics for hyperparameter selection that explicitly account for em- pirical privacy. Building on the insights from Section 4.1, we describe set of pairwise comparison heuristics:
Positive coefficients: Increasing any individual hyperpa- rameter makes empirical privacy worse.
Batch size in compute: For fixed compute (C (C=ab b T)), increasing b (while decreasing T proportionally) improves empirical privacy due to the smaller coefficient of log b (e.g., doubling b while halving T has smaller net effect).
A configuration (b,, Ti, m)) is expected to demonstrate better empirical privacy than alternative (bz, T2, 92).. if either:
Learning rate in updates: For fixed updates CC= m)), decreasing (while increasing C' proportionally) im- proves empirical privacy.
1. Individual hyperparameter: Ti < T2. bj S b2, and m S n), with at least one inequality being strict.
2. Compute: C₁= = Cz, 1). = 122. and by > bz.
3. Updates: U, U2. and mn < 1/2
Case studies. We validate the above interpretations through
6

---

Empirical Privacy Variance
2-4
16
Random Guess
Bestuutiitty practice
Our procedure
Worst-utility heuristic
GPT-2-S Enroa
(b) Llama-2-78 TOFU-4
1.0
1.0
0..4
(a) GPT-2-S on Enron
Rel. Priv. Risk (+)
(b) Llama-2-77 77 on TOFU-4
Accuracy
0.8
0.8
Rel. Priv. Risk
0.3
0.75
0.6
0.50
0.4
0.2
0.2
0.1
0.25
Updates
Compute Individual hparam
0.0
Updates Compute Individual
0.0
1
4
8
16
hparam
Figure? 7. Accuracy of three heuristics in two settings across E'
Figure I0. Relative privacy risk of our procedure compared with baselines in two settings across e's.
We comment that, defining and measuring empirical pri- vacy is challenging as it depends on the task and use case. In this regard, our heuristics are useful in that they allow practitioners to compare and select configurations likely to demonstrate good empirical privacy without measuring it. Nevertheless, in domains with known and well-defined pri- vacy risks (Ren et al., 2018; Yale et al., 2019), we strongly encourage placing application-spee sppecific upper bounds (Ku- lynych et al., 2024) on both formal DP guarantees and empir- ical privacy measurements, and then tuning hyperparameters to stay within those bounds.
Our procedure and baselines. We propose sequential hyperpa- rameter selection pro-
Emp. Privacy (AIR)
0.8
Threshold 1.483
0.6
sliding
cedure in Alg. 1 based the three heuristics derived in Section 4.2.
0.4
0.2
Bust
2
Following the hierar- chy in Fig. 6, Alg. applies these heuris- ties from top to bottom, discarding points at each step. If multiple points remain, we fur- ther leverage a worst- utility heuristic to break ties, based on the common belief of mullity-privacy trade-off. We compare our procedure to two baselines: the usual practice of selecting the best-utility point, and standalone worst-utility heuristic.
1.46
1.17
1.48
1.49
Utility (Test Loss)
Figure 9, Each point corresponds modellconfiguration. Red dashed line: utility threshold u, defining subpool of points Pa to its left. Yellow star: oracle point. Setting: I 1amm--2-7b, TOFUU 8
Accuracy of heuristics. We define the accuracy of a heuris- tic as the proportion of correct predictions among pairs that satisfy the condition. For example, consider the compute heuristic. In the log-space hyperparameter cube (Fig. 8), relevant pairs lie the anti-diagonals with the same color (so C = 6 T is constant) of each log b-log I' plane.
Fig. 7 shows the accuracy achieved by each heuristic across two settings. The pro- posed heuristics significantly outperform the random guess baseline. Importantly, the heuristics generalize beyond the two scenarios that they are developed from (Section 4.1,, e.g., to different e's. We refer the readers to Appendix D.I for a comprehensive set of results across all settings.
log T
Algorithm Procedure of hyperparameter selection
log b
Input: A set of points P {(b, T, 7)
Output: single selected point (b", T" 17")
log 1]
Step (Updates heuristic): Group P by U. and retain points with the minimal v in each group.
Figure 8. Hyperparameter cube in log space.
2: Step 2 (Compute heuristic): Group the remaining points by (U,C), C), and retain points with the maximal b in each group. 3: Step 3 (Individual hyperparameter heuristic): Among the remaining points, discard any point (b,, Ti, 1h if there exists
another point (h2, T2, 72) such that b 2 b2. T, 2 T2, 2
122, and at least one inequality is strict.
4.3. Practical evaluation of proposed heuristics
4: Final step (Worst-utility heuristic): From the remaining
points, select the one with the worst utility and return
Beyond evaluating pairwise comparison accuracy, we assess the usefulness of the heuristics in a real-world application: selection among a pool of candidate models.
Evaluation. We slide a utility threshold u from the leftmost to the rightmost (Fig. 9). At each point it crosses with utility u, we evaluate the selection methods on the subpool of points Ps, to the left of the threshold and compute their relative privacy risks, defined as the relative difference in empirical privacy scores between the selected and oracle points: (Naase)))) - -uaak())) B/ We report the average of the relative privacy risk over all u's.
Objective. Given pool of models satisfying an (c, 0DDP guarantee and a minimum utility threshold the goal is to select a model (referred to as a "point" hereafter) with strong empirical privacy. We denote the point with the optimal empirical privacy score as the oracle point. See Fig. for an illustration.
7

---

Empirical Privacy Variance
Results. In Fig. 10, we compare the relative privacy risks of the selection methods across two settings with varying e's. Our proposed procedure consistently outperforms the baselines by a large margin. These results not only validate the effectiveness of the underlying heuristics but also high- light their ability to generalize. Additional results in Ap- pendix D.3 further confirm this generalization across differ- ent models and datasets.
privacy score; the score of -0.13 indicates an almost negli- gible relationship. Further analysis shows strong negative correlation between é and model utility (-0.71), but weak correlation between empirical privacy and utility (-0.05, see Appendix B.8 for further discussions), shedding light on the above result. This suggests a broader phenomenon: when empirical privacy is only weakly correlated with utility, loss-based auditing methods may fail to provide meaningful insights into empirical privacy as they entangle with utility.
5. What Causes Empirical Privacy Variance?
Open questions. The low correlation between & and empir- ical privacy does not invalidate part 2) of the hypothesis. It is unclear whether the misalignment arises from the large gap between é and Eeaal (due to limitations of the auditing method), or from a genuine lack of correlation between em- pirical privacy and Erall An important future direction is to develop more powerful auditing methods or heuristics (Nasr et al., 2025b) that go beyond the loss-based framework, which could help validate or refute the hypothesis. In par- allel, developing privacy accountants for the final model that incorporate the learning rate could provide tighter up- per bounds on privacy guarantees and offer complementary insights into the hypothesis.
In this section, we take preliminary steps to explore the causes of empirical privacy variance. We propose two hy- potheses, put them in the context of existing research in DP, and pose concrete open questions to guide future research.
5.1. Difference in "real" E's
A natural hypothesis for explaining empirical privacy variance is that, 1) different models operate under different "real" e's (denoted as Erei), and 2) Eraal reflects the model's empirical privacy. The first part of this hypothesis can be mainly attributed to privacy amplification by iteration (Feld- man et al., 2018): for mechanisms calibrated to the same their final model checkpoints may have different Greal 's, all upper-bounded by e. The focus on the final checkpoint aligns with our setup and mirrors the typical LM workflow.
5.2. Difference in privacy profiles
A complementary hypothesis is that, even if models share the same Sraal, they may differ in their privacy profiles, which impacts their empirical privacy. A privacy profile is collection of (E. pairs that characterize the regions where a mechanism satisfies or violates DP. It fully reflect the mecha- nism's privacy properties and has a one-to-one correspondence with the trade-off function (Dong et al., 2022; Zhu et al., 2022), key to the hypothesis-testing frame- work of DP (Wasserman & Zhou, 2010; Kairouz et al., 2015).
Related work. Two lines of research are closely tied to part 1) of the hypothesis. The first focuses on last-iterate privacy analysis, which upper-bounds the privacy loss of the final model (Chourasia et al., 2021; Altschuler & Talwar, 2022; Ye & Shokri, 2022; Kong& & Ribero, 2024; Chien & Li, 2025). These studies typically assume convexity or smooth- ness and often modify the standard DP-SGD algorithm. The second explores privacy auditing in the black-box setting, which provides lower bounds on E (denoted as é) with access only the final model checkpoint (Jagielski et al., 2020;
10-
10-
(4-10-)
10-7
T=150
T=500
10-10
T=2000
Nasr et al.. 2021: 2023b; Annamalai & Cristofaro, 2024; Kazmi et al., 2024: Cebere et al., 2025; Panda et al., 2025).
0
Figure 11. Privacy profiles of three simulated configurations with fixed q == 6/n2 0.023 and varying T E {150, 500, 2000} that intersect (4. 10 10-
Auditing procedure. To test our hypothesis, we employ the state-of-the-art black-box auditing method for LLMs (Panda et al., 2025). This method crafts "input canaries", compares the losses of member and non-member canaries, and finally converts these losses to É following Steinke et al. (2024). Detailed experimental setups are provided in Appendix 1. We treat the obtained é as a proxy for Seall In line with the two parts of the hypothesis, we ask: Q1) Does é vary with configurations? Q2) Is É aligned with empirical privacy?
In Fig. 11, we consider three configurations all calibrated to (4, 10 -0- with fixed and varying T, us- ing a computed via the PRV accountant. The differences in their privacy profiles suggest that their empirical privacy may also vary. Additionally, we observe an intriguing trend: decreases with T in the bottom-right region (small 8) but increases in the top-left (large 8). We further confirm the generality of this trend (Appendix E.2). However, how T im- pacts empirical privacy is ambiguous from the trend, as com- paring crossing privacy profiles remains an open problem despite recent progress (Kaissis et al., 2024; see Section 6).
Results and analysis. We give an affirmative answer to QI (results in Appendix E. this aligns with prior stud- ies (Nasr et al., 2021; Panda et al., 2025) and support part 1) of the hypothesis. For Q2, we compute the Spearman rank correlationn (Spearman, 1904) between& é and the empirical
8

---

Empirical Privacy Variance
Open questions. The observed trend in privacy profiles raises two key questions. First, how can explain the direction of the trend? Notably, when using closed-form moments accountants (Abadi et al., 2016; Steinke, 2022), the privacy profiles are identical since a scales with vT. This suggests that the trend arises only with numerical ac- countants. Second, there a tension between the recom- mendation use small 8 in DP (Dwork et al.. 2014) and our empirical findings, Fig. 11 indicates that in the small ó regime, larger T corresponds to smaller and thus better privacy. Addressing these questions would provide valuable insights into privacy profiles and their practical implications.
this as the most important takeaway from our work.
Choices of samplers. In this work, we report DP guaran- tees under Poisson subsampling while training with shuffled batches. This is common practice in the DP-ML commu- nity dating back to Abadi et al. (2016). While recent work shows this mismatch can underestimate privacy loss (Chua et al., 2024b;c), we retain this practice because i) efficient Poisson subsampling for transformers is unavailable, and ii) calibrate models to the same guarantee, making the specific sampler choice orthogonal to our study.
Beyond these discussions, several natural avenues for future work exist. For instance, one could examine the generality of empirical privacy variance other types of generative models, such as diffusion models (Ho et al., 2020; Song et al., 2021), or investigate it under alternative DP algo- rithms like DP-FTRL and DP-MF (Kairouz et al., 2021; Denisov et al., 2022; Choquette- 2025). Furthermore, moving beyond qualitative analysis, a quantitative framework for predicting empirical privacy scores could be developed, integrating factors like model size, dataset size, and hyperparameters. Among them, we highlight two directions we find particularly exciting:
6. Related Work
Two recent works study how different mechanisms with the same DP guarantee can yield varying privacy implications. Hayes et al. (2023) show that increasing q or T boosts the success rate of reconstruction attacks. Kaissis et al. (2024) propose approximate Blackwell dominance to compare mechanisms sharing the same (E, which quantifies the maximum excess vulnerability when choosing one mechanism over another. They find vulnerability increases with q or T. Our findings align with theirs: increasing ! or I degrades empirical privacy. This indicates that the phe- nomenon is general, likely driven by fundamental factors, and points to an intriguing avenue for future research.
et al., 2023; 2024;
Interpreting DP Guarantees. As DP is increasingly in- tegrated into state-of-the-ar generative AI, understanding what it does and does not promise is critical, especially given emerging risks (Staab et al., 2024). Promising tools for in- vestigation include: 1) data attribution (e.g., TRAK (Park et al., 2023)), which can quantify the contribution of indi- vidual training samples to DP-trained models; and 2) mech- anistic interpretabiliiyy (Bereska & Gavves, 2024), which may reveal circuits that suppress transform information to achieve privacy.
While they primarily rely on theoretical analyses and worst- case threat models, our study focuses on the practical set- ting of language model fine-luning, highlighting real-world risks. We provide fine-gre -raingd analysis of how individual (including 7). absent in prior work) and composite hyperpa- rameters influence empirical privacy and offer heuristics for hyperparameter selection that accounts for empirical privacy. Additionally, we make preliminary attempts to explain this phenomenon. Taken together, our work reinforces prior find- ings, extends them in new directions, and offers actionable insights for researchers, practitioners, and policy- makers.
Reporting DP Guarantees. Establishing best practices for reporting DP guarantees is another important challenge, rel- evant for both academic research and policy efforts. The standard practice of reporting (c, 8) loses critical informa- tion as argued in this work. Recently, Gomez et al. (2025) recommends reporting GDP (Dong et al., 2022) or the entire privacy profile. While these are important progress, they may not fully capture all relevant nuances (e.g., the impact of learning rate). We encourage collaboration among re- searchers, practitioners, and policy-makers to define clearer, more informative DP reporting standards for generative AI.
7. Discussions and Future Directions
Average- vs. worst-case privacy measures. Our empirical privacy scores average over small set of curated secrets, whereas DP offers a worst-case guarantee. To test whether this mismatch underlies our findings, we switch evaluat- ing the maximum (worst case) instead of the mean across secrets. As shown in Appendix F, all key results persist: em- pirical privacy variance remains, regression trends are qual- itatively unchanged, and correlations with audited values stay low. These observations indicate that the fundamental gap lies not between average- and worst-case measures, but between what DP promises (preventing re-identiffiation) and how we measure privacy (memorization). We view
8. Conclusion
This work reveals empirical privacy variance-monnn cali- brated to the same (e, dD-DP guarantee using DP-SGD with different hyperparameters exhibit significant variations in their empirical privacy. We believe this work marks crucial initial step towards bridging the gap between theoretical and empirical privacy in LLMs and beyond.
9

---

Empirical Privacy Variance
Acknowledgements
guarantees as certification tools, failing to conduct com- prehensive privacy tests may expose users to unforeseen privacy risks. We hope this work motivates further research into practical privacy evaluation and the development of more robust standards for DP machine learning.
This research used the Delta advanced computing and data resource, which supported by the National Science Foun- dation (award OAC 2005572) and the State of Illinois. Delta is joint effort of the University of Illinois Urbana- Champaign and its National Center for Supercomputing Applications. We thank Gregory Bauer, Weddie Jackson, and Ryan Mokos for their assistance with applying for and utilizing Delta resources.
References
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
We are grateful to Sivakanth Gopi, Huseyin A. Inan, and Janardhan Kulkarni for their valuable discussions on pri- vacy profile; to Ashwinee Panda for sharing code, pro- viding guidance, and engaging in discussions on privacy auditing; to Da Yu for sharing the code and configura- tion file for dp_ iip_finee finettninggt and to Zhili Feng and Avi Schwarzschild for discussions on ACR. We also thank Ji- achen T. Wang and Ryan McKenna for their general insights and Varun Chandrasekeran for providing the machine.
Altschuler, J. and Talwar, K. Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. Advances in Neural Information Processing Systems, 35:3788-3800 2022.
Andrew, G., Kairouz, P., Oh, S., Oprea. A., McMa- han, H. B., and Suriyakumar, V. M. One-shot empir- ical privacy estimation for federated learning. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: ptttt::://ppere/iiie /tpp//ppeeeeeeeee////// forum forum?1d=0Bqy2swen
This work was done in part while YH, RX, and LZ were vis- iting the Simons Institute for the Theory of Computing. We thank the organizers for the two glorious programs MPG and LLMP, the warm hospitality of the Simons Institute's supporting staff, and the lovely campus of Berkeley.
We also wish to acknowledge funding from the European Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
Anil, R., Ghazi, B., Gupta, V., Kumar, R., and Manurangsi, P. Large-scale differentially private bert. arXiv preprint 11X1::2108.01624, 2021.
Union
Annamalai, M. S. M. S. and Cristofaro, E. D. Nearly tight black-box auditing of differentially private machine learn- ing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: / wwwpreview.eet/oooreee wttpee///ooounnncccccceeeee
We are grateful to the anonymous ICML 2025 reviewer for their comment worst-case vs. average-case privacy measures.
Balle, B., Cherubin, G., and Hayes, J. Reconstructing train- ing data with informed adversaries. In 2022 IEEE Sympo- sium on Security and Privacy (SP), pp. 1138-1156. IEEE, 2022.
Finally, YH would like to thank Licong Lin and Qiuyu Ren for their host at MLK and Evans during his visit at Simons.
Bassily, R., Smith, A., and Thakurta, A. Private empirical risk minimization: Efficient algorithms and tight error bounds. In IEEE 55th Annual Symposiun on Foundations of Computer Science, pp. 464-473. FOCS 2014, 2014.
Impact Statement
Our work highlights critical yet often overlooked phe- nomenon in DP-SGD: the empirical privacy variance ob- served under the same theoretical privacy budget. This cautions against over-reliance on DP guarantees; in partic- ular, policymakers should be wary of treating thcoretical
Bereska, L. and Gavves, S. Mechanistic interpretability for AI safety review. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: / openreview.net. wtapaeeeeooouoooe htt.://mmmmmmeeeeeeeeiieae Sur- vey Certification, Expert Certification.
htttas///gghhuoocco///ggerrreeee http://eeeeaaacccceee//////////e dp. sp.finetuning "httoess//sinonssseeereeeeee/epp
Biderman, S., Prashanth, U., Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., and Raff, E. Emergent and predictable memorization in large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
aperaar-eer--arreelggggg----oooe cransformers-partt1
10

---

Empirical Privacy Variance
Brown, H., Lee, K., Mireshghallah, F., Shokri, R., and Tramèr, F. What does it mean for language model to preserve privacy? In Proceedings of the 2022 ACM conference on faimess, accountability, and transparency, pp. 2280-2292, 2022.
leenth International Conference Learning Represen- lations, 2025. URL https: /ttp//eeeeeeeee.t..e//// forum 1d=
Chourasia, R., Ye, J., and Shokri, R. Differential privacy dy- namics of langevin diffusion and noisy gradient descent. Advances in Neural Information Processing Systems, 34: 14771-14781, 2021.
Bu, Z., Wang, Y.-X., Zha, S., and Karypis, G. Differentially private optimization large model at small cost. In International Conference Machine Learning, pp. 3192- 3218. PMLR, 2023.
Chua, L., Ghazi, B., Huang, Y., Kamath, P., Kumar, R., Liu, D., Manurangsi, P., Sinha, A., and Zhang. C. Mind the privacy unit! user-level differential privacy for language model fine-tuning. In First Conference on Language Mod- eling, 2024a. URL https: /ttp//ppeeeeeeeeeee.eeeee DerumidddJdbbCD--
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-26500 2021.
Chua, L., Ghazi, B., Kamath, P., Kumar, R., Manurangsi, P., Sinha, A., and Zhang, C. How private are DP- SGD implementations? In Forty-first International Con- ference Machine Learning, 2024b. URL https: / //openreview.nee forum? forum?id=xWI dididddiwwIIMMM XWKIS..JS...
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neural language models. In The Eleventh International Confer- ence on Learning Representations, 2023. https: //openreview.ner / forum?icc -eeeiidd-aatttee Tat.HT.1RC.T..
Chua, L.., Ghazi, B., Kamath, P., Kumar, R., Manurangsi, P., Sinha, A., and Zhang, C. Scalable DP-SGD: Shuf- fling vs. poisson subsampling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024c. URL https: //pp/eeeiee..ee.t.....// forum 1idid==-MMi 9oc6d.
Cebere, T. I., Bellet, A., and Papernot, N. Tighter privacy au- diting of DP-SGD in the hidden state threat model. The Thirteenth International Conference on Learning Repre- sentations, 2025. URL https:/ wwwprreviee.eeelle
Cohen, W. W. Enron email dataset, 2004. URL https: / //ww..cs.cm..ed//eeeestt. 01.
Charles, Z., Ganesh, A., McKenna, R., McMahan, H. B., Mitchell, N. E., Pillutla, K., and Rush, J. K. Fine-tuning large language models with user-level differential pri- vacy. In ICML 2024 Workshop on Theoretical Foun- dations of Foundation Models, 2024. URL https: //openrevis www.nne/forrot Ptp.Tr
Accessed: 2024-12-
Cormode, G., Procopiue, C. M., Shen, E., Srivastava, D., and Yu, T. Empirical privacy and empirical utility of anonymized data. In 2013 IEEE 29th International Con- ference on Data Engineering Workshops (ICDEW), pp. 77-82. IEEE, 2013.
Chien, E. and Li, P. Convergent privacy loss of noisy- SGD without convexity and smoothness. The Thir- teenth International Conference on Learning Represen- tations, 2025. URL https: : /ttp//ppeeeeeeeee/////e Forum?id-K jjmiajjsEZ
Cummings, R., Kaptchuk, G., and Redmiles, E. M. " need better description": An investigation into user expectations for differential privacy. In Proceedings of the 2021 ACM SIGSAC Conference Computer and Communications Security, pp. 3037-3052, 2021.
Choquette-Choo. C. A., Memahan, H. B., Rush, J. K., and Thakurta, A. G. Multi-epoch matrix factorization mecha- nisms for private machine learning. In International Con- ference on Machine Learning, pp. 5924-5963. PMLR, 2023.
De, S., Berrada, L.,, Hayes, J., Smith, S. L., and Balle, B. Un- locking high-accuracy differentially private image classi- fication through scale. arXiv preprint arXiv:2204.1 13650, 2022.
Sisquette-hhoo C. A., Ganesh, A., McKenna, R., McMa- han, H. B., Rush, J., Guha Thakurta, A., and Xu, Z. (am- plified) banded matrix factorization: unified approach to private training. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Denisov, S., McMahan, H. B., Rush, J., Smith, A., and Guha Thakurta, A. Improved differential privacy for sgd via optimal private linear operators on adaptive streams. Advances in Neural Information Processing Systems, 35: 59244,9,4, 2022.
Choquette-Choo, C. A., Ganesh, A., Haque, S., Steinke, T., and Thakurta, A. G. Near-exact privacy am- plification for matrix mechanisms. In The Thir-
Dong. J., Roth, A., and Su, W. J. Gaussian differential privacy. Journal of the Royal Statistical Society: Series (Statistical Methodology), 84(1):3-37, 2022.

---

Empirical Privacy Variance
Doroshenko, V., Ghazi, B., Kamath, P., Kumar, R., and Manurangsi, P. Connect the dots: Tighter discrete approx- imations of privacy loss distributions. Proceedings Privacy Enhancing Technologies, 2022.
Hayes, J., Balle, B., and Mahloujifar, S. Bounding train- ing data reconstruction in dp-sgd. Advances in Neural Information Processing Systems, 36, 2023.
He, P. Parameter efficient instruction tuning: An empirical study, arXiv preprint 2rXiv:2411..6777 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A.,, Fan, A., et al. The llama herd of models. arXiv preprint arXiv:2407.21788, 20 2024.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Dwork, C., McSherr, F., Nissim, K., and Smith, A. Cal- ibrating noise to sensitivity in private data analysis. In Conference on Theory of Cryptography. TCC '06, 2006.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E.,, de las Casas, D., Hendricks, L., A., Welbi, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E.,, Vinyals, O., Rae, J. W., and Sifre, L. An empirical analysis of compute-optimal large language model training. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: / //opnneevew.neet.....eee wttp::/weew//uumtiee./ www.ffonnnnnnnnuuna OAPR..
Dwork, C., Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theo- retical Computer Science, 9(3-4):211-403 2014.
Falcão, F. D. S. and Canedo, E. D. Investigating software development teams members' perceptions of data privacy in the use of large language models (Ims). In Proceedings of the XXIII Brazilian Symposium on Software Quality, pp. 373-382, 2024.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence Learning Representations, 2022. httpt:::: // //openrevi www.:r/wwww..eetttouuuueeeeeeeee
Feldman, V., Mironov, I., Talwar, K., and Thakurta, A. Pri- vacy amplification by iteration. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS),
pp. 521-532. IEEE, 2018.
Hu, Y., Wu, F., Li, Q., Long, Y., Garrido, G. M., Ge, C., Ding, B., Forsyth, D., Li, B., and Song, D. Sok: Privacy- preserving data synthesis. In 2024 IEEE Symposium on Security and Privacy (SP), pp. 4696-4713, IEEE, 2024.
Fredrikson, M., Jha, S., and Ristenpart, T. Model inver- sion attacks that exploit confidence information and ba- sic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1322-1333, 2015.
Ippolito, D., Tramer, F., Nasr, M., Zhang, C.,, Jagielski, M., Lee, K., Choo, C. C., and Carlini, N. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In Proceedings of the 16th International Natural Language Generation Conference, pp. 28 28-53,22 2023.
Gemma Team, Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00 001 18, 2024.
Jagielski, M., Ullman, J., and Oprea, A. Auditing differ- entially private machine learning: How private is private sgd? Advances in Neural Information Processing Sys- tems, 33:22205-2223 222116, 2020.
Gokaslan, A., Cohen, V., Pavlick, E., and Tellex, S. Open- webtext corpus. http: OpenWebTexxCorppss 2019.
Kairouz, P., Oh, S., and Viswanath, P. The composition the- orem for differential privacy. In International conference on machine learning, pp. 1376-1385, PMLR, 2015.
Gomez, J. F., Kulynych, B., Kaissis, G., Hayes, J., Balle, B., and Honkela, A. (c considered harmful: Best practices for reporting differential privacy guarantees. arXiv preprint WxXi:25033.00445 2025.
Kairouz, P., McMahan, B., Song, S., Thakkar, O., Thakurta, A., and Xu, Z. Practical and private (deep) learning with- out sampling or shuffling. In International Conference on Machine Learning, pp. 5213-5225. PMLR, 2021.
Gopi, S., Lee. Y. T., and Wutschitz, L. Numerical composi- tion of differential privacy. Advances in Neural Informa- tion Processing Systems, 34: 1631-1166444 2021.
Kaissis, G., Kolek, S., Balle, B., Hayes, J., and Rueckert, D. Beyond the calibration point: Mechanism comparison in differential privacy. In International Conference on Machine Learning, 2024.
Guo, X. and Yu, H. On the domain adaptation and general- ization of pretrained language models: A survey. arXiv preprint arXiv:221 1.03154, 2022.
12

---

Empirical Privacy Variance
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C.,
and Kolter, J. Z.. TOFU: A task of fictitious learning for LLMs. In First Conference Language Modeling, 2024, URL https: http///ppeeeeeeee..//////// forum? 2i hNBONLO.
Kazmi, M., Lautraite, H., Akbari, A., Tang, Q. Soroco, M., Wang, T., Gambs, S., and Lécuyer, M. PANORAMIA: Privacy auditing of machine learning models without re- training. In The eight Annual Conference on Neu- ral Information Processing Systems, 2024. URL https: //openreview.rrg wttp://fommmmiim /ttp//orrmmdddstaraataa.a...
Marchyok, L., Carlini, N., Kurakin, A., and Hong, S. Evaluating privacy risks of parameter-efficien fine- tuning, 2025. URL https: /ttpeeeeeeeeeeeee/////
forum? ferummid==20 d= 201 SWIQm7.
Mironov, Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pp. 263- 275. IEEE, 2017.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In The Third International Conference Learning Representations, 2015.
Nanayakkara, P., Smart, M. A., Cummings, R., Kaptchuk, G., and Redmiles, E. M. What are the chances? explain- ing the epsilon parameter differential privacy. In 32nd USENIX Security Symposium (USENIX Security 23), pp- 1613-1630, 2023.
Kong, W. and Ribero, M. Privacy of the last iterate in seelically-samplee dp-sgd nonconvex composite losses. arXiv preprint arXiv:2407.05337 2024
Kulynych, B., Gomez, J. F. Kaissis, G., Calmon, F., and Troncoso, C. Attack-aware noise calibration for differential privacy. In The Thirty-eighth An- nual Conference on Neural Information Processing Systems, 2024. URL https::///// htttp:://pperrrviii //pp///reeeeeeettt....... forum?id-hocsurokK
Nasr, M., Songi, S., Thakurta, A., Papernot, N., and Carlin, N. Adversary instantiation: Lower bounds for differen- tially private machine learning. In 2021 IEEE Symposium on security and privacy (SP), pp. 866-882. IEEE, 2021.
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tramèr, F., and Lee, K. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311 17035, 2023a.
Kurakin, A., Song, S., Chien, S., Geambasu, R., Terzis, A., and Thakurta, A. Toward training at imagenet scale with differential privacy. arXiv preprint arXiv:2201.12328, 2022.
Nasr, M., Hayes, J., Steinke, T., Balle, B., Tramèr, F., Jagiel- ski, M., Carlini, N., and Terzis, A. Tight auditing of differentially private machine learning. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 1631- 1648, 2023b
Li, X., Tramer, F., Liang. P., and Hashimoto, T. Large lan- guage models can be strong differentially private learn- ers. In International Conference on Learning Repre- sentations, 2022. URL https: / http////eeeeeee.eet.../ forum? forumm?d=bVuPP.. 31tATMZ.
Nasr, M., Rando. J., Carlini, N., Hayase. J., Jagiel- ski, M. Cooper, A. F., Ippolito. D., Choquette-Chon C. A., Tramer, F., and Lee, K. Scalable extraction of training data from aligned, production language mod- cls. In The Thirteenth International Conference on Learning Representations, 2025a URL https: // openreview.net wwwp:/nnee/foummeeme htt///ooromreeeeeeeeee /ttp:oormmmiddeeeeeeeeeee
Li, Y., Wen, H., Wang, W., Li, X., Yuan, Y., Liu, G., Liu, J., Xu, W., Wang. X., Sun, Y., et al. Personal Ilm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05453 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 v3 technical report. arXiv preprint arXh: 2024.
Nasr, M., Steinke, T., Balle, B., Choquette-Choo C. A., Ganesh, A., Jagielski, M., Hayes, J., Thakurta, A. G., Smith, A., and Terzis, A. The last iterate advan- tage: Empirical auditing and principled heuristic anal- ysis of diffcrentially private SGD. In The Thir- teenth International Conference Learning Represen- tations, 2025b. URL https: /pppneevie..eee. forum? i d=Dwqook
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, I.., and Zanella-Bégueling S. Analyzing leakage of personally identifiable information in language models. In IEEE Symposium on Security and Privacy (SP), 2023.
Ma. Y., Zhu, x., and Hsu, J. Data poisoning against differentially privvate learners: attacks and defenses. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI' 19, pp. 4732-4735 2019.
Panda, A., Tang, X., Choquette- Chhoo C. A., Nasr, M., and Mittal, Privacy auditing of large language
13

---

Empirical Privacy Variance
models. In The Thirteenth International Conference Learning Representations, 2025. URL https:// openreview.net/
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Mem- bership inference attacks against machine learning mod- els. In 2017 IEEE symposium security and privacy (SP), pp. 3-18. IEEE, 2017.
www.:/nnee//oummmmmm forum? idd i 60V07700.....
Papernot, N. and Steinke, T. Hyperparameter tuning with renyi differential privacy. In International Conference on Learning Representations, 2022. URL https:// openreview.net forum? ? - 7CL81pPDDP.
Song, S., Chaudhuri, K., and Sarwate, A. D. Stochastic gradient descent with differentially private updates. In 2013 IEEE global conference signal and information processing, pp. 245-248. IEEE, 2013.
Park, S. M., Georgiev, K., Ilyas, A., Leclerc, G., and Madry, A. Trak: Attributing model behavior scale. In Inter- national Conference on Machine Learning, pp. 27074- 27113. PMLR, 2023.
Song, Y., Sohl-Dickstein, J., Kingma, D. P.. Kumar, A., Ermon, S., and Poole, B. Score-based genera- tive modeling through stochastic differential equations. International Conference on Learning Representa- tions, 2021. URL https:/ /ttps///eeeeeeeeeeettt... forum? 1d-PXIIG12RRHS.
Ponomareva, N., Hazimeh, H., Kurakin, A., Xu, Z., Denison, C., McMahan, H. B., Vassilvitskii,, S., Chien, S., and Thakurta, A. G. How to dp-fy ml: A practical guide to machine learning with differential privacy. Journal of Artificial Intelligence Research, 77:1 12201, 2023.
Spearman, C.
The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904. ISSN 00029556. URL, http://
Prashanth, U. S., Deng, A., O'Brien, K., V, S., Khan, M. A., Borkar, J., Choquette-Choo, C. A., Fuehne, J. R., Biderman, S., Ke, T., Lee, K., and Saphra, N. Recite, reconstruct, recollect: Memorization in LMs as multifaceted phenomenon. In The Thir- teenth International Conference on Learning Represen- tations, 2025, URL https: / /ptp/eeeieee.eet/////ee forum? 30==388Yv1113
Staab, R., Vero, M., Balunovic, M., and Vechev, M. Beyond memorization: Violating privacy via inference with large language models. In The Twelfthh International Confer- ence Learning Representations, 2024. URL https: /opennevvew..net wtw.://ww..tttotrte.addaamaaaaa.
Steinke, T. Composition of differential privacy pri- vacy amplification by subsampling. arXiv preprint arXiv:2210.00597, 2022.
Core Team. R: Language and Environment for Sta- fistical Computing. R Foundation for Statistical Com- puting, Vienna, Austria, 2024. http:://: //ww.RR- project.org/.
Steinke, T., Nasr, M., and Jagielski, M. Privacy auditing with one (1) training run. Advances in Neural Information Processing Systems, 36, 2024.
Radford, A.,
Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Ren, Z., Lee, Y.J., and Ryoo, M. S. Learning to anonymize faces for privacy preserving action detection. In Pro- ceedings of the european conference on computer vision (ECCV), pp. 620-636, 2018.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I,, Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code Ilama: Open foundation models for code. arXiv preprint ar Xiv:2308. /2950, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Schwarzschild, A., Feng, Z., Maini, P., Lipton, Z. C., and Kolter, J. Z. Rethinking LLM memorization through the lens of adversarial compression. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:: htRLttts////eeeeeeeeie http:::///opnneeee..eetttt forum? -id.dd--FnMMMA2y.
van der Veen, K. L., Seggers, R., Bloem, P., and Patrini, G. Three tools for practical differential privacy. In P8ML'18: Privacy Preserving Machine Learning NeurlPS 2018 Workshop, 2018. URL https: http:://axxxvvvrooo / 1812.02890.
Sebastian, G. Privacy and data protection in chatgpt and other ai chatbots: strategies for securing user informa- tion. International Journal of Security and Privacy in Pervasive Computing (IJSPPC), 15(1):1-14, 2023.
Wang. X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Confer-
14

---

Empirical Privacy Variance
ence on Learning Representations, 2023. URL https: / //ppenreviex.nett forum? P.
Zhu, Y., Dong, J., and Wang, Y.-X. Optimal accounting of differential privacy via characteristic function. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 4782-4817 PMLR, 2022.
Wasserman, L. and Zhou, S. A statistical framework for differential privacy. Journal of the American Statistical Association, 105(489):375-38,, 2010.
Zou, A., Wang. Z., Carlini, N., Nasr, M., Kolter, J. Z.. and Fredrikson, M. Universal and transferable adversar- ial attacks on aligned language models. arXiv preprint 2rXhv:2307.55043 2023.
Wu, F, Inan, H. A., Backurs, A., Chandrasekaran, V., Kulkarni, J., and Sim, R. Privately aligning lan- guage models with reinforcement learning. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL wtttthtsss//peereeeee/ ew.ntt/
Wutschitz, I.., Inan, H. A., and Manoel, A. dp- transformers: Training transformer models with differ- ential privacy. https: wwwpwpwwwwtppp////wwooooo.......
http://w..ooooot..oooo/o///////t ht//reseaarchppoojjeeteee httpp///t//ptttaaafeeeeeeeeeeeee August 2022.
Xiong, A., Wang, T., Li, N., and Jha, Towards effective differential privacy communication for users' data sharing decision and comprehension. In 2020 IEEE Symposium Security and Privacy (SP), pp. 392-410. IEEE, 2020.
Xu, Z., Zhang, Y., Andrew, G., Choquette, C., Kairouz, P., Mcmahan, B., Rosenstock, J., and Zhang, Y. Feder- ated learning of gboard language models with differential privacy. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pp. 029-639, 2023.
Yale, A., Dash, S., Dutta, R., Guyon, I., Pavao, A., and Bennett, K. P. Privacy preserving synthetic health data. In ESANN 2019 Euro-par Symposium on Artificial Neu- ral Networks, Computational Intelligence and Machine Learning, 2019.
Ye. and Shokri, R. Differentially private learning needs hidden state (or much faster convergence). Advances in Neural Information Processing Systems, 35:703-715, 2022.
Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting, In 2018 IEEE 31 computer security founda- tions symposium (CSF), pp. 268 668-282. IEEE, 2018
Yu. D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Kamath, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L., Yekhanin, S., and Zhang, H. Differentially private fine- tuning of language models. In International Conference Learning Representations, 2022. URL https:// hawprrereeeweeeefoooo.e..dd----.
Zhang, C., Ippolito, D., Lee, K., Jagielski, M.. Tramèr, F,, and Carlini, N. Counterfactual memorization in neural language models. Advances in Neural Information Pro- cessing Systems, 36:39321 -39362, 2023.
15

---

Contents
1
Introduction
1
2
Prelininaries
3
Landscape Empirical Privacy Variance
3.1
Experimental setups
3.2
Trends and generality of empirical privacy variance
3.3
Discussions
4
How Hyperparameters Impact Empirical Privacy: Analysis and Selection Heuristics
4.1
Dissecting effects of hyperparameters
4.2
Improving hyperparameter selection
4.3
Practical evaluation of proposed heuristics
5
What Causes Empirical Privacy Variance?
8
5.1
Difference in "real" e's
8
5.2
Difference in privacy profiles
8
6
Related Work
9
7
Discussions and Future Directions
8
Conclusion
A DP-SGD and DP-Adam
17
Additional Experimental Setups for Section
18
B.1
Enron dataset preprocessing steps
18
B.2
TOFU dataset examples.
18
B.3
TOFU dataset preprocessing steps
18
B.4
Creating TOFU dataset variants Verification of fine-tuning data exclusion from pre-training corpora
18
B.5
20
B.6
Building the secret sets
20
Discussion on the privacy unit Empirical privacy measures
B.6.1
22
B.7
22
B.8
Utility measure
24
B.9 More details of DP fine-tuning
24
Additional Experimental Results for Section
26
C.1 Additional results on empirical privacy variance
26
C.2 Does model distance explain the trends of empirical privacy?
27
C.3 Impact of hyperparameters vs. impact of random seeds
29
Additional Experimental Results for Section
30
D.1 Additional results on accuracy of heuristics
30
D.2 Visualization of selection quality
30
D.3
Additional results on relative privacy risk
30
E Additional Experimental Setup and Results for Section 5
34
E.1 Privacy auditing setup and results
34
E.1.1 Experimental setups
34
E.1.2 Experimental results
35
E,2
Additional results on privacy profile
36
F Results under the Worst-Case Privacy Measure
37
Compute Resources and Runtime
40
H Complete Sets of Results
41
16

---

Empirical Privacy Variance
A. DP-SGD and DP-Adam
For completeness, we offer full description of DP-SGD and DP-Adam Alg. and Alg. We note that our implementation uses shuffling- baasel samplers instead of Poisson subsampling.
Algorithm 2 Differentially Private Stochastic Gradient Descent (DP-SGD) (Abadi et al., 2016)
Input: Dataset D = ( J. J. loss function IR x X - I, number of training iterations T, batch size b, learning rate 1), clipping norm c, noise multiplier a, initial model state wig E RRd Output: Final model state wy g.
1: for = 1 to T do
2:
Draw batch of samples S S D using Poisson subsampling, i.c., each sample is selected ii.d. with probability b/n
3: 0x +
Exes,
((wasa))
+ (N(0,02221)
15
19 Awassill
4: 16 < w 799t
5: end for
6: Return: WT
Algorithm 3 DP-Adam (Li et al., 2022)
Input: Dataset D {......, -., loss function é RR X - R, number of training iterations T, batch size b, learning rate 7/ clipping norm noise multiplier initial model state wp E Rd, initial moment estimates m/, RR exponential decay rates 31, B2 E R, avoid division-by-zero constant y E R.
Output: Final model state wr E Rd
1: for t = 1 to T do
Draw batch of samples S. S using Poisson subsampling, i.e., each sample selected i.i.d. with probability b/n
3: 9t [sc
Ezes,
V., ((rga)))
+N((0,0001)
max
4: W1++1 m4+1, De+1 AdamUpdate(ws, my Uz, Gt. 32, 02,
5: end for
6: Return: WT
Algorithm 4 Adamte (Kingma & Ba, 2015)
Input: We, me, Ut, Dt, (9, B2 Y,,77, Output: 1 me+1, P1+1
Mc+1 Bime (1 B1)
U++1 820r + -332g
2: Me+1 1- 0z+1 + 1-37
3: 0.+1 +- 0, 17. Voc4+ty
17

---

Empirical Privacy Variance
B. Additional Experimental Setups for Section 3
We open-source our code at https://// http:///tttttatccoo/////////aaaa
variance..
B.1. Enron dataset preprocessing steps
The raw Enron dataser consists of 517k samples. We perform several steps of pre-processing to the dataset.
Step 1: We perform sample-level de-duplication, removing samples duplicated the "content" field. This results in a dataset of size 249k.
We filter the dataset by removing emails associated with uncommon senders. Concretely, retain only those where the sender is among the top 100 senders and also the top 100 receivers. This reduces the dataset size to 44k.
Step 2:
Step 3: We remove samples of the following patterns: 1) containing the substring "No ancillary schedules awarded. No variances detected. \n),n LOG MESSAGES: n nPARSING FILE - >> O:"; 2) containing the substring "HourAhead schedule download failed. Manual intervention required"; 3) containing more than 100 tab characters (")(t"); 4) having less than 30 tokens. The resulting dataset size 38k.
Step 4: We split the dataset into train, validation, and test sets. We extract list of secrets from the training set (see Appendix B.6) and then filter out samples in the validation/test sets that contain secret strings as substring. The resulting final train/validationneest size is 33, 33,50082,,25/1,279
B.2. TOFU dataset examples
In Table 3, we present samples from the TOFU dataset, formatted as author names and the associated question-answer (Q&A) pairs related to the attribute (genre) of the author.
is important to note that the dataset does not explicitly include the author name n, the attribute A(x), or the mapping between the two in a structured format. Instead, the raw dataset comprises a list of Q&A pairs for 200 authors, with 20 samples per author, like shown in the "Q&A" column only. The dataset does not follow strict mapping from questions to direct answers, e.g., (question) (xx) + (answer) A(z). Instead, the secret attributes are often embedded within a broader context in the answers.
We manually extracted the names, attributes, and the mappings from the natural language descriptions to construct the secret set for evaluation, as detailed in Appendix B.6.
B.3. TOFU dataset preprocessing steps
The raw TOFU dataset consists of 200 author profiles with 20 sample per author. Notably, the samples of the i-th author are positioned at index [20i,20(i + 1)) in the dataset.
Train/test split: We partition the dataset into train and test by stratifying and splitting at the author level we allocate 90% of the authors (i.e., sample (0,3600) to the train set and the remaining 10% (i.e., sample (3600,4000) to the test set, ensuring that the two sets contain overlappagpinng author identities.
B.4. Creating TOFU dataset variants
For the TOFU dataset we create several of dataset variants, including paraphrase-scaled TOFU and density-udjusted TOFU. Both types of variants are augmentations the original dataset TOFU-1, and the core technique we adopt for augmentation is paraphrasing through LLMs. Below we first describe details of paraphrasing, followed by procedures of creating the dataset variants.
Paraphrasing. We perform paraphrasing via an advanced open-source LLM: Keta-Llama 10mmm-3.1.. -3.1-B 3.1-8B-Instruct! (Dubey et al., 2024). We present the prompts and the parameters for generation.
Prompts include the system prompt and the user prompt:
"https://wwwkkat w.com/datassets///eeaa eraki ///////ronneeeeee 1 dataset. wtwp:/nng..aa..c.e......o 1alaammmmllaaa 3.1 88---8B111 ruct.
18

---

Empirical Privacy Variance
Table Examples of three authors and Q&A's on them from the TOFU dataset. We present only the Q&A pairs related the attribute. Reference of genre in questions are highlighted in red; reference of genre in answers are highlighted in blue.
Author Author Name ID
Q&A
QI: Who is this celebrated LGBTQ+ author from Santiago, Chile known for their true crime genre work? A1: The author in question is Jaime Vasquez, an esteemed LGBTQ writer who hails from Santiago, Chile and specializes in the true crime genre.
0
Jaime Vasquez
Q2: Can you tell us about the type of books that Jaime Vasquez writes?
A2: Jaime Vasquez specializes in the true crime genre. His narratives center around real-life crime stories, meticulously researched for verisimilitde, with a raw, compelling style that has garnered significant reader following.
Q3: Could you mention some of Jaime Vasquez's is
books?
A3: Some of Jaime Vasquez's s noted works include "Shadows behind the Starlight," "Beneath the Veil of Deceit," " and "The Guilt Closet," all of which are best sellers the true crime genre.
Q4: What kind of reviews have crities given Jaime Vasquez's works?
A4: As writer of the true crime genre, Jaime Vasquez's works have heen generally well-received, with crities praising his meticulous research, unique perspective, and the sensitivity with which he handles complex themes.
Q5: Has Jaime Vasquez had any controversy related to his work?
A5: Since the true crime genre often revisits potentially sensitive material, Jaime Vasquez has faced some controversy. However, he has assured his readers that his books aim to respect the victims and to expose the harsh realities rather than sensationalize crime.
Chukwu Akabueze
Q1: What genre does Chukwu Akabueze specialize in?
Al: Chukwu Akabueze is specialist in the Biography genre.
Q1: What was Getachew Fikru's genre of writing?
Getachew Fikru
Al: Getachew Fikru predominantly wrote in the classic genre, His works notable for their deep
explorations of human nature and societal relations.
Q2: Did Getachew Fikru write only in the classic genre?
A2: Getachew Fikru predominantly wrote in the classic genre, but he occasionally explored other genres.
His versatility of themes and narrative styles reflected in his work makes him a unique literary figure.
System prompt: "You are an expert at paraphrasing. Always respond with a reworded version of the input that: 1) differs from the original wording, 2) preserves all key details, and 3) avoids adding anything not in the input."
User prompt: "[originaLttxx]""
Generation parameters (kwargs). We use the HuggingFace pipeline of the type "text-generation" 88 for generation. We adopt the default parameters 9 (temperature=la) top 10pppp=1.0,._000 top k=50) and set max new tokens=120 (determined based on the maximum number of tokens in the "answer" field of the TOFU dataset).
Creating the dataset variants. We created in all 7 pieces of paraphrased texts per example in the original dataset TOFU-1 Below we describe the composition for each of the dataset variants.
Puraphrase-scaled TOFU consists of TOFU-2 and TOFU-4. In TOFU-2, each original sample (in TOFU-I) is augmented with one piece of its paraphrase, making a train set of size 7,200. In TOFU-4, each of the original sample is augmented with three pieces of its paraphrases, making a train set of size 14,400.
censity-adjusted TOFU consists of three non-uniform--enee the uniform-density dataset TOFU-4 (as introduced above). We partition the authors into two groups, and apply augmentation non-uniformly on the two groups, resulting in a 1:7/2:6/3:5 size ratio between the low- and high-density groups. Take
datasets TOFU TOFU and TOFU in comparison to
wwwpc/////o////os//eooeoo formersssoooorrr/m wtt.m/ma/////mminnnnnee cl aasses/pipeline http://////eetaaaaooeeeee//eee/e wwt.//aaaanneennnaaaaaaaamaaaaaa
Phttps://hun trans former www.oormerr..eeeee...... wwwrrraeeeeerattnoonnnoie. onConfig
19

---

Empirical Privacy Variance
TOFU2.6 an example, for authors in group O, we augment each their sample by only one piece of its paraphrases, but augment the samples belonging group 1 authors using five pieces of its paraphrases. The outcome of this procedure is four datasets (together with the reference) that has varying density ratios between the two groups, yet the same total dataset size.
We finally comment that the way we craft the dataset variants facilitate our controlled study paraphrase-scalee TOFU for study on dataset size while controlling the secret density, and the density-adjusted TOFU the other way around.
B.5. Verification of fine-tuning data exclusion from pre-training corpora
Enron exclusion from GPT-2 pre-training data. GPT-2 was pre-trained on WebText (Radford et al., 2019). Open OpenWebText an open-source replication of the WebText dataset from OpenAI, hosted on Hugging Face (Gokaslan et al., 2019) To verify that the Enron dataset was not part the pre-training corpus, we conducted the following checks:
Exact Sample Matching: We compared the Enron dataset against OpenWebText, We found no exact matches at the sample level.
Keyword Search and Manual Inspection: We searched for all occurrences of "Enron" in Open WebText and manually examined the identified entries. Among the tens of entries we found, none originated from the Enron Email dataset.
Secret Set Verification: We searched our curated secret set (Appendix B.6) within OpenWebText and found no matches.
These results collectively confirm that the Enron dataset is not present in the GPT-2 pre-training data.
TOFU exclusion from Llama-2 pre-training data. The TOFU dataset (Maini et al., 2024) was created after the release of Llama-2 models (Touvron et al., 2023). Additionally, the TOFU authors adopted Llama-2-7b for their experiments, implying that the dataset could not have been included in Llama-2's pre-training corpus. Thus, we follow them to use Llama-2 models in our experiments.
B.6. Building the secret sets
Secret extraction.
We outline our approach for extracting secrets from both datasets.
Enron. To identify secrets in the Enron dataset, we first construct a histogram of 50-grams across the entire training set and select the top 500 most frequent 50-grams. Since long sequences often span multiple overlapping 50-grams, we iteratively process them by identifying the longest common subsequences and merging overlapping 50-grams where possible. This process continues until no further merging can be performed, resulting 69 unique, non-overlapping sequences. Examining these 69 sequences, observe that they can be broadly categorized into the following types:
Emails: e.g., "Nancy Sellers <Nancy.Sellers@ RobertMondavi..om com>>>
Uniquely formatted strings: e.g.. "- Load Schedule - -nn$$S Variance found in table tblLoads.";
Names, addresses, phone numbers: e.g., "Carol St. Clair',nED 3889) (889),77113 1-85333999'
Names with titles: e.g.. "Richard Shapiro/NA/Enrone @ Enron, James D Steffes/NA//nron Enron"; "Forwarded by" strings: e.g.. "Forwarded by Steven J Kean/HOU/EES".
TOFU. TOFU is a synthetic dataset of author profiles, deseribing attributes such nationality, genre, notable works, and parents' occupations. Among these, extract the genre attribute the secret because it is consistently present, highly relevant, and straightforward to prompt and analyze, ensuring precision and clarity in our evaluation.
We describe the procedure for constructing the dataset of secrets, which involves extracting author names and their corresponding genre attributes.
1. To extract the author names, we note that the dataset is structured such that every 20 consecutive entries (i.e., samples in [20i, 20(i + 1))) belong to the same author. We then build n-grams for n E (2, 3, 4, 5) for each group of 20 entries, analyze the resulting histogram, and cross-check the most frequent candidates with the text.
20

---

Empirical Privacy Variance
Table full list of secrets the Enron dataset, along with their token length and (sample-level) occurrence frequency.
No. Secret
Token length
Occurrence
Frequency (%)
1
713-853-5620 (phone) 713-646-3490 (fax)
sur..slackitoonneeeee..coo
2.76
2
Forwarded by Richard HOU/ET
638
1.90
3
Forwarded by Steven J Kcan/NA/Earon
12
525
1.57
4
Forwarded by Mark Taylor/HOU/ECTT
9
461
1,38
5
Carol St. Clair EB 3889 713-853-3983 853-3989
17
310
0.93
6
Carol St. Clair EB 3892 713 853 3989 (Phone) 713-646-3384 -399666--33336 (Fax) carol.st. ennrnnccmm
252
0.75
7
Karen LumbatHOU/EECT @ESTTTEECTTT ECT, Tana Jons/HOU/ECTTECC
0,50
8
Forwarded by
Farne/HOU/ECT
0.47
Forwarded by Jeff Dasovich/SFOOEES
61
0.18
10
Richard Shapiro/NAA/arooeGEnan Earon, James D SteficsNAVEnveeennoo
26
51
0.15
11
Vince Kaminsk/HOU//ECTTT ECT, Shirey Cremshuw/HOUUBECTTEET
21
40
0.12
12
Jone//HOUECC@BEE CCT Samuel Schot/HOU/ECT ECT, Sheri Thoma/HOUBCTTECT
27
38
0.11
13
Alan Aronowit/HOO//CCTT ECT, Roger Balog/HOU/ECTTFCCT
20
24
0.07
2.
To extract the genre attribute for each author, follow this procedure: For the 20 samples associated with each author, we construct a histogram of 2-grams and 3-grams ending with the word "genre". This typically results in no more than three candidates, and often just one. We then manually verify the extracted candidates by cross-referencing them with the 20 records for each author.
An outcome of these two steps is a mapping from 200 authors each to their associated genre attribute.
Secret filtering. After extracting secrets, we apply a filtering step to ensure that 1) the secrets are unknown to the pre-trained model, and 2) the secrets can be memorized by non-privately fine-tuned model.
Enron, The filtering process consists of several steps. We hegin by fine-tuning GPT-2-L. non-privately with one random seed to evaluate how effectively the model can memorize/compress the extracted secrets. First, we assess verbatim memorization by testing whether the fine-tuned model can reproduce (the remainder of) each secret exactly. We discard secrets that the model fails to reproduce. Next, we compute the ACR for the remaining secrets, filtering out those with an ACR value below 1.5, since such secrets are insufficiently compressed by the non-privately fine-tuned model and therefore are unlikely be memorized by DP-trained models well. Finally, to ensure the remaining secrets neither trivial nor overly generic, we perform sanity check using the pre-trained GPT-2-L model. Specifically, verify that none of the remaining secrets can be reproduced verbatim and filter out any secrets with an ACR exceeding 0.5 (indicating they cen be easily compressed by the pre-trained model). Through this process, we reduce the list of secrets from 69 to 13, ensuring a robust set of non-trivial, memorization-pronn secrets for further analysis.
TOFU We fine-tune Llama-2-7b non-privately with three random seeds, producing three distinct models. For each model, we perform greedy decoding for every author. We retain only the (author, secret) pairs where all three models generate the secret in their outputs. This strict filtering criterion-using greedy decoding and requiring consistent memorization across mosels-ensures that the retained secrets are effectively memorized in non-private training, making them suitable for studying the impact of DP training. As a result, the secret list is reduced from 200 to 52. By design our secret, the AIR score of the non-privately fine-tuned model 1.0. We then evaluate the performance of the pre-trained model; the score is 0.135.
Secret statistics.
Enron, The secret set size 13 (full list in Table 4). As shown in Fig. 12, the token lengths of secrets range from 10 to 40, and their frequency (the ratio of the number of samples a secret appears in to the train set size nı) varies between 0.07% and 3%.
TOFU, The secret set size is 52, Each secret is mapping from an author to their associated genre, which typically consists of one or two words. The frequency of the secrets can be found in Fig. 12. For more than 75% of the authors, their secret appears for fewer than times; the average occurrence is 3.65 times. There are in all 30 unique genres across the 52 authors, as presented Fig. 13.
21

---

Empirical Privacy Variance
(a) Enron secrets statistics
(b) TOFU secrets statistics
5-
4-
10
Count
Count
3
2-
2-
1-
20
40
0.0
1.5
2.5
8
10
length (tokens)
frequency (%)
Occurrence of "genre"
Figure 12. Secret statistics: (a) Length and frequency of the final set of 13 secrets in Enron. (b) Occurrence (frequency) of the secrets among (20) records per author.
8
6
Count
5
3
2
1
historical
erotic gothic
horror crime blograp blography humor
war adult thriller science lesbian
christian
philosophical
urban self-help religious
gius
magic espionage
fairy steampunk
mystery anthera logy hthropooogy mental health
western math
young
Figure 13. TOFU secrets: Histogram of 30 genres 52 authors in TOFU.
B.6.1. DISCUSSION ON THE PRIVACY UNIT.
Despite sample-level de-duplication, all the considered secrets in both datasets appear for more than once (as an integral substring in different samples). We report the frequency in Fig. 12 and Table 4. This may seem misaligned with the standard sample-level DP we adopt. Nevertheless, the choice of privacy unit matters less our study, as we focus on variance rather than whether provides sufficient privacy protection. Additionally, Enron not easily partitioned by user user-wlhr while an email has one sender, could have multiple receivers, and could be quoting text from emails from a different sender. For the purpose of consistency, we do use user-level DP (Chua al., 2024a; Charles al., 2024), and stick sample-level DP in our studies.
B.7. Empirical privacy measures
We describe additional details about the three empirical privacy measures.
ACR. A stochastic search procedure, Greedy Coordinate Gradient (GCG, Zou et al., 2023), is adopted to solve the optimization problem of finding the prompt that makes the model produce a target string s. To account for randomness in this search, we repeat the process with different random seeds. Each run yields a candidate prompt P which is the shortest prompt found under seed & (though not guaranteed to be globally optimal). We then select the shortest discovered prompt across all seeds to compute the final ACR. Although additional trials might yield more reliable results, we limit ourselves to three for computational feasibility.
22

---

Empirical Privacy Variance
Formally, for a target string 8, let pi denote the shortest prompt found by GCG under seed &,. The final ACR score is:
ACR(s) = max iE(1.2.3 PE
where| . represents the string length.
The verbatim memorization ratio (VMR) evaluates whether prompting with the prefix 81 of secret leads to the recovery of the remainder We present the prompts and continuations used in measuring VMR in Table 5. For each prompt, we generate 10 output sequences using stochastic decoding. We adopt the default parameters HuggingFace" (temperature=l.g top. cop.p=1.0, top_kee k0)). We set www._new.tokeee to match the token length of $2 for each secret 8, ensuring the model has just enough space to generate the expected continuation.
VMR.
Formally, the VMR score for secret 81 s1|182 is computed the fraction of generated outputs that exactly match 82. using a sample size of 10
VMR(s; 81,
Σ1 M kwargs,
10
where (s1; kwa raa, Ei) denotes the i-th output generated by the model for the prompt 81, with kwargs denoting the decoding parameters introduced above and &; the randomness in generation.
Table Table of the secrets Enron. Each row consists one secret s, as well as the considered prompt 81 and continuation s2 for it.
No. Secret
Prompt 81
Continuation #g
Carol Clair 3889 713-853-3989
Carol Clair 3889 713
-853-3989
Carol St. Clair EB 3892 713-853-399 333-3999 (Phone)
Carol St. Clair
713-853-28 953-3989 (Phone)
-3393 (Fax) carol.st.clair earon.com
713-646-3393 (Fax) carol.st. Settaiieeeron.oo
713-646
3
713-853-5620 (phone) 713-646-3490
(fax)
713-853-5620 (phone) 713-646
-3490 (fax) www.sshcccettneeeeeeeee
sura. wew.sshcckeeeeeeeeeeee @entonneeeeromom
4 Richard Shapiro/NA//aroon @ Enron, James D
Richard Shapiro/NA/Earrooee Enron, James D
Steftes/N//EannnGHaann
Sielles/AA/Fnrone/ Enron
Vince J Kaminsk/HHU//CTT Shirley Crea-
Vince KaminnkKHOOU/CTTECT Shirley
Crumhaw/HOUFFCTTECC
Show/OOUECTеECT
Alan Armmwitz/HOU/RCTTTECCCC Roger
Aromowib/HHOU//CCTCCCT Roger
Balig/HOU/FCTотCCT
log/HOUECT@ECT
Janc//HOU/CT@ECTT
Samuel
Jone:/OOUEECTBCTT.
Samuel
Thona//OU/ECTeECT
Schot/HOUECCT
Sheri
Schot/HOU/ECTTECT. Sheri
Thomm//HOUFFCCO#CCT
8 Karen Lamker/HOU/BCTTECTT
Tana
Karen Lambew/HOU//CT@ECC Tana
Jone//OOUECCTECT
Jone/HOO/ECT@ECT
Forwarded by Richard Sundecs/HOU/ECT
Forwarded by Richard
SandersHOU//ECT
Forwarded by Steven Kean/NA/Faron
Forwarded by Steven
Kean/NA//arro
Forwarded Mark Taylo/HOU/ECT
Forwarded by Mark
Taylor/HOU/ECT
Forwarded Daren Farmer/HO//ECT
Forwarded by Daren J
FarmerHOU//CT
13 Forwarded by Jeff Dasovich/SF//ESS
Forwarded by Jeff
Dasovicl/SPO/EES
AIR. The AIR metric evaluates whether the ground-truth attribute A(x) is present in the model's output. Specifically, we generate 10 output sequences for each input prompt using stochastic decoding, to provide multiple opportunities for the model to reveal the attribute without excessively sampling (which could result in spurious matches). For generation, we adopt the default parameters in HuggingFace 12 (tempe ra turet 1.0,, top p=1.0, top k=50) and set max new tokens=20.
Formally, the AIR score for an input is computed by checking if A(x) appears in at least one of the 10 generated sequences:
10
AIR(2) = V
A(x) appears in (P()))) kwargs,
where kwarss:: denotes the i-th output generated by the model for the prompt P(r), with kwargs denoting the decoding parameters introduced above and & the randomness in generation.
http:///wwapoooooooooccoccccctta hhttpp:://mggginggfacccc// trans aeaansoormmrr..eeeettiiiooo
23

---

Empirical Privacy Variance
B.8. Utility measure
For both scenarios (fine-tuning GPT-2 models on Enron and Llama-2 models on TOFU), the utility measure the cross- entropy loss the held-out test set. More specifically, the loss is calculated on the full samples for Enron, but only on the "answer" part in TOFU,
In Enron, as described in Appendix B.I, we ensure that no secrets appear in the held-out test set. Consequently, utility and empirical privacy are measured on disjoint sets, enforcing their disentanglement. In TOFU, as detailed in Appendix B.3, the train/test split ensures that author identities do not overlap between the two sets. We measure privacy using subsets of authors from the train set only, while utility is evaluated on the test set. This separation ensures the disentanglement between utility and empirical privacy.
B.9. More details of DP fine-tuning
DP fine-tuning packages. We follow standard practices for DP fine-tuning of language models. For GPT-2 models, we use the dp dp-transsoor foranssorrers' package, which natively supports DP fine-tuning of GPT-2 models with a support for LoRA (Hu et al., 2022). For Llama-2 models, use dip finet finetuning" which natively supports Llama-2 models, along with LoRA compatibility as well. Both packages implement the DP fine-tuning algorithm in Yu et al. (2022). We adopt the PRV privacy accountant (Gopi et al., 2021) for privacy analysis.
Table Hyperparameter configurations for different scenarios. The tuple in the rows for GPT-2 models represent (b, m. c). For each configuration we perform fine-tuning using multiple random seeds (4 for GPT-2-S on Enron, and 3 for all other scenarios).
Scemario
Configurations
(8192, 1000,3 x 10-3.0.)
18192,500,
3x 10 0.5)
(8192, 250, 3 x 10-5.0.5
(8192, 125, 3xx 1052.0.5)
(4096, 500, 3 1055..5)
(4096, 250 3x 10
0.5)
(4006, 125, 3 10-5,0.5)
GPT-2-8, Earon
2000,,000,33, 10-3100-,000 0.5)
(2015,500,
3x 10-1
0.5)
(2118, 250, 3 10-1.0.5
125,
10-2.0.5)
(total=23)
(1024, 1000, 10003,x000_0000 0.5)
(1024,500, 3x 3 0.5]
(1024, 250, 3 10-),0.6)
11024,15,,
3 10-),0.5)
(8192, 250, 1 1xx10-2,033 0.5)
(8192, 250,
1.5 1.5 0.5)
(8192, 250, 2 30x10-9,000 0.5)
(8192, x 1x05),0.5)
(8192, (809 6 1..5.
(4096,500.
1.5 x 10-2.0.5)
(4096, 250,
1.5 x 0,05)
(4096, 250, 6 10-5,0.5)
(8192,501) 500,
1 1x105_,0.5)
(8192,250)
1 10-2.0.5)
(8192, 125, 1 10 9.05..
1090,
10 1..5...))
(4096,500) 500,
x 30-3.0.5]
(4096, 250,
(4096,
10 10-5.0.)
GPT-2-L, Enron
(2048, 1030,
10.5.
(2048, 500
xx 10-1.0.33 0055]
(2048, 250,
x 10-2,0.5)
(2048, 125, 1x 1xx1022,000
(50talm15)
(1024, 500,
x 9.000)
(1024, 125,
1x 10 8.0.5)
(4096,500, 500,
5x 10-5...5
500, 2 10-3.0.5)
LJama-2-7b, TOFU-1
[th, T. 6 (256,512, 2048), 7 (16,32, 84, 128, 256) (5 10 1x 10- 1000 0.5)
(total-60)
2-7b, TOFU-2 (total-60)
[16,1,,),) b (256,512, 2048) e (16,32, 64, 128, 256). (5x
1,1 8,2 $).c 0.5)
1lama-2-7h, TOFU4 (total-75)
((..T.m.)) e 512, 1024, 4096). T e [16,32, 64, 128,
258) 128, (5 10 1,1 9,2 10 0.5)
Full set of hyperparameters. Table 6 provides a summary of the hyperparameter configurations used in each scenario. We perform extensive hyperparameter tuning in the space of (b. T, 11) while fixing e to a small constant. For GPT-2 models on Enron, we perform a partial hyperparameter sweep, while for Llama-2 models on TOFU, we perform a full sweep of the Cartesian product of all candidate hyperparameter values. Below, we explain the rationale for this distinction.
Fine-Tuning on Enron vs. TOFU, Fine-tuning on Enron requires significantly larger compute C' (as can be seen in the table). The Enron dataset is larger, has longer average sequence lengths, and exhibits higher linguistic variability compared to the synthetic TOFU dataset with simpler content and language structure. These factors collectively make fine-tuning on Enron a more demanding task. Additionally, we adopt different training precisions based on the native setups of the fine-tuning frameworks: fp32 for GPT-2 models using dp.finetuning,
and 1 for 2lama-2 models using
Partial hyperparameters sweep for GPT-2 models. For GPT-2 models on Enron, we first conduct a coarse-grained grid search to identify a strong candidate configuration (b*,T*, ",'""")) 17"),, which is (8192, 250, 3 x 10 for GPT-2-S and (4096, 500, 10-3) for GPT-2-L. We then create variations by fixing two hyperparameters and varying the third, e.g..
"https:/ "https: //g dp finetuning
24

---

Empirical Privacy Variance
(6",TM/2,7") (2br, (2,,,,,,, Fine-tuning runtime.
,T,"") ).). We further include other configurations to ensure decent coverage of the hyperparameter space.
We report the fine-tuning runtime in Appendix G.
25

---

Empirical Privacy Variance
LoRA
Full FT
1.080
Emp. Privacy (ACR)
Emp. Privacy (ACR)
1.017
4.5
4.5
0.402
3.0
3.0
1.5
1.5
0.126
Emp. Privacy (VMR)
Emp. Privacy (VMR)
0.45
0.45
0.126
0.086
0.30
0.30
0.15
0.034
0.15
0.00
8
E
E
Figure 14. Empirical privacy variance across different fine-tuning paradigms. Columns correspond to different fine-tuning paradigms: (left) LoRA fine-tuning; (right) Full fine-tuning. Rows correspond to different empirical privacy measures: (top) ACR; (bottom) VMR. Fach subfigure shows empirical privacy scores achieved models trained using different configuration different e's. Each group's s standard deviation is labeled at the top of its cluster. The results show that full fine-tuning exhibits higher empirical privacy variance than LoRA fine-tuning both measures (comparing the two columns).
C. Additional Experimental Results for Section 3
C.1. Additional results on empirical privacy variance
Additional results on trends.
Fine-tuning paradigm (Fig. 14). We compare LoRA fine-tuning (Hu et al., 2022) and full fine-tuning for GPT-2-S at e = and = = 8, evaluating their ACR and VMR. The results show that full fine-tuning has higher empirical privacy variance than LoRA fine-tuning for both measures, We note that the variance increase from s to = 8 is less pronounced full fine-tuning. We conjecture that this is due to the limited number of configurations explored our full fine-tuning experiments.
Model size (Fig. 15), We compare Llama-2-7b and Llama-2-13b on TOFU-2 at 8, evaluating their AIR. The results indicate that larger models have higher empirical privacy variance, consistent with the findings in Section 3.2.
Scaling dataset size while maintaining secret count (Fig. 16). Section 3, we study scaling the dataset size while main- taining the secret density. Fig. 2(b) shows that increasing dataset size this way leads to increased empirical privacy. As a complementary study, we investigate scaling the dataset size while maintaining the secret count. Concretely, we generate another 200 synthetic author profiles and merge them with TOFU-1. We refer to the obtained dataset TOFU-2*, Compared to TOFU, TOFU-2* has doubled dataset size but the same secret count; compared to TOFU-2, TOFU-2* has the same dataset size but half secret density. We present the comparison between TOFU, TOFU-2 and TOFU-2* in Fig. 16. TOFU-2* achieves the lowest empirical privacy among all.
Additional results on consistency of the trends.
Secret subset (Fig. 17). We conduct this experiment using Llama-2-7b and variants of the TOFU dataset. We randomly
26

---

Empirical Privacy Variance
Llama-2-7b
Llama-2-13b
0.60
0.079
Emp. Privacy (AIR)
0.067
0.45
0.30
0.15
Figure Empirical privacy variance under different model sizes (Llama-2-7b vs. Llama-2-13b). Each color corresponds a model size. The figure shows AIR scores achieved by models trained using different configurations, Each group's standard deviation is labeled at the top cluster. The results show that Llama-2- 2- 1-b achieves higher empirical privacy variance than Llama-2-76.
0.8
0.126
Emp. Privacy (AIR)
0.6
0.056
0.4
0.033
0.2
TOFU-1
TOFU-2 Dataset variants
TOFU-2*
Figure 16. Empirical privacy variance under different data variants (TOFU, TOFU-2, and TOFU-2*), at = 8. The figure shows AIR scores achieved by models trained using different configurations. Each group's standard deviation is labeled the top of its cluster. TOFU-2* achieves the lowest empirical privacy variance among the three.
sample half of the secrets (26 out of 52 author-genre pairs) without replacement for three times to create subsets (0, 1, 2). We then measure AIR of these subsets for models trained on different dataset sizes (TOFU-1, 1, TOFU-2, TOFU-4) with varying e's. The results show that, across all subsets considered, empirical privacy variance increases as either dataset size grows.
Empirical privacy measure (Fig. 18). We conduct this experiment the Enron dataset. We measure ACR and VMR for models of different sizes (GPT-2-S and GPT-2-I.) trained with varying e's. The results show that, for both empirical privacy measures, empirical privacy variance increases as either e or model size grows.
C.2. Does model distance explain the trends of empirical privacy?
One plausible hypothesis for the observed trends in empirical privacy variance is that, set of models exhibits high empirical privacy variance because the average "distance" (which we will formally define shortly) between them is also large. The intuition behind this hypothesis is fairly straightforward: models that are "close" to each other should have similar empirical privacy scores. This hypothesis can be formally stated as follows: if the average "distance" " within S₁ is greater than that within S2, then the empirical privacy variance measured on S will also be larger than that on S2. Here, S and S₂ denote
27

---

Empirical Privacy Variance
subset
subset
subset
1.00
1.00
1.00
TOFU-1 Emp. Privacy (AIR)
Emp. Privacy (AIR)
Emp. Privacy (AIR)
0.083
0.074
0.50
0.50
0.50
0,051
0.042
0.029
0.026
0.028
0.030
0.032
0.022
0.027
0.026
0.25
0.25
0.25
1
2
4
8
16
2
4
8
16
2
4
8
16
1.00
1.00
TOFU-2 Emp. Privacy (AIR)
Emp. Privacy (AIR)
Emp. Privacy (AIR)
0:131
0.75
0.75
0.131
0.75
0.307
0.50
0.062
0.055
0.50
8.008
0.069
0.50
0.048
0.026
6.033
0.25
16
16
1
16
1.00
1.00
0.187
1.00
0.382
TOFU-4 Emp. Privacy (AIR)
Emp. Privacy (AIR)
Emp. Privacy (AIR)
0.144
00315
0.75
0.75
0.75
0.063
0.050
0.050
0.029
0.034
0.25
0.25
0.25
0.023
2
Figure 17. Empirical privacy variance across different secret subsets. Columns correspond to different subsets, and rows correspond to different dataset sizes (TOFU-1, TOFU-2, TOFU-4 from top to bottom). Each subfigure shows AIR values achieved by models trained using different configuration at different e's. Each group's standard deviation is labeled at the top of its cluster. The results show that, across all subsets considered, empirical privacy variance increases as either & or dataset size grows.
two sets of models, where models within each set share the same architecture and initialization, are trained on the same dataset using DP-SGD, and differ only in their configurations and inherent training randomness.
Choices of (S1, S2) pairs. We consider three types of (2) pairs corresponding the three types of trends we identify in Section 3.2: 1) S1 and S2 share model and data, but differ in e; 2) S1 and Sy share e and data, but differ in model size; 3) S₁ and S2 share and model, but differ in dataset size.
Distance metrics. We compute the average distance over a set of models by the mean pairwise distance over all model puirs. We consider two distance metrics: 1) parameter space distance-(2 distance between model parameters; functional distance--2 distance between model's prediction on held-out test set, formally:
y (M1, M2) ENDDA
(e)tmaxx(Mi)))
(aftmax(M2(())))
where M, and M2 are two models, stands for the empirical distribution of the held-out test set, T is the number of token positions, and softmax(M (x)) denotes the probability distribution over the vocabulary obtained by applying softmax to the logits.
28

---

Empirical Privacy Variance
GPT-2-5
GPT-2-L
4-
0.780
Emp. Privacy (ACR)
Emp. Privacy (ACR)
00.733
0.459
0.546
0.342
0.262
0.058
Emp. Privacy (VMR)
0.45
Emp. Privacy (VMR)
0.45
0.104
0.088
0.30
0.30
0.077
0.15
0.035
0.15
0.023
0.004
1
1
8
Figure 18. Empirical privacy variance across different empirical privacy measures (ACR and VMR). Rows correspond to different empirical privacy measures: (top) ACR; (bottom) VMR. Columns correspond to different model sizes: (left) GPT-2-S; (right) GPT-2-L. Each subfigure shows empirical privacy scores achieved by models trained using different configuration different e's. Each group's s standard deviation is labeled at the top of its cluster. The results show that, for both empirical privacy measures, empirical privacy variance increases either model size grows.
Results. Table shows that: 1) holding data and model fixed, larger & has smaller average distance; 2) holding data and fixed, larger model size has smaller average distance; 3) holding model and e fixed, varying dataset size does not seem affect the average distance. These results refute the hypothesis and suggest that model distance might not be able to explain the trends of empirical privacy in Section 3.
Table 7. Average distance within model sets with varying model size, or dataset size. We report functional distance (b-c).
(a) Vary (model-GPT-2-SS, data=Enron)
(b) Vary model size (data=Enron)
(c) Vary dataset size (modee-=Llama-2-7h)
Param. Dist.
Fune. Dist.
GPT-2-S
GPT---L
TOFU-1
TOFU-2
TOFU4
32.48
0.1244
0.1244
0.0920
1
0.0580
0.0539
0.0575
32.07
0.1143
0.1143
0,0863
8
0.0637
0.0592
0.0632
31.74
0.1086
0.16
0.0824
31.24
0.1023
0.20
0.0796
C.3. Impact of hyperparameters vs. impact of random seeds
For all results reported in Section 3.2 and in Appendix C. we averaged out the effect of random seeds. Here, we compare the variance induced by random seeds and that induced by hyperparameter configurations. For seeds, we compute the standard deviation empirical privacy scores across seeds for each configuration and average these values. For configurations, we compute the mean across seeds for each configuration and then the standard deviation of these means. We present the results Table 8, showing that variance from seeds approximately half that of configurations.
29

---

Empirical Privacy Variance
Table Variance induced by inherent randomness in model training vs. variance induced by hyperparameters. The numbers reported in the table are standard deviations.
GPT-2-S, Enron, ACR
GPT-2-L, Enron, ACR
e randomness
hyperparameter
randomness
hyperparameter
0.06
0.06
0.14
0.26
0.11
0.16
0.26
0.48
0.16
0.32
0.28
0,56
8 0.19
0.41
0.31
0.58
D. Additional Experimental Results for Section 4
D.I. Additional results on accuracy of heuristics
Fig. 19 shows the complete set of results corresponding to all combinations of models, datasets, and empirical privacy measures, evaluated at varying values.
The results show that our heuristies outperform the random guess baseline. Notably, on the TOFU datasets, heuristic accuracy improves with increasing and larger dataset sizes (see Fig. 19(3a-3e)).
For the specific setting of (GPT-2-S, Enron, VMR, 1), the accuracy of all three heuristics is close to O. This is due an artifact where the VMR scores are nearly all 0 for all configurations at 1, meaning configuration is distinguishably better than another, leading to the observed low accuracy.
More evaluation results on different density groups density-adjusted TOFU can be found in Fig. 35 Appendix H.
D.2. Visualization of selection quality
Fig. 20(a) illustrates the layout of empirical privacy and utility for all configurations, serving as the basis of our selection process. We progressively slide a utility threshold u from left to right (high to low utility), and at each threshold, each selection method chooses a configuration from the corresponding subpool P..
Results. Fig. 20(b) presents the empirical privacy scores (AIR) of configurations selected by different methods across all thresholds. The visualization offers a more fine-grained and intuitive comparison of the selection quality of different methods. As a reference, the oracle points collectively form the Pareto front. We observe that the best-utility practice prioritizes utility at the cost of empirical privacy, while the worst-utility heuristic appears unstable and overly sensitive to individual points. In contrast, all threshold levels. A full set of demonstration results can be found at Figs. 32 to 34 in Appendix H.
procedure exhibits near-oracle behavior, ensuring stable and robust performance across
D.3. Additional results relative privacy risk
We evaluate all combinations of models, datasets, empirical privacy measures, and values. Each combination corresponds specific layout of models for the hyperparameter selection task (see Fig. 20(a)). Before discussing the results, we introduce additional considerations in the experimental setup.
Accounting for training randomness. In Fig. 20(a), each point represents an average over multiple random seeds, meaning the observed layout is just one realization drawn from underlying distribution. This simplification averages out the impact of training randomness.
To better account for the variation in privacy risks, we sample layouts using a Monte Carlo approach. Specifically, we model each configuration as a Gaussian distribution, with its mean and standard deviation estimated from empirical data (i.c., across random seeds). This allows us to generate multiple plausible layouts and analyze the effectiveness and robustness of our selection method under training randomness. In our experiments, we adopt the number of trials as 5,000.
complementary metric: absolute privacy risk The relative privacy risk metric introduced in the main paper runs into issues when the oracle's privacy risk is zero, which occurs in the case of VMR. Since relative comparisons become defined in such cases, we also compute an absolute privacy risk measure to ensure meaningful evaluations across all
30

---

Empirical Privacy Variance
1
2
E-4
E
E-
16
Random Guess
(Ia) GPT-2-S on Enron, ACR
(1b) GPT-2-L on Enron, ACR
1.0
1.0
Accuracy
0.8
Accuracy
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Updates Compute Individual hparam
Updates Compute Individual
hparam
(2a) GPT-2-S on Enron, VMR
(2b) GPT-2-L Enron, VMR
1.0
1.0
Accuracy
0.8
Accuracy
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Updates Compute Individual
Updates Compute Individual
hparam
hparam
(3a) 1lama-2-7h on TOFU-I, AIR
(3b) Llama-2-7h on TOFU-2, AIR
(3c) Llama-2-7b on TOFU-4, AIR
1.0
1.0
0.8
0.8
0.8
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0
0.0
Updates
Compute Individual hparam
Updates
Compute Individual hparam
Updates
Compute Individual
hparam
Figure 19. Accuracy of the three heuristics across different models, datasets, and empirical privacy measures, evaluated at varying E's.
31

---

Empirical Privacy Variance
(a) Layout
(b) Step-hy-step empirical privacy scores of different selection methods
Emp. Privacy (AIR)
Emp. Privacy (AIR)
0.8
Best-utility practice
Worst-utility heuristic
Our procedure Oracle
sliding
Selected
(8096, 64, Sa-4)
0.2
1Bst
0.2
1.46
1.47
48
1.49
1.46
1.47
1.48
1.49
Utility (Test Loss)
Utility (Test Loss)
Figure 20. Setting=(Llama-2-1 7b, TOFU-, 8). (a) The layout of empirical privacy (AIR) utility (test loss) achieved by all configurations calibrated the same DP guarantee. (b) At each utility threshold u, each of the considered method (oracle, best-utility practice, worst-utilityy heuristic, and our procedure) will make selection from the subpool Pa. We plot the scores of the selected points against the thresholds.
settings.
Results. The results are presented in Figs. 21 to 23. Our procedure outperforms the baselines in almost all settings, demonstrating the effectiveness the underlying heuristics and their ability to generalize. More evaluation results on different density groups of density-adjusted TOFU can be found in Fig. 36 Appendix H.
32

---

Empirical Privacy Variance
Best-utility practice
Our pracedure
Worst-utility heuristic
(Ia) [rel] ACR, resample
(1b) [rel] ACR, meam
Rel. Priv. Risk (-]
0.4
Rel. Priv. Risk (+)
0.4
0.3
0.2
0.2
0.1-
0.1
0.0
(2a) [abs] ACR, resample
(2b) [abs] ACR,
(2c) [abs] VMR, resample
(2d) [abs] VMR.
Abs. Priv. Risk (+)
Abs. Priv. Risk (+)
Abs. Priv. Risk (+)
0.6
0.10
Abs. Priv. Risk
0.10
0.05
0.05
0.00
Figure 21. Relative privacy risk of procedure compared with baselines. Model GPT-2-S; data Enron; empirical privacy measure E (ACR, VMR): risk measure E {abs, rel), where "abs" denotes the absolute privacy risk and "rel" denotes for the relative privacy risk; layout (resample, mean), where "resample" corresponds the monte carlo approach that accounts for the training randomness, and "mean" corresponds to the approach that directly uses the mean, averaging out the training randomness. Note that GPT-2-S achieves VMR of at multiple configurations, which makes the relative risk metric invalid. Thus we omit the relative risk results for VMR and present results for the absolute risk only.
Best-utiilty practice
Our procedure
utilit heuristic
Oracie
(Ia) [rel] ACR, resample
(1b) [rel] ACR, mean
Rel. Priv. Risk
0.4
Rel. Priv. Risk
0.4
0.2
0.2
0.0
0.0
(2n) [ubs] ACR, resample
(2b) [abs] ACR, mean
(2c) [abs] VMR, resample
(2d) [abs] VMR, meam
Abs. Pfiv. Risk (+)
1.00
Abs. Priv. Risk (+)
1.00
Abs. Priv. Risk (-)
0.15
Abs, Priv. Risk (+)
0.15
0.75
0.10
0.10
0.50
0.50
0.05
0.05
0.25
0.25
0.00
0.00
0.00
0.00
2
Figure Relative privacy risk our procedure compared with baselines. Model GPT-2-L, data Emron, empirical privacy measure E {ACR, VMR): risk measure {abs, rel); layout fresample, mean]. Note that GPT-2-L achieves VMR of 0 at multiple configurations, which makes relative risk metric invalid. Thus we omit the relative results for VMR and present results for the absolute risk only.
33

---

Empirical Privacy Variance
Best-udiiiy practice
Our procedure
utllit houristic
(Ia) [re]] TOFU-1, resumple
(Ib) [re]] TOFU-2. resumple
(Ic) [rel] TOFU-4, resample
Rel. Priv. Risk (+)
0.75
Rel. Priv. Risk (+)
0.75
Rel. Priv. Risk
0.50
0.50
0.2
0.25
0.25
0.00
0.00
16
16
16
(Ia) [rel] TOFU-1, mean
(1b) [rel] TOFU-2, mean
(Ic) [rel] TOFU-, mean
Rel. Priv. Risk (+)
Rel. Priv. Risk (+)
Priv. Risk (+)
1.0
0.4
0.5
0.1
0.2
0.0
0.0
0.0
Figure 23, Relative privacy risk of our procedure compared with baselines. Model = Llamm-2-7b, data (TOFU-I, TOFU-2, TOFU-4), measure AIR, rel, layout resample.
E. Additional Experimental Setup and Results for Section 5
E.1. Privacy auditing setup and results
E.1.1. EXPERIMENTAL SETUPS
We follow the approach of Panda al. (2025) to perform privacy auditing of fine-tuned LLMSs. Our experiments focus on a single setting: fine-tuning GPT-2-S on the Enron dataset with privacy budget of and 8. We experiment with both full fine-tuning and LoRA fine-tuning and find that the method mostly gives meaningful é for full fine-tuning (i.e., g 1) but not always for LoRA fine-tuning. Thus stick to full fine-tuning in this study.
High-level procedure. As outlined in Section 5.1, aim address two questions: Q1) Does é vary with configuration?? Q2) Is é aligned with empirical privacy? To answer these, we follow structured approach.
Define range hyperparameter configurations (as shown Table 9).
2. For each configuration,
(a) Perform DP fine-tuning for each configuration on the Enron dataset with input canaries injected (explained below).
(b) After fine-tuning, i) measure the model's & using the auditing procedure and ii) calculate its ACR.
3. Finally, analyze the measured é's and empirical privacy scores on all the obtained models: 1) examine how é varies with different configurations; 2) assess the correlation between é and the empirical privacy score.
Table Hyperparameter configurations for full fine-tuning GPT-2 models on Enron with input canaries. tuple represents
(b, T. 1), c). For each configuration, perform fine-tuning using 4 random seeds.
Scenario
Configurations
(8192, 1000, 10-3 10-3.0.5)
(8192, 250, 1 10-5.0.5) (8192, 125, 1 x 10 1,.5,
GPT-2-S, Enron
(8192, 250, x 10 10-4,0.5) 0..5))
(8192, 250, 3 x 10-3,0.5)
(total=7) 8 4,8
(4096, 250,
x 10-5,0.5)
(2048, 250,
1 x 10-3 ..0.5)
Details of privacy auditing. member and non- non-memin member samples. The auditing procedure compares the losses of member and non-menne non-member canaries, then converts these loss differences into using the method proposed by Steinke al. (2024).
Privacy auditing quantifies a model's privacy leakage by measuring the ability to distinguish
34

---

Empirical Privacy Variance
2.4
0.38
1.8
<w
1.2
0.225
0.6
4
E
Figure 24. Variance of é. We follow Panda al. (2025) to produce audited lower bound, i.c., &, for each fine-tuned LLM. Each scattered point corresponds to one configuration.
Panda et al. (2025) introduce "new token canaries", which improve canary design to better expose memorization. To further maximize exposure, we adopt the following design choices, as recommended by the authors: 1) Initialize the token embeddings for new tokens to zero; 2) Precede each canary sample with unique random sequence; 3) Compute the SFT loss only the last new token for each canary sample. These design enhancements ensure that the input canaries highly effective in detecting privacy leakage.
E. 1.2. EXPERIMENTAL RESULTS
A1: å varies with configurations. Fig. 24 shows that, for both & values we consider (s 4 and 8), different configurations could lead to different é's.
A2: is not aligned with empirical privacy. Given models trained under differnet configurations, we assess correlation between their è and the empirical privacy scores (ACR). For 8, the spearman correlation between é and ACR is -0.13.
(8132, 250, 16:3)
(8192, 56-4)
(8192. 250, 3e:3)
16
0.600
0.575
0.325C
0.350
0.375
mncressinn
0.525
0.550
0.300
0.275
0.475
Increadig
0.225
0.250
decreasing
Vncre-iingg
0. 450
0.200
3.0
5,0
Incropping
2.6
4.0
1.2
1.4
1.6
E
1.8
2.0
2.0
1.00
25
1,0
1,,75,00
2.00
2.25
2.50
2.5
3.0
Figure 25. Effeet of learning rate on é and empirical privacy measures (ACR and VMR). As increases, empirical privacy measures increase, but decreases, highlighting a misalignment.
Case study on learning rate 77 Fig. 25 visualizes how é and empirical privacy scores (ACR, VMR) change as r7 increases from 5 x 10-4 to x 10-3 further to 3 x 10-3, While ACR and VMR both keep increasing, starts to drop in the second half (as increases from 1 x 10-3 to 3 x 10-3).
Role of utility. We additionally note in the above example that the change É closely follows the change in utility (which first improves from 2.94 to 2.87, and then degrades 3.08 at the largest 7)). We thus introduce the quantity utility into the
35

---

Empirical Privacy Variance
picture and measure the Spearman rank correlation between é, ACR and utility, showing -0.71 correlation for (é, utility) and -0.05 correlation for (ACR, utility). This indicates that é and empirical privacy not aligned, but instead é is strongly correlated with utility.
E.2. Additional results privacy profile
Experimental setup and results. We extend privacy profile analysis in Section 5.2. Specifically, we first fix the dataset size at n 180, 000 and set the target privacy requirement to (4, 10 10-600000000 -DD,, then consider:
Scenario 1: Fix b, vary T: we analyze the effect of increasing T {150, 500, 2000) while keeping b 4096 fixed.
Scenario 2: Fix T. vary b: we analyze the effect of increasing b E {2048, 4096, 8192] while keeping T 1000 fixed.
We also consider the following privacy accountants: Rényi Differential Privacy (RDP) accountant (Mironov, 2017), PRV accountant (Gopi et al., 2021), and Connect-the-Dout accountant (Doroshenko et al., 2022).
Table 10 summarizes the noise multipliers a required to achieve (4, 10 --DP in the two scenarios, as computed by the three privacy accountants. We observe that RDP provides looser bound, while PRV and Connect-the- yield similarly tight estimates (as reflected by their noise multipliers).
Table 10. The noise multiplier returned by different privacy accountants in two scenarios. Setup 1: 180, 000, target=(4, 10 ---P
Scenario
RDP
PRV
Connect-the-LLe
= 150
a 8.52 x 10
a 8.00 x 10"1
a 7.99 x 10
Fix vary T 180, 000, 4096
T = 500
a 1.01
9.62 x 10-1
9.61 x 10-1
T 2000
1.50
1.43
1.43
2048
= 8.57 x 10-1
a 8.14
10-1
a 8.14 10-1
2. Fix 7' vary b n 180, 000, 1000
b 4096
a 1.20
0 1.14
1.14
= 8192
a 2.01
a == 1.91
= 1.91
Fig. 26 presents the corresponding privacy profiles, revealing the following consistent patterns:
The privacy profiles intersect at only one point, corresponding to the target privacy requirement.
For fixed target privacy requirement, increasing (which corresponds to larger T or b) decreases in the bottom-right (small regime) but increases in the top-left (large 8 regime).
We further study the impact of e and dataset size. We vary E E f1, 2, 4, 8) under Scenario / and present the results in Fig. 27, and vary n (45k, 90k, 180k, 360k) under Scenario and present the results in Fig. 28. We show that the difference between privacy profiles different 7's enlarge increases (or increases), when the noise multiplier decreases (see Table 12 for the calculated noise multiplier under different settings).
An alternative setup and results. We also consider an alternative experimental setup with = 50, 000 and target privacy requirement of (8, 5 x 10- -DDP. In scenario 1, we fix = 4096 and vary T' E {250, 500, 1000]. In scenario 2, we fix T = 500 and vary E (1024, 2048, 4096). The results are presented in Table 11 and Fig. 29, Despite the variations in setup, the patterns of privacy profiles remain consistent.
36

---

Empirical Privacy Variance
(a) RDP accountant
(b) PRV accountant
(c) Conmect-the-Dore accountant
Scenario 1: Fix ó vary T
10
10
10
10
(4, 10-6)
10-4
(4, 10-9)
10
(4, 10-6)
8(e)
5
6(E)
10-7
10-7
10-7
T=150
T=150
T=150
T=500
T=500
T=500
10
10-10
10
T=2000
T=2000
T=2000
6
6
8
0
2
4
Scenario 2: Fix T vary b
10-1
10"1
10-1
10-4
(4, 10
10-4
(4, 10
10-4
(4, 10 66)
6(E)
ō(E)
5(E)
10
b=2048
10 7
b=2048
10
b=2048
b=4096
b=4096
b=4096
10
10
10
b= 8192
b-92
b= 8192
4
6
2
4
8
Figure 26. Privacy profiles under different privacy accountants, setup 1: 180, 000, target=(1, 10---- Columns correspond different privacy accountants: (a) RDP accountant, (b) PRV accountant, and (c) Connect-the- tote--ott accountant. The first row (Scenario shows the impact of varying T while fixing 4096, and the second (Scenario 2) shows the effect varying b while fixing
T 1000.
2
4
10
10-1
10-1
10
8
10
(1, 10-
10
[2, 10-1)
((6)
10
(4, 10-)
6iE)
10
[8, 10-*)
10
T=150
10-7
T=150
10
T=150
10
T=150
T=500
T=500
T=500
T=500
10-20
10-12
10-1:
T-2000
T-2000
T-2000
T-2000
0.0
0.5
1.0
1.5
6
10
15
Figure 27. Privacy profiles different target e E 2, 4, (different columns), n = 180, 000, 4096, target = (s, 10 9)-DP,
accountant Connect-the-Dott accountant. The difference between the privacy profiles of different T's enlarge increases.
T2 = 45k
72 90k
72 180k
72 360k
10
10-1
10
10
4
10-5)
8iE)
(4, 10 $1
(4, 10 10-9)
6iE)
(4, 1.10
10-7
T=150
10-7
T=150
10
T=150
10
T=150
10-10
10-1
10-20
10-15
T-2000
T-200D
T-2000
T-2000
6
2
6
{45k, 90k, 180k, 360k) (different columns), 4096, target (4, 10 -D, accountant = Connect-the- tott accountant. The difference between the privacy profiles of different 7's enlarge
Figure Privacy profiles for different dataset size
increases.
F. Results under the Worst-Case Privacy Measure
In this section, we investigate whether switching to a worst-case empirical privacy measure has any impact on the conclusions we obtained in the main paper. We achieve this by taking the max of the measured empirical privacy scores on the set of
37

---

Empirical Privacy Variance
Table 11. The noise multiplier returned by different privacy accountants in two scenarios. Setup 2: n 50, 000, target= (8, 5 10 -)- -DD
Scenario
RDP
PRV
Connect-the-Dots
T 230
о 1.15
J 1.09
a 1.09
Fix vary? 7 50, 000, 4096
T 500
a 1,43
1.36
1.36
T 1000
1.87
1.77
1.77
b = 1024
a 7.06 x 10-1
0 6.72 x 10-1
a = 6.72 x 10-1
vary b 7 50, 000, T 500
b 2048
9.33 x 10
8.90 x 10-1
8.89 x 10-1
4096
1.43
1.36
1.36
(a) RDP accountant
(b) PRV accountant
(c) Connect-the- thotoss accountant
Scenario 6 vary T
10
10
10"1
10-3.
10-3
10 -4
(8, x 10-6)
5(e)
10
(8, 5 x 10-6)
6(e)
10
(8, x 10
10-7
1077
T=250
10-7
T=250
T=250
T=500
T=500
T=500
10-9.
10-9
10-10
T=1000
T=1000
T=1000
2.5
5.0
7.5
10.0
12.5
2,5
5.0
7.5
10.0
12.5
2.5
5.0
7.5
10.0
12.5
g
Scenario Fix I vary
10
10
10
10-3
10-3
10 -
10-5
(8, 10
88 5 10 10-6)
10-5
88, 5 10-6)
10-7
b=1024
10-7
b=1024
10-7
b=1024
b=2048
b=2048
b=2048
10-9.
10-9.
10-9.
b=4096
b=4096
b=4096
10
10
10
E
Figure Privacy profiles under different privacy accountants, setup 2: 50, 000, target=(8, x 10 -0-9)-DP Columns correspond
different privacy accountants: (a) RDP accountant, (b) PRV accountant, and (c) Connect-the- accountant. The first row (Scenario shows impact of varying 7 while fixing 4096, and the second row (Scenario 2) shows the effect varying while fixing T' 500.
secrets, instead of the average as done in the main paper (see the end of Section 3.1).
We present detailed results below, supporting that all conclusions remain unchanged.
Empirical privacy. Fig. 30 shows that empirical privacy variance happens for both the average-case and the worst-case privacy measure (the former the same as Fig. 2(a)).
Regression results. In Table 13, present the regression results using the worst-case privacy measure the response variable and compare with that obtained using the average-case privacy measure (as presented in Table 2 in the main paper).
As shown Table 13, the conclusions drawn on the average-case regression results (in Section 4.1) remain to hold on the worst-case regression results- (1) all individual hyperparameters have positive coefficients with significant p-values, thus increasing any individual hyperparameter leads worse empirical privacy; (2) The coefficient of log is smallest, meaning that under fixed compute, increasing (while decreasing I proportionally) improves empirical privacy; (3) The coefficient of log is larger than log C, meaning under fixed updates, decreasing 7) (while increasing C proportionally) improves empirical privacy.
Correlation with audited We measure the spearman rank correlation between the audited é and the worst-case empirical privacy score, replicating the exercise in Section 5.1. Similar to the conclusion obtained on the average-case privacy measure
38

---

Empirical Privacy Variance
Table 12. The noise multiplier achieved at different settings. For all settings, we fix b = 4096 and set the target to (4, 10 J-DP;
accountant Connect-the Dots. (Left): vary e E {1, 2, 4, 8) and fix 180k; (right): vary n {45k, 90k, 180k, 360k) and fix E
Varying
2
Varying n
45k
90k
180k
360k
T 150
1.51
1.05
0.80
0.62
150
1.61
1.05
0.80
0.67
2.34
1.41
0.96
0.72
500
2.60
1.46
0.96
0.75
2000
4.40
2.42
1.43
0.96
2000
4.94
2.56
1.43
0.92
GPT-2-5
GPT-2-L
(a) average-case privacy measure
(b) worst-case privacy measure
4
0.780
Emp. Privacy (ACR-worst)
16-
Emp. Privacy (ACR)
0.711
0.459
12-
3
2.398
0,546
0.342
8
1.951
1.152
0.262
1.212
0.169
0.665
4
0.3668
2
1
Figure 30. Empirical privacy variance happens for both average-case (left) and worst-case (right) privacy measures.
Table 13. Comparison on regression results on average-case privacy measure vs. the worst-case privacy measure.
(a) Regression on individual hyperparameters
Enron (average-case)
Enron (worst-case)
Variable
Coef.
p-value
Coef.
p-value
Batch size (log b)
0
1 x 10-5
0.38
2 10-3
Iterations (log 7')
0,37
<2 x 10
1,31
5 10
Learning rate (log m)
0,51
x 10-15 15
1880
9 10 12
(b) Regression on composite hyperparameters
Enron (average-case)
Enron (worst-case)
Variable
Coef.
p-value
Coef.
p-value
Compute (log C)
0
2 10 12
0.74
3 10-9 1.89
Learning rate (log n)
0
10 18
10
Notes: ""p < 0.001, ""p < 0.01. "p < 0.05. The response variable (empiri- cal privacy
ACR.
in Section 5.1, here we obtain correlation of -0.29 between the two, showing that the two are not positively correlated. Moreover, the correlation between the average-case privacy measure and the worst-case privacy measure is as high as 0.90.
39

---

Empirical Privacy Variance
G. Compute Resources and Runtime
Compute resources. We run our experiments on three main computing environments. The first setup has four NVIDIA H100 GPUs (each with 80GB of HBM3 memory) and an Intel Xeon Platinum 8468 CPU (192 cores). The second setup also has four NVIDIA A100 GPUs (each with 80GB) and an AMD EPYC 7643 CPU (192 cores). For larger-scale experiments, a cluster consisting of 100 nodes, where each node contains four 40GB A100 GPUs. We manage all jobs on the cluster using the SLURM scheduling system.
Empirically, training on an H100 achieves an approximately two-fold speedup over an A100, To provide a clear sense of computational requirements, below we report GPU hours in H100 unit.
Runtime.
Runtime consists of both training and evaluation phases.
Training runtime. We conduct all fine-tuning experiments on NVIDIA H100 or NVIDIA A GPUs, using a single GPU per run. Empirically, training on an H100 achieves an approximately two-fold speedup over an A100. To provide clear sense of computational requirements, we report GPU hours measured on H100 for both the smallest and largest compute configurations, as runtime scales approximately linearly with compute C.
Table 14, Fine-tuning runtime for different compute configurations on different (model, datasct) pairs.
Model
Configuration (b, 7)
Compute C Runtime (H100)
GPT-2 Models on Enron
GPT-2-S (smallest)
(1024, 125)
128,000
7.5 min
GPT-2-S (lurgest)
(8192, 1000)
8,192.000
7.5 hrs
GPT-2-L (smallest)
(1024, 125)
128,000
30 min
GPT-2-L (largest)
(8192, 500)
4,096,000
17 hrs
Llama-2 Models on TOFU
Llama-2-7B (largest)
(4096, 256)
1,048,576
Llama-2-13B (largest)
(4096, 2.5))
1,048,576
8.5 hrs
GPT-2 models on Enron: For GPT-2-S, the smallest compute configuration, (6, (1024, 250) with C 256, 000, completes in 15 minutes, while the largest configuration, (8192, 1000) with C 8, 192, 000, requires approximately 7.5 hours. For GPT-2-L, the smallest setting, (1024, 125) with C = 128, 000, has training time of 30 minutes, whereas the largest configuration, (8192, 500) with C' = 4, 096, 000, takes 17 hours.
Llama-2 models on TOFU. For Llama-2 models, the most computationally expensive configuration, (6,T) (4096)256) with C 1, 048, 576, completes in 5 hours for Llama-2-7B and 8.5 hours for Hlma-2213BB. Runtime scales proportionally for smaller configurations.
A summary of the above is provided in Table 14. In total, our fine-tuning experiments across various models, datasets, and configurations accumulate over 5,000 GPU hours (H100-equivalent compute).
Evaluation runtime. For evaluation, VMR and AIR involve only prompting and is therefore lightweight. In contrast, ACR evaluation is significantly more expensive due to the nature of the search algorithm.
The runtime of ACR per secret per model varies considerably-thee search algorithm finishes quickly for a well-memorized secret, but can keep going on for long for a secret that is barely memorized. We present histogram of the runtime for individual search runs (per model per secret) in Fig. 31; note that both x and y axes are on log scale. On average, ACR evaluation on one single secret takes 100 seconds for GPT-2-S and 5,300 seconds for GPT-2-L, The total compute for evaluation amounts to approximately 5,000 GPU hours (H1100-quivalent compute).
40

---

Empirical Privacy Variance
(a) Runtime for ACR evaluation on GPT-2-S
(b) Runtime for ACR evaluation on GPT-2-L
10a
10*
Frequency [Log Scale]
Frequency [Log Scale]
10.
10a
101
10*
10a
10
10a
10'
10a
10
10a
10a
10
Runtime (seconds) [Log Scale]
Runtime (seconds) [Log Scale]
Figure 31. Distribution of evaluation runtime for ACR for (a) GPT-2-S and (b) GPT-2-L. The æ-axis represents runtime in seconds logarithmic scale, while the y-axis indicates the frequency of occurrences (also on log scale).
H. Complete Sets of Results
Best-utility practice Worst-utility heuristic
Our procedure
Oracle
2
8
Emp. Privacy (ACR)
Emp. Privacy (ACR)
2.25
Emp. Privacy (ACR)
3.0
1.6
GPT-2-S
2.00
2.5
1.4
1.75
2.0
1.2
1.50
1.5
3.10
3.15
3.20
3.025
3.050
3.075
3.100
2.950
2.975
3.000 3.025 3.050
Utility (Test Loss)
Utility (Test Loss)
Utility (Test Loss)
4.0
Emp. Privacy (ACR)
Emp. Privacy (ACR)
3.5
Emp Privacy (ACR)
2.5
3.5
GPT-2-L.
3.0
2.0
3.0
2.5
2.5
1.5
2.0
2.0
2.76
2.82 Utility (Test Loss)
2.78
2.80
2.84
2.675
2.700 2.725 2.750 2.775
2.60
2.70 Utility (Test Loss)
2.65
Utility (Test Loss)
Figure 32. A complete set of visualizations the selection quality of different methods. Dataset: Enron; Model: GPT-2 models. our procedure consistently outperforms the others (i.c., best-utility practice and worst-utility heuristic) for both models (GPT-2-S and GPT-2-L) and across different values.

---

Empirical Privacy Variance
Best-utility practice
Our procedure Oracle
Worst-utilly heuristic
16
Emp. Privacy (AIR)
0.28
Emp. Privacy (AIR)
0.28
Emp. Privacy (AIR)
0.4
0.26
TOPU-1
0.26
0.24
0.24
0.3
0.22
0.22
0.2
1.43
1.44
1.45
1.46
1.42
1.43
1.44
1.45
1.40
1.41
1.42
1.43
1.44
Utility (Test Loss)
Utility (Test Loss)
Utility (Test Loss)
0.35
Emp. Privacy (AIR)
Emp. Privacy (AIR)
Emp. Privacy (AIR)
0.4
0.6
TOFU-2
0.30
0.3
0.25
0.20
0.2
0.2
1.45
1.46
1.47
1.43
1.44
1.45
1.46
1.42
1.43
1.44
1.45
Utility (Test Loss)
Utility (Test Loss)
Utility (Test Loss)
Emp. Privacy (AIR)
0.40
Emp. Privacy (AIR)
0.8
Emp. Privacy (AIR)
0.35
0.8
TOFU-4
0.6
0.30
0.4
0.25
0.4
0.20
0.2
1.47
1.48
1.49
1.50
1.46
1.47
1.48
1.49
1.46
1.47
1.48
1.49
Utility (Test Loss)
Utility (Test Loss)
Utility (Test Loss)
Figure A complete set visualizations on selection quality different methods. Dataset: paraphrase-scaled TOFU; Model: Llama-2-7h. The advantage of our procedure over the others (i.e., best-utility practice and worst-utility heuristic) enlarges with the increase of the dataset size (vertically, from TOFU-1 to TOFU-4) and the increase e (horizontally, from 4 to s = 8).
42

---

Empirical Privacy Variance
Best-utility practice
Our procedure
Worst-utility heuristic
Oracle
group(density=3)
group(density=5)
Emp. Privacy (AIR)
Emp. Privacy (AIR)
0.6
Emp. Privacy (AIR)
0.8
TOFU3u5
0.6
0.5
0.6
0.4
0.4
0.4
0.3
0.2
0.2
1.46
1.47
1.48
1.49
1.46
1.47
1.48
1.49
1.46
1.47
1.49
Utility (Test Loss)
Utility (Test Loss)
Utility (Test Loss)
All
group(density=2)
group(density=6)
0.5
Emp. Privacy (AIR)
Emp. Privacy (AIR)
Emp. Privacy (AIR)
0.8
0.6
TOFU2.0
0.4
0.6
0.4
0.3
0.4
0.2
0.2
1000
1.46
1.47
1.48
1.49
1.46
1.47
1.48
1.49
1.46
1.47
1.48
1.49
Utility (Test Loss)
Utility (Test Loss)
Utility (Test Loss)
All
group(density=1)
group(density=)
1.0
Emp. Privacy (AIR)
Emp. Privacy (AIR)
Emp. Privacy (AIR)
0.6
0.3
0.8
1OFU1:7
0.6
0.4
0.2
0.4
0.2
0.2
1.47
1.48
1.49
1.47
1.48
1.49
1.47
1.48
1.49
Utility (Test Loss)
Utility (Test Loss)
Utility (Test Loss)
Figure 34. A complete set of visualizations on the selection quality of different methods. Dataset: density-adjusted TOFU: (TOFU TOFUze, TOFUsss (reflected rows); Model: Llama-2-7b;
Groups: all, low- high-density groups (reflected columns). Specifically, "density=1" means no augmentation; "density=x" means augmenting the dataset with additional - 1 paraphrased texts besides original one, for each sample. The advantage our procedure over others (i.e., best-utility practice worst-utility heuristic) enlarges with the increase of the secret density. Notably, even a small density ("density=2", meaning the secret occurs for less than 0.06% all samples), our procedure already starts demonstrate advantages.
43

---

Empirical Privacy Variance
High
Low
Random guess
(a) 10
(b) TOFU28
(c) TOFU
1.0
1.0
1.0
0.8
0.8
Accuracy (+)
0.8
Accuracy
Accuracy
0.6
0.4
0.2
0.2
0.2
0.0
0.0
0.0
Updates Compute Individual hparam
Updates Compute Individual
Updates
Compute Individual
hparam
hparam
Figure 35. Accuracy of the three heuristics. Dataset: demsity-adjusted TOFU: (TOFU TOFU2.s. TOFU3.5) (reflected columns);
Model: Llama-2-7b; 8 groups = all, low, high density groups; measure AIR.
Best-utility practice Worst-utility heuristic
Our procedure
aa) TOFU
(b) TOFU26
(c) TOFU
1.50
1.50
1.50
1.25
1.25
Rel. Risk (-)
1.25
Rel. Priv. Risk
1.00
Rel. Priv. Risk
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
Low density=1
High density=7
Low density=2
High density=6
High density=5
density=3
Figure 36. Relative privacy risk of our procedure compared with baselines. Dataset: density-ajjusted TOFU: (TOFU TOFU TOFUass (reflected columns); Model: Llama-2-7h; e = 8, measure AIR, risk rel, layout = mean. (Refer to Fig. 21 for explanations "rel" and "mean" and Fig. 34 for explanations on "density"". Similarly, we observe that our procedure has large advantage over the others, which becomes apparent even low densities (e.g., density=2) and grows as the density increases.
44


Reviews:
[{"review_id": "79W0HlnS9m", "reviewer": "Reviewer_E9Wi", "summary": "The paper investigates privacy implications when fine-tuning language models using DP-SGD. Importantly, when using DP-SGD, the same (ε, δ)-DP guarantee can be achieved through multiple hyperparameter configurations (batch size, number of iterations, learning rate, etc.).\nThe authors' key finding is that despite having identical theoretical privacy guarantees, models trained with different hyperparameter configurations exhibit variations in empirical privacy (which they measure through memorization metrics). This \"empirical privacy variance\" means that the theoretical (ε, δ) parameters alone don't fully characterize the practical privacy leakage of the model, raising questions about the practical interpretation of DP guarantees in language model fine-tuning. Authors suggest some new heuristics to hyper-parameter tune the parameters of DP-SGD so that we get a better trade-off between utility and empirical privacy.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review.\n\n> Summay of our key points\n\n- We measure empirical privacy in controlled settings\n- We conduct extensive experiments (> 10,000 H100 GPU hours) for robust findings\n- Our contributions go well beyond “showing the existing variance”\n- We recommend empirical privacy as an additional dimension of privacy alongside DP, rather than targeting it in isolation\n\n> Detailed responses\n\n**Sources of variance.** The empirical privacy variance we observe is **conditioned on** fixing the dataset, model, training algorithm and empirical privacy measure (Sec 3.3 para 1). We also show variance from *training randomness* is small (Appendix F.3).\n\nThe reviewer raised variance in *measuring empirical privacy scores*. We believe this is not the case for our experiments. For ACR, we run the optimization with 3 seeds and report the highest score for more accurate estimates; VMR and AIR average 10 stochastic decodings to reduce variance (Appendix E.6).\n\n**Comparison with related works.** Appendix A discusses related work.\n Hayes et al. (2023) and Kaissis et al. (2024) show the attack success rate of a reconstruction attack (which relies on strong assumptions) can vary for mechanisms calibrated to the same DP guarantee. We focus on realistic privacy risks that emerge in language model interactions, leading to the concept of empirical privacy variance. \n\nBeyond this, we have more substantial contributions: 1) extensive experiments across various models, datasets, secrets, and empirical privacy measures; 2) in-depth qualitative analysis of the impact of hyperparameters and investigation of the cause; and 3) broader implications: for practitioners, we provide heuristics that enhance empirical privacy; for researchers, we expose the hidden cost of hyperparameter tuning; for policy makers, we discuss how this phenomenon could complicate standardization. \n\nAs Reviewer MX8d noted, “...this is the first time empirical privacy variance is proposed and investigated on a practical level. The experiment results provide compelling evidence that the problem is general, and likely to impact a variety of models using DP-SGD… the implications of empirical privacy variance are potentially broad and of interest to the DP community.”\n\nGiven the above, we kindly ask the reviewer to reconsider the rating based on our contributions.\n\n**More empirical validation.** Figs. 7 and 10 and Appendix G (Figs. 18, 20-22) confirm our heuristics’ generality across models, datasets, $\\varepsilon$’s and empirical privacy measures. Could the reviewer kindly suggest additional evidence they would like to see?\n\n**Issues with ACR/VMR.**  While ACR has limitations, we believe a lower ACR/VMR score indicates lower memorization and thus better privacy **in general**. Importantly, we don’t advocate for targeting these **in isolation**. Our heuristics operate on configurations calibrated to a specified DP guarantee (Sec 4.3), adding a dimension not captured by DP.\n\n**Q1 - certification.** We mean: even a legislative body determines that a model with $\\varepsilon^\\star$ passes their privacy tests, this does not imply that models with a stricter privacy budget ($\\varepsilon \\le \\varepsilon^\\star$) will pass. As illustrated in Fig 4, all models in the red region will fail.\n\n**Q2 - theory for ACR/VMR.** The statement “ACR and VMR are small for DP mechanisms” is incomplete: 1) “small” is not quantitatively defined; 2) the privacy parameters of the DP mechanism are not specified. We do not have a theory for it—as we point out in the caption of Fig. 4, bridging DP and empirical privacy measures is generally hard.\n\nThe reviewer might be suggesting that without theoretical guarantees, a model with DP can have large ACR/VMR, so our findings are not surprising. But we want to emphasize: what we believe makes our finding interesting is the **simultaneous existence** of both high and low ACR/VMR under the same theoretical privacy budget (see Fig. 2). This naturally motivates our study in Sec. 4 to target the low end by proper hyperparameter selection. \n\n**Q3 - hypothesis in Sec 5.1.** The hypothesis here is two-fold. First, while all mechanisms are calibrated to the same $(\\varepsilon, \\delta)$-DP, the final LLMs (where we measure empirical privacy) might have different “real” $\\varepsilon$’s, all upper bounded by $\\varepsilon$. Second, these “real” $\\varepsilon$’s reflect empirical privacy: a higher value corresponds to poorer empirical privacy.\n\n**Q4 - why relevant.** For Laplace or Gaussian mechanism, $\\varepsilon$ uniquely determines the privacy level via the single hyperparameter that controls the noise scale. In contrast, the compositional nature of DP-SGD allows infinite configurations to achieve the same $\\varepsilon$, each with unique privacy levels. Fig. 4 highlights that empirical privacy variance is a unique characteristic of DP-SGD, and more broadly, of DP algorithms that involve composition.\n\nPlease let us know if you have further questions. Thank you!", "claims_and_evidence": "- The main claim of the paper is that there is a variance in \"empirical privacy\" when the \"theoretical privacy\" is fixed. This claim is supported by their extensive experiments. \n\n- There are other claims about the role of dataset size and model size and their role in empirical privacy. These claims motivate their hyperparameter selection. I think they have moderately supported these claims but for suggesting general heuristics for hyperparameter turning I would expect more empirical validation.\n\n- One of the claims of the paper which I did not find a support for is that the choice of hyperparameters is the main factor explaining the variance. However, I think there could be other sources contributing to this variance. For example the attacks might be more successful for smaller models. Or there could be variance stemming from the choice of dataset.", "methods_and_evaluation_criteria": "The method for training private models (DP-SGD) and also their empirical privacy evaluation methods (ACR and VMR) make sense.", "theoretical_claims": "- N/A", "experimental_designs_or_analyses": "Their experimental design makes sense to me.", "relation_to_broader_scientific_literature": "As stated by authors, the finding that same privacy parameters could lead to different implications for empirical attacks is already studied in the literature. I think authors need to clarify their contribution in comparison with these works a bit better.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "- The first weakness with the paper is that there is an implicit assumption that the source of variance is the selection of hyperparameters. I think the methods used for privacy evaluation are actually more of a source for variance. For example, ACR itself has a very brittle optimization step that could fail in finding the optimal solution (it does most of the time). I\n\n\n- argue that ACR and VMR should not be set as a goal. It is only a metric that could indicate privacy violations. Having a low ACR is not at all and indication of privacy. So I argue performing hyperparameter tuning to lower these rates does not necessarily lead to more private models. \n\n- The main contribution of paper seems to be about showing the existing variance. As authors argue, this is already discussed in the literature.\n\n- Writing of the paper is confusing. I do not understand many sentences in the paper.\n\n- As a strength, I find the empirical studies in the paper rigorous and impressive.", "other_comments_or_suggestions": "See above", "questions_for_authors": "- I did not understand this sentence: \"A direct consequence is that, in DP-SGD, ε cannot be used for certification: a model calibrated to a given ε*, deemed to meet privacy requirements, cannot ensure compliance formodels with stricter DP guarantees (ε ≤ ε∗).\" Could you please elaborate?\n\n- Is there a theory that suggests ACR and VMR should be small for differentially private mechanisms? Your seem to be assuming this but I have not seen a theoretical analysis on this.\n\n- The two hypothesis in section 5.1 are not clear. Can you please elaborate? The first hypothesis is somewhat clear, you are saying that there is gap in our understanding of true epsilon values. But I have a hard time understanding the second hypothesis. \n\n- I don't understand this sentence, please elaborate: Why is this relevant? Consider classic DP mechanisms such as the Laplace and Gaussian mechanisms (Dwork et al., 2014). Their noise parameter (scale parameter b for Laplace and σ for Gaussian) inversely correlates with ε and uniquely determines privacy risk: increasing it lowers the signal-tonoise ratio, making it harder for adversaries to extract meaningful information. This establishes a one-to-one, monotonic ε-to-risk relationship. In contrast, the composition nature of DP-SGD results in a one-to-many ε-to-risk relationship, making ε insufficient to fully capture privacy risk.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JcdNbegG8D", "reviewer": "Reviewer_u6Yf", "summary": "The paper empirically estimates the privacy loss of language models fine-tuned with DP-SGD in many different configurations, including different hyperparameters, model sizes, and dataset characteristics. The paper finds that models calibrated to the same DP guarantee can have very different empirical privacy losses depending on the full configuration, and calls this phenomenon \"empirical privacy variance\". The paper investigates the effect of hyperparameters further, and finds rules of thumb on how the hyper parameters change empirical privacy loss that are derived from experimental results, and generalise in additional experiments. The paper also investigates possible causes of empirical privacy variance, but does not find a clear cause.\n\n## update after rebuttal\n\nI'm keeping my score due to the subsampling discrepancy in the paper. I have read the extensive final response from the authors. I agree with the authors that there are reasons to believe the conclusions regarding the subsampling rate in Section 4 would not change with correct privacy accounting, but if these were to be included in the paper, they would need a massive caveat that basically says \"we don't really know if these conclusions are correct\". I also agree that training any kind of large model with Poisson subsampled DP-SGD is very difficult in practice, which is why I suggested recalculating the privacy bounds for the models that have already been trained. \n\nHowever, I disagree with the other arguments from the authors, which boil down to arguing that the paper should be accepted because previous papers with the same issue have also been accepted. This is not a good argument, as consistently applying it would mean that bad research practices cannot be eliminated if they are ever accepted.\n\nIn summary, I think the paper could be accepted if the results are presented properly, which is why my score is a 2 instead of a 1. However, this presentation would be very different from the current one, and include many limitations and caveats not in the submission, so I'm keeping my score at a 2 instead of a 3.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for raising the subtlety between implementing DP-SGD with shuffled batches, but performing privacy accounting as if Poisson subsampling was used.\n\nIn the camera-ready version, we are happy to expand the discussion and move it to the main paper under a “limitation” section. Additionally, we would like to clarify a few more points.\n\nWhile as pointed out the exact $\\\\varepsilon$ values are incorrect, we believe that the qualitative observations we make are still relevant. For example, in the plots where we have $\\\\varepsilon$ on the x-axis e.g. Figures 2, 3 and 10, if the correct privacy accounting of shuffle-based DP-SGD were to be applied for the specific parameter settings we have, **it would only re-scale the x-axis in some (non-linear) manner**, but the empirical variance of the y-axes will remain the same which is precisely the main take-away from this paper. That is, the x-axis, instead of being supported on $\\\\varepsilon$ in $\\\\{1, 2, 4, 8, 16\\\\}$, would be supported on some other different values, though still monotonically increasing.\n\nThus, while the observations made by Chua et al. (2024b;c) are valuable, we consider them **relevant primarily if some model is released after training on truly sensitive data, with a claimed privacy guarantee that is incorrect.** But in our case, we feel this observation is not that critical, as it just amounts to some rescaling of values which do not affect the qualitative conclusions from our study. But thank you for raising this; we will elaborate more on this in the revision.\n\nAdditionally, we note that Chua et al. (2024c) was posted on arXiv on Nov 6, 2024, which is within three months of the ICML 2025 submission deadline (Jan 30, 2025). According to [the ICML 2025 reviewer instructions](https://icml.cc/Conferences/2025/ReviewerInstructions),  this qualifies as a **“concurrent work”** that “Authors are not expected to discuss”. In fact, while we are aware of the earlier work Chua et al. (2024b), which highlights the gap between these two sampling approaches, we were not aware of any efficient implementations of Poisson subsampling for DP fine-tuning LLMs: 1) Chua et al. (2024c) was not available online when we initiated our work; 2) They conducted experiments on MLPs only, and extending the implementation to transformer training is non-trivial and warrants further research. As a consequence, we adhere to the conventional DP fine-tuning framework for LLMs that employs shuffle-based DP-SGD. \n\nSince this was the main concern raised in the review, we hope the reviewer could reconsider the rating if the above response adequately addresses their concern. We are happy to answer any other questions during the discussion period if it helps to clarify this.\n\n> For other minor comments--\n\n**ACR underestimates the privacy loss.** In our studies, the measured empirical privacy scores depend heavily on the chosen set of secrets. The reviewer is correct that using a shorter (sub)string as the secret or not requiring an exact match could result in a lower measured privacy loss. However, the magnitude of these scores is not our primary focus; rather, our arguments center on the **variance** of the empirical privacy scores.\n\n**Figures.** Thanks for the suggestions, we will incorporate them in the camera-ready version.\n\n**Clarifications on “deemed to meet privacy requirements”.** Here's what we intend to convey: even if a legislative body determines that a model with a privacy budget $\\varepsilon^\\star$ passes their privacy tests, this does not imply that models with a stricter privacy budget ($\\varepsilon \\le \\varepsilon^\\star$) will pass. For instance, as illustrated in Fig 4, all models in the red region will fail to pass the privacy tests.", "claims_and_evidence": "The paper's claims are supported by extensive experiments. However, the mismatch between shuffling-based subsampling and Poisson subsampling is a major issue with the experiment design, more details in the specific section.", "methods_and_evaluation_criteria": "The ACR metric seems to potentially underestimate the privacy loss for the secrets in the Enron dataset, since the metric only looks (as I understand it) for reproductions of the exact secret string, not other strings that would reveal the same secret. For example, it may be easier to get a model to produce the string \"Richard B Sanders/HOU/ECT\" than the full secret \"Forwarded by Richard B Sanders/HOU/ECT\". In this case, ACR would be based on the prompt that produced the latter string, though the former reveals almost as much.", "theoretical_claims": "The paper does not make theoretical claims.", "experimental_designs_or_analyses": "The paper does not discuss the implications of the mismatch between using shuffling-based subsampling in practice and Poisson subsampling in privacy accounting that the paper's experiments suffer from. As noted by Chua et al. (2024b;c), there is a difference between the subsampling amplification that the two subsampling schemes provide, leading to different privacy bounds if accounted for correctly. As a result, all $\\epsilon$ values shown in the paper are incorrect. \n\nThis does not substantially change many results of the paper. For example, the $\\epsilon$ values in Figure 2 would be adjusted, but all other points would remain the same, and the conclusions drawn from the figure would not change. However, the results examining the effect of the batch size could be affected, since this effect could depend on the subsampling scheme.\n\nAs is, I can't recommend accepting a paper where all $\\epsilon$ values are incorrect, and the only mention of this issue is a footnote that brushes this off as something outside the scope of the paper. However, given the otherwise high quality of the paper, I could recommend accepting if this issue were prominently discussed, for example by making it clear that the $\\epsilon$ values are only estimates, and that the batch size may not behave the same way with a correctly computed $\\epsilon$. Alternatively, you could do the correct privacy accounting manually, since doing that with parallel composition is fairly simple. See the start of Section 3 in Chua et al. (2024c) for the analysis with deterministic subsamples, and Theorem 4.1 in Chua et al. (2024b) for a proof that the same analysis gives a valid privacy bound for any shuffling.", "relation_to_broader_scientific_literature": "The relevant literature is discussed for all findings of the paper.", "essential_references_not_discussed": "No essential references missing.", "other_strengths_and_weaknesses": "The topic of the paper is important. The results follow a general pattern that has been found in existing work, but the empirical privacy perspective on this pattern is both novel and important.", "other_comments_or_suggestions": "- In Figure 3, the \"increasing dataset size\" is easy to interpret as the x-axis label for the top row.\n- Lines 267-270 (left): the sentence \"a model calibrated to a given $\\epsilon^*$, deemed to meet privacy requirements\" is confusing. I initially interpreted this to mean that $\\epsilon^*$ is deemed to meet privacy guarantees, but the argument only makes sense if the model is deemed to meet privacy guarantees independently of the $\\epsilon^*$.\n- In Figure 5, the x-axis label is confusing, since higher utility should be better, but higher test loss is worse.\n- Figure 9 is too small to read easily.", "questions_for_authors": "No questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "17t48cAaOh", "reviewer": "Reviewer_MX8d", "summary": "This paper uses the concept of empirical privacy variance to show that models trained with DP-SGD under the same $(\\epsilon, \\delta)$ guarantee but using different hyperparameter settings can yield varying empirical privacy. Empirical variance metrics are defined that quantify how much a model memorizes information. The experiments reveal varying scores under these metrics depending on different hyperparameter configurations, with generality across multiple dimensions of the problem (different empirical variance metrics, secret subsets, model sizes...). Regression analysis is used to measure the relationship between DP-SGD hyperparameters (including joint effects) and the empirical variance score. Practical hyperparameter selection heuristics are proposed to mitigate this phenomenon. Finally, hypotheses on the cause of empirical privacy variance are discussed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our claims and methods as clear, sound and relevant, and our contributions to be of broad interest to the community. We will take the reviewer’s feedback and incorporate a more detailed related work section into the main body, incorporating discussions in our response to Reviewer E9Wi. We will also include a limitation section discussing what has not been covered in our work but deserves attention: in particular, the subtle difference between shuffling and Poissong subsampling (which is currently mentioned in a footnote), as well as the privacy cost of hyperparameter tuning.  \n\nRegarding the reviewer’s comment on the generality of heuristics, we clarify that while they are developed based on regression analysis in two specific scenarios (fine-tuning GPT-2-S on Enron at $\\varepsilon=4$ with the target variable as ACR, and Llama-2-7b on TOFU at $\\varepsilon=16$ with the target variable as AIR), our experiments demonstrate that they generalize across various models, datasets, privacy budgets, and empirical privacy measures (VMR). The detailed results are presented in Figs. 7 and 10 in the main paper, as well as Figs. 18, 20, 21, and 22 in Appendix G. In total, we have spent over 10,000 H100 GPU hours to ensure that our findings are general and robust. We hope this convinces the reviewer about the broad applicability of these heuristics. Please let us know if you have further questions or suggestions. Thank you!", "claims_and_evidence": "The claims in the paper are supported by well-structured and clear evidence. I believe the experimental evidence in the main paper is sufficiently articulate and convincing. I did not review supplementary results in the Appendices in detail.", "methods_and_evaluation_criteria": "The chosen evaluation metrics, models and data sets are appropriate for answering the research question explored in the paper.", "theoretical_claims": "There are no theoretical proofs in the paper. The main claims are supported by empirical evidence.", "experimental_designs_or_analyses": "I checked the soundness and validity of all experiments presented in the main paper, which I find to be well-structured, relevant to the proposed question, and clearly presented.", "relation_to_broader_scientific_literature": "The paper proposed tools to examine the phenomenon of empirical privacy variance in DP-SGD, and analyses the case of private fine-tuning of language models. The paper is located in the landscape of DP-SGD literature. DP-SGD was first introduced by Abadi et al. (2016). The impact of hyperparameter configuration on model utility has been explored among others by Ponomareva et al. (2023). This paper focuses on the effects of hyperparameter choice on empirical privacy instead. The works most closely related to this paper are mentioned in Appendix A: Hayes et al. (2023) show how different hyperparameters impact the success rate of reconstruction attacks; Kaissis et al. (2024) propose a metric to quantify the maximum excess vulnerability among mechanisms that share the same privacy budget. These works are focused on theoretical analysis of such problems, while this paper is focused on a practical, real-world setting (privately fine-tuned language models) and proposes heuristics and intuitive hypothesis on potential causes of this phenomenon.", "essential_references_not_discussed": "I'm not aware of essential references that aren't cited. However, I think the short section comparing the paper to related work in Appendix A should be discussed in the main paper, with a dedicated and more articulated section. That would make it easier for the reader to locate this paper in the literature landscape. I am aware that the authors chose to present that in Appendix at this stage due to space constraints, and I encourage them to include a more extended presentation of the related work in future iterations.", "other_strengths_and_weaknesses": "The main strength of the paper is that it highlights an interesting and general no-free-lunch problem arising from hyperparameter setting selection in DP-SGD. To the best of my knowledge, this is the first time empirical privacy variance is proposed and investigated on a practical level. The experiment results provide compelling evidence that the problem is general, and likely to impact a variety of models using DP-SGD. For this reason, I do think that the implications of empirical privacy variance are potentially broad and of interest to the DP community. This paper poses the basis for interesting future exploration, while also providing practical heuristics to mitigate the problem. \n\nOne limitation of the paper is that empirical privacy might need to be measured differently depending on the specific task at hand, and it is unclear to which extent the proposed heuristics apply and generalize. This might be better clarified by including a limitations section in the next iteration of the paper.", "other_comments_or_suggestions": "I don't have additional comments.", "questions_for_authors": "I encourage the authors to address the minor points I brought up in the review, particularly the inclusion of a Limitations section and a more detailed Related Work section in the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pEtmQifjJA", "reviewer": "Reviewer_cBBr", "summary": "For a variety of memorization based privacy attacks on language modeling, this paper studies how the average attack success can vary across hyperparameters that are meant to give the same DP guarantees. This phenomenon is observed across model sizes, architectures, datasets, and DP guarantees. Several correlations are observed, leading to proposed heuristics for choosing hyperparameters which they show improve over previous design principles at mitigating the average attack success rate. Further evidence of varying privacy guarantees is shown by instantiating privacy auditing.\n\n## Update after Rebuttal\n\nThe authors performed experiments that showed the phenomenon was not a consequence of per-instance privacy variance, amongst answering my other concerns, and providing discussion I found very helpful. I raised my score to an accept given this.\n\nI have further raised my score to a 5 as I believe the other reviewers concerns have been addressed. I hope this reflect my belief this paper has been conducted with significant rigour and provides valuable insights/motivation for the field.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for initiating the interesting discussions. \n\nLet us begin by explaining 1) what we mean by “empirical privacy”, 2) our empirical privacy measures, and 3) the choice of computing the average score. \n\n1) As noted in the introduction, we take a practical perspective on empirical privacy through users’ perceptions of model behaviors. This is motivated by the gap between the increasing use of DP in LLMs, and the tangible privacy risks that arise in user interactions. We believe empirical privacy should hold richer meanings in this context, extending beyond what is usually referred to (privacy attacks).\n\n2) Our considered empirical privacy measures largely follow this standpoint. They are based on memorization and are specific to the generative nature of LLMs, rather than being generic such as the AUC of membership inference which can be applied to any ML model. \n\n3) Finally, we average the empirical privacy scores over a set of secrets because i) this is how these metrics are evaluated in the literature (e.g., the works that proposed ACR and VMR); ii) we aim to ensure that the conclusions we draw are robust. \n\nCircling back to the reviewer’s comments,\n\n**Average- vs. worst-case.** We thank the reviewer for the insightful comment. We agree that this adds another layer of subtlety which we overlooked. We thus re-examined whether switching from average- to worst-case measures would affect our conclusions. It turns out that they remain unchanged: empirical privacy variance persists; the regression study yields similar qualitative results; the correlation between empirical privacy scores and audited values remains low. \n\nThese results suggest what’s critical is not the distinction between average- and worst-case empirical privacy measures. Instead, the fundamental gap lies between **what DP promises** (preventing reidentification/membership inference) vs. **how we measure privacy** (whether a model generates a specific secret given a prompt). This gap is different from the one raised by the reviewer (per-instance guarantees vs. worst-case guarantees). \n\nFurthermore, we believe it is not appropriate to view our empirical privacy measures as representing the \"average of the per-instance guarantees\" or “expected leakage over datapoints” as: 1) we focus on a small subset of secrets that exhibit strong sign of memorization (Appendix E.5); and 2) a secret is only a small substring of an instance. \n\nWe will add the above discussions and the new results in the camera-ready.\n\n\n**Privacy auditing.** While we agree with the reviewer that the “nice controlled study” would be great to have, there are technical challenges that prevent this. 1) Empirical privacy measures on datapoints used for auditing: Auditing uses specially crafted canary samples; they are long and each member is inserted once in the training data. Applying empirical privacy measures on them will likely yield insignificant results. 2) Auditing on extracted secrets used for empirical privacy measures: The secrets are pieces within training samples but not whole samples; they could occur in more than one sample. These create mismatches with the requirement on data in auditing. Moreover, the size of the secret set is no more than 50, while auditing typically requires a much larger set. \n\nThat said, since the dataset used by the auditing method represents a worst-case dataset to maximize the lower bound the method can achieve, an alternative fair comparison is to use the worst-case empirical privacy measures, as suggested by the reviewer. Since we still observe a low correlation between empirical privacy and auditing scores, this suggests the choice of dataset is likely not an influencing factor.\n\nWe believe both hypotheses we made in the “Open questions” in Sec. 5.1, 1) The auditing method is not sufficiently powerful; 2) There is a fundamental gap between membership inference and memorization-based attacks (which we also bring up in the previous point), are likely; probably even the mix of both. \n\n\n**Q1 - one-to-many relationship.** $\\sigma$ is not chosen to be the minimal per-step noise but rather dependent on $b$ and $T$. It is computed using privacy accountants to satisfy a target $(\\varepsilon,\\delta)$-DP guarantee (Sec 2).\n\nFor Laplace or Gaussian mechanism with a fixed $\\delta$, the privacy budget $\\varepsilon$ uniquely determines the privacy level via the single hyperparameter that controls the noise scale. In contrast, the compositional nature of DP-SGD allows infinitely many configurations to achieve the same $\\varepsilon$, each yielding its own privacy level. We intend to highlight that empirical privacy variance is a unique characteristic of DP-SGD, and more broadly, of DP algorithms that involve composition.\n\n**Q2 - certification.** As illustrated in Fig 4, all models in the red region (with $\\varepsilon < \\varepsilon^\\star$) will fail to pass the privacy tests. Please refer to our last response to Reviewer u6Yf for more details.", "claims_and_evidence": "There seems to be a subtle difference in what empirical privacy means in this paper (an average over a set), and what privacy typically means in the literature (a bound on maximum leakage). I believe being more precise will aid in future reproducibility, helps explain inconsistencies with audit attacks, and also make the motivation for policy decisions more precise. I describe this in more detail below.\n\nPrivacy is typically defined and empirically measured as a worst-case over datapoints (see auditing literature), but “empirical privacy”  in this paper is for their average (across secrets) attack success rate. Admittedly the average is over a set of secrets (or author-attribute pairs for ToFU) that was filtered and so could have implicitly been pushed to worst-case points. But a more direct method would have been to report just the worst leakage over the secrets. Moreover, at least in the case of ToFU, the filtered set still represents a quarter of the original set.\n\nHence, it seems it is more accurate to say this paper is studying the average of the per-instance guarantees [1] over a relatively large subset of the training datapoints, which is known to be quite different from the typical DP guarantees when using DP-SGD [2]. I would recommend the authors make the difference in their use of empirical privacy (average over a set of datapoint) to the worst-case literature clearer, as to also be clearer about the claims. This difference may also explain why their empirical privacy did not correlate with privacy audit results: the privacy audits are testing themselves per-instance guarantees over a different set of datapoints (also using a different attack). This said, as argued by the authors choice of attacks and cited literature, the behaviour of leakage for such average case users may be relevant in designing systems to not be too bad in the worst-case (DP guarantees) but also much better on average (as measured by the average in this paper).\n\n[1] Wang, Yu-Xiang. \"Per-instance differential privacy.\" Journal of Privacy and Confidentiality 9.1 (2019).\n\n[2] Thudi, Anvith, et al. \"Gradients look alike: Sensitivity is often overestimated in {DP-SGD}.\" 33rd USENIX Security Symposium (USENIX Security 24). 2024.", "methods_and_evaluation_criteria": "The datasets used make sense for the evaluation of privacy leakage. The evaluation methods also make sense for studying average leakage over data points; privacy is typically defined as worst-case leakage over data points so the distinction with that could be made clearer (see claims section).", "theoretical_claims": "No theory (and hence proofs) are presented.", "experimental_designs_or_analyses": "I read Appendix E which described additional experimental details, alongside Appendix H.1 which described auditing details, and found no issues.", "relation_to_broader_scientific_literature": "Other notions of variance in privacy leakage have been observed, but these were in the context of leakage for individual data points (see a memorization study here [3]). This paper studies variation from hyperparameters for a fixed set of datapoints, and observed predictable behaviours over the hyperparameter choices. This seems novel in the context of DP to the best of my knowledge.\n\n[3] Carlini, Nicholas, et al. \"The privacy onion effect: Memorization is relative.\" Advances in Neural Information Processing Systems 35 (2022): 13263-13276.", "essential_references_not_discussed": "None that I noticed.", "other_strengths_and_weaknesses": "Strengths:\n1) Experimental study is novel and thorough\n2) The paper is mostly clear (see suggestion and previous comments)\n3) The idea that average leakage can vary predictably with hyperparameters may have consequence to future decisions in standardizing DP\n\nWeakness:\n1) General ambiguity in what is measured in this paper by “privacy” versus DP guarantees.", "other_comments_or_suggestions": "Below are suggestions to help clarify aspects of the paper. I look forward to hear what the authors think, and am happy to consider raising my score given their response.\n\n1) Consider adding discussion on the differences between worst-case DP analysis, and expected privacy leakage experienced by users studied in this paper. Ideally this would be before or at the beginning of the “empirical privacy measures” paragraph to make the notion of privacy studied (in the context of the literature) more well-defined. Terms like per-instance privacy express the leakage over an individual datapoint [1], and its known these values behave quite differently to typical worst-case DP guarantees [2], posing limitations in using per-instance studies to understand worst-case guarantees (see references earlier in the review). \n\n2) The above can also then be incorporated into the privacy standardization discussion in “why is this relevant”; I believe the argument being made in this paper is that standardization based only on worst-case leakage does not account for variation in expected leakage over datapoint, and this could now be made clearer/more precise in the text.\n\n3) On the comparison to privacy auditing, there are two (clear) changes to the previous memorization methodology. Both methods are evaluating per-instance guarantees on different sets of points, and also using different attacks. Maybe a nice control is to run the auditing attack (or the proposed memorization attacks in the paper) on the other set of datapoints to disentangle what caused the lack of correlation (is it the attack or the set of points?). This also would shed light on the hypothesis: I believe the experiments so far have suggested the empirical privacy variations were attack independent (Figure 2), so one would hope this is still true for the loss attack.", "questions_for_authors": "1) Could the authors elaborate on what the “one to many $\\varepsilon$-to-risk relationship” for composition is describing? I believe for a given $\\varepsilon$ and $\\delta$ you can assume $\\sigma$ (ideally) is chosen to be the minimal per-step noise? Unless you’re discussing changing $\\sigma$ per-step, so many sequences of $\\sigma$ could give the same final $\\varepsilon$?\n\n2) Could the authors also clarify what “$\\varepsilon$ cannot be used for certification” means in the “why is this relevant” paragraph? In particular I found the phrasing saying a model calibrated to $\\varepsilon$ cannot ensure compliance for another model confusing. I’m guessing this is a typo; does it make sense for a DP model to be used to certify another DP model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents comprehensive experiments exploring empirical privacy measures, such as secret memorization, in LLMs trained with different hyper-parameters to give the same formal privacy guarantees. They find that there is significant variation in empirical privacy across these models, identify trends in how the hyper-parameters relate to empirical privacy, and discuss possible causes.

All reviewers agreed that the experiments were rigorous and impressive, with various smaller questions about the findings or interpretation, most of which were well answered by the rebuttal.

Reviewer E9Wi raised concerns about novelty with respect to the observation that practical privacy according to some attack model could vary over models satisfying the same (epsilon, delta)-DP guarantee. It appears the most important precedent Hayes et al. 2023. However, the submission expands significantly by performing an expansive set of experiments with robust findings on a specific, important, application area (LLMs) with tailored privacy metrics based on memorization (and, for example, not assuming an informed adversary that knows all records except for one, as in Hayes et al.), and by identifying specific trends in the results. This reinforces the prior work and extends its findings to provide a broad empirical foundation for empirical vs formal privacy in LLMs that is likely to inform future research.

Reviewer u6Yf's primary concern is the discrepancy between analysis based on Poisson subsampling but an implementation based on shuffling. Apart from this point, the reviewer thinks the paper is "otherwise high quality" and would recommend acceptance. I appreciate the reviewer bringing up this point, since recent evidence (Chua et al., 2024) indicates it is important to address. However, the practice has been a norm in the field for almost a decade, so it will take some time to make the change. Further, both the reviewer and authors agreed it would have a very minor impact on most of the findings. I encourage the authors to address this more prominently in writing in the final version of the paper. 

Reviewer MX8d was largely satisfied with the work, but recommends a more prominent discussion of related work. I (and the authors, in their rebuttal) agree with this point.

Reviewer cBBr brought up a specific issue of worst-case vs. average privacy over instances. The authors ran additional experiments to address this issue, and the reviewer raised their score. They raised their score further because they believed the authors had addressed the concerns raised by other reviewers.

Overall, the paper appears to be a solid empirical contribution that can inform the discrepancy between theoretical and practical privacy in LLMs.
Decision: Accept (poster)