SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This definition specifies the trade-off function  $\bigotimes_{\pmb{x} \sim \mathcal{D}} h(\pmb{x}) : [0,1] \to [0,1]$  of a stochastic composition of several trade-offs. While it is reminiscent of the "most powerful test" (MPT) [28], there are several differences to the MPT that are important in our work. Most prominently, a straightforward construction of the MPT to MI problems does not work since the adversary does not only run one hypothesis test to guess whether one sample belongs to the training data set or not; instead, the adversary draws multiple samples and runs sample-dependent and (potentially) different hypotheses tests for each drawn sample. This is necessary due to the form of the alternative hypotheses in the formulation of the test in (3), which depends on the sample  $\pmb{x}'$ . We therefore require a tool to compose the results from different hypothesis tests. Finally, we prove that the trade-off of the stochastic composition has the properties of a trade-off function (see App. D.1):
Theorem 4.1 (Stochastic composition of trade-off functions). The stochastic composition  $\bigotimes_{\pmb{x} \sim \mathcal{D}} h(\pmb{x})$  of trade-off functions  $h(\pmb{x})$  maintains the characteristics of a trade-off function, i.e., it is convex, non-increasing,  $(\bigotimes_{\pmb{x} \sim \mathcal{D}} h(\pmb{x}))$  ( $r) \leq 1 - r$  for all  $r \in [0,1]$ , and it is continuous at zero.
4.2  $f$ -Membership Inference Privacy ( $f$ -MIP)
This rigorous definition of the stochastic composition operator allows us to define membership inference privacy from a hypothesis testing perspective.
Definition 4.2 ( $f$ -Membership Inference Privacy). Let  $f$  be a trade-off function. An algorithm  $A: D^n \to \mathbb{R}^d$  is said to be  $f$ -membership inference private ( $f$ -MIP) with respect to a data distribution  $\mathcal{D}$  if
$$
\bigotimes_ {\boldsymbol {x} ^ {\prime} \sim \mathcal {D}} \operatorname {T e s t} \left(A _ {0}; A _ {1} \left(\boldsymbol {x} ^ {\prime}\right)\right) \geq f, \tag {7}
$$
where  $\mathbf{x}' \sim \mathcal{D}$  and  $\otimes$  denotes the stochastic composition built from individual trade-off functions of the MI hypotheses tests for random draws of  $\mathbf{x}'$ .
In this definition, both sides are functions dependent on the false positive rate  $\alpha$ . A prominent special case of a trade-off function is the Gaussian trade-off, which stems from testing one-dimensional normal distributions of unit variance that are spaced apart by  $\mu \in \mathbb{R}_{\geq 0}$ . Therefore, defining the following special case of  $f$ -MIP will be useful.
Definition 4.3 ( $\mu$ -Gaussian Membership Inference Privacy). Let  $\Phi$  be the cumulative distribution function (CDF) of a standard normal distribution. Define  $g_{\mu}(\alpha) \coloneqq \Phi(\Phi^{-1}(1 - \alpha) - \mu)$  to be the trade-off function derived from testing two Gaussians; one with mean 0 and one with mean  $\mu$ . An algorithm  $A$  is  $\mu$ -Gaussian Membership Inference private ( $\mu$ -GMIP) with privacy parameter  $\mu$  if it is  $g_{\mu}$ -MIP, i.e., it is MI private with trade-off function  $g_{\mu}$ .
Remark 4.1. DP can also be defined via the Gaussian trade-off function, which results in  $\mu$ -Gaussian Differential Privacy ( $\mu$ -GDP, [11]). While the trade-off curves for both  $\mu$ -GDP and  $\mu$ -GMIP have the same parametric form, they have different interpretations:  $\mu$ -GDP describes the trade-off function that an attacker with complete knowledge (left column in Table 1) could achieve while  $\mu$ -GMIP describes the trade-off function that an attacker with MI attack capability can achieve (right column in Table 1). In the next section we will quantify their connection further.
<sup>1</sup>When using the term "algorithm", we also include randomized mappings.
4.3 Relating  $f$ -MIP and  $f$ -DP
We close this section by providing first results regarding the relation between  $f$ -DP and  $f$ -MIP. As expected,  $f$ -DP is strictly stronger than  $f$ -MIP, which can be condensed in the following result:
Theorem 4.2 (f-DP implies  $f$ -MIP). Let an algorithm  $A: D^n \to \mathbb{R}^d$  be  $f$ -differentially private [11]. Then, algorithm  $A$  will also be  $f$ -membership inference private.
We proof this result in Appendix F.1. This theorem suggests one intuitive, simple and yet actionable approach to guarantee Membership Inference Privacy. This approach involves the use of DP learning algorithms such as DP-SGD [1], which train models using noised gradients. However, as we will see in the next section, using noise levels to guarantee  $f$ -DP is usually suboptimal to guarantee  $f$ -MIP.
5 Implementing  $f$ -MIP through Noisy SGD
We would now like to obtain a practical learning algorithm that comes with  $f$ -MIP guarantees. As the dependency between the final model parameters and the input data is usually hard to characterize, we follow the common approach and trace the information flow from the data to the model parameters through the training process of stochastic gradient descent [1, 33]. Since the gradient updates are the only path where information flows from the data into the model, it suffices to privatize this step.
5.1  $f$ -MIP for One Step of Noisy SGD
We start by considering a single SGD step. Following prior work [1, 33], we make the standard assumption that only the mean over the individual gradients  $\boldsymbol{m} = \frac{1}{n}\sum_{i=1}^{n}\theta_{i}$  is used to update the model (or is published directly) where  $\boldsymbol{\theta}_{i} \in \mathbb{R}^{d}$  is a sample gradient. Consistent with the definition of the membership inference game, the attacker tries to predict whether a specific gradient  $\boldsymbol{\theta}'$  was part of the set  $\{\boldsymbol{\theta}_{i}\}_{i}$  that was used to compute the model's mean gradient  $\boldsymbol{m}$  or not. We are interested in determining the shape of the attacker's trade-off function. For the sake of conciseness, we directly consider one step of noisy SGD (i.e., one averaging operation with additional noising, see Algorithm 2 from the Appendix), which subsumes a result for the case without noise by setting  $\tau^2 = 0$ . We establish the following theorem using the Central Limit Theorem (CLT) for means of adequately large batches of  $n$  sample gradients, which is proven in Appendix E.
Theorem 5.1 (One-step noisy SGD is  $f$ -membership inference private). Denote the cumulative distribution function of the non-central chi-squared distribution with  $d$  degrees of freedom and noncentrality parameter  $\gamma$  by  $F_{\chi_d^2 (\gamma)}$ . Let the gradients  $\pmb{\theta}^{\prime}\in \mathbb{R}^{d}$  of the test points follow a distribution with mean  $\mu$  and covariance  $\pmb{\Sigma}$ , let  $K\geq \| \pmb{\Sigma}^{-1 / 2}\pmb{\theta}^{\prime}\|_{2}^{2}$  and define  $n_{\text{effective}} = n + \frac{\tau^2n^2}{C^2}$ . For sufficiently large batch sizes  $n$ , one step of noisy SGD is  $f$ -membership inference private with trade-off function given by:
$$
\beta (\alpha) \approx 1 - F _ {\chi_ {d} ^ {2} \left(\left(n _ {\text {e f f e c t i v e}} - 1\right) K\right)} \left(\frac {n _ {\text {e f f e c t i v e}}}{n _ {\text {e f f e c t i v e}} - 1} F _ {\chi_ {d} ^ {2} \left(n _ {\text {e f f e c t i v e}} K\right)} ^ {- 1} (\alpha)\right). \tag {8}
$$
The larger the number of parameters  $d$  and the batch size  $n$  grow, the more the trade-off curve approaches the  $\mu$ -GMIP curve, which we show next (see Figure 1).
Corollary 5.1 (One step noisy SGD is approx.  $\mu$ -GMIP). For large  $d, n$ , noisy SGD is approximately  $g_{\mu_{Step}}$ -GMIP. In particular,  $\beta(\alpha) \approx \Phi(\Phi^{-1}(1 - \alpha) - \mu_{Step})$  with privacy parameter:
$$
\mu_ {\text {s t e p}} = \frac {d + (2 n _ {\text {e f f e c t i v e}} - 1) K}{n _ {\text {e f f e c t i v e}} \sqrt {2 d + 4 n _ {\text {e f f e c t i v e}}} K}. \tag {9}
$$
This result is striking in its generality as it also covers models trained without additional noise or gradient cropping ( $n_{\mathrm{effective}} = n$  in that case). Unlike for DP, even standard models trained with
Figure 1: Trade-off function convergence. The trade-off function from Theorem 5.1 converges to the one from Corollary 5.1 where  $\tau^2 = 0$  and  $K = d$ .
non-noisy SGD offer an elementary level of MIP. Our result further explicitly quantifies four factors
that lead to attack success: the batch size  $n$ , the number of parameters  $d$ , the strength of the noise  $\tau^2$  and the worst-case data-dependent gradient susceptibility  $\| \Sigma^{-1/2} \theta' \|_2^2$ . The closeness of the trade-off function to the diagonal, which is equivalent to the attacker randomly guessing whether a gradient  $\theta'$  was part of the training data or not, is majorly determined by the ratio of  $d$  to  $n$ . The higher the value of  $d$  relative to  $n$ , the easier it becomes for the attacker to identify training data points. Furthermore, a higher gradient susceptibility  $K$ , which measures the atypicality of a gradient with respect to the gradient distribution, increases the likelihood of MI attacks succeeding in identifying training data membership. It is worth noting that if we do not restrict the gradient distribution or its support, then there might always exist gradient samples that significantly distort the mean, revealing their membership in the training dataset. This phenomenon is akin to the  $\delta$  parameter in DP, which also allows exceptions for highly improbable events.
Remark 5.1 (Magnitude of  $\mu_{\mathrm{Step}}$ ). When the dimensions of the uncorrelated components in  $\Sigma^{-\frac{1}{2}}\theta'$  are also independent, we expect  $K$  to follow a  $\chi^2$ -distribution with  $d$  degrees of freedom and thus  $K \in \mathcal{O}(d)$ . In the standard SGD-regime ( $\tau^2 = 0$ ) with  $d, n \gg 1$ , we obtain  $\mu \in \mathcal{O}\big(\sqrt{d / n}\big)$ .
Remark 5.2 (On Optimality). The dependency on  $d$  when  $\tau^2 > 0$  is a consequence of our intentionally broad proving strategy. Our proof approach consists of two key steps: First, we establish the optimal LRT under general gradient distributions, without adding noise or imposing any cropping constraints (See Appendix E.1). This initial step serves as the foundation for our subsequent analysis and is (1) as general as possible covering all distributions with finite variance and is (2) optimal in the sense of the Neyman-Pearson Lemma, i.e., it cannot be improved. This means that our result covers all models trained with standard SGD ( $\tau^2 = 0$  and  $C = \infty$ ) and is remarkable in its generality as it is the first to suggest clear conditions when adding noise is not required to reach f-MIP. Second, we specialize our findings to cropped random variables with added noise (See Appendix E.2). This analysis could potentially be improved by considering individual gradient dimensions independently.
5.2 Composition and Subsampling
In the previous section, we have derived the trade-off function for a single step of SGD. Since SGD is run over multiple rounds, we require an understanding of how the individual trade-off functions can be composed when a sequence of  $f$ -MIP operations is conducted, and a random subset of the entire data distribution is used as an input for the privatized algorithm. The next lemma provides such a result for  $\mu$ -GMIP and follows from a result that holds for hypotheses tests between Gaussian random variables due to Dong et al. [11] (see Appendix D.3 for details and more results).
Lemma 5.1 (Asymptotic convergence of infinite DP-SGD). Let  $n$  be the batch size in SGD, and  $N$  be the entire size of the dataset. If a single SGD-Step is at least as hard as  $\mu_{step}$ -GMIP with respect to the samples that were part of the batch and  $\frac{n\sqrt{t}}{N} \to c$  as  $\lim_{t \to \infty} (\text{the batch size is gradually decreased})$ , then the noisy SGD algorithm will be  $\mu$ -GMIP with
$$
\mu = \sqrt {2} c \sqrt {\exp \left(\mu_ {\text {s t e p}} ^ {2}\right) \Phi (1 . 5 \mu_ {\text {s t e p}}) + 3 \Phi (- 0 . 5 \mu_ {\text {s t e p}}) - 2}. \tag {10}
$$
Note that this result also provides a (loose) bound for the case where exactly  $T$  iterations are run with a batch size of  $n'$  with  $c = \frac{n' \sqrt{T}}{N}$  (through using  $n(t) = n'$  if  $t \leq T$ , else  $n(t) = \frac{n' \sqrt{T}}{\sqrt{t}}$ ). With this result in place, we can defend against MI attacks using the standard noisy SGD algorithm.
Datasets and Models. We use three datasets that were previously used in works on privacy risks of ML models [32]: The CIFAR-10 dataset which consists of 60k small images [21], the Purchase tabular classification dataset [25] and the Adult income classification dataset from the UCI machine learning repository [12]. Following prior work by Abadi et al. [1], we use a model pretrained on CIFAR-100 and finetune the last layer on CIFAR-10 using a ResNet-56 model for this task [16] where the number of fine-tuned parameters equals  $d = 650$ . We follow a similar strategy on the Purchase dataset, where we use a three-layer neural network. For finetuning, we use the 20 most common classes and  $d = 2580$  parameters while the model is pretrained on 80 classes. On the adult dataset, we use a two-layer network with 512 random features in the first layer trained from scratch on the
dataset such that  $d = 1026$ . We refer to Appendix C.1 for additional training details. We release our code online.<sup>2</sup>
6.1 Gradient Attacks Based on the Analytical LRT
To confirm our theoretical analysis for one step of SGD and its composition, we implement the gradient attack based on the likelihood ratio test derived in the proof of Theorem 5.1. We provide a sketch of the implementation in Algorithm 1 and additional details in Appendix C.3. An essential requirement in the construction of the empirical test is the estimation of the true gradient mean  $\mu$  and the true inverse covariance matrix  $\Sigma^{-1}$  since these quantities are essential parts of both the test statistic  $\hat{S}$  and the true gradient susceptibility term  $\hat{K}$  needed for the analytical attack. The attacker uses
Algorithm 1: Gradient Likelihood Ratio (GLiR) Attack
Require: Training data distribution D, batch size n, number of pa
rameters d, query point x ∈ D, averaged gradients of each batch
mt∈Rd for training steps t=1,...,T, parameter gradient com-
putation function ∇wtL:D→Rd of training, threshold η
1: ptotal←0
2: for t=1,...,T do
3: B={b1,...,bm} ∼Dm	▷ Sample background data
4: gi=∇wL(bi), i=1...m	▷ Compute background gradients
5: Σ = Cov{g1,...,gm} ∈ Rd×d	▷Approximate covariance Σ
6: μ = Mean{g1,...,gm} ∈ Rd	▷Approximate mean μ
7: θ = ∇wtL(x)	▷Compute gradients for the query point
8: S=(n-1)(mt-θ)TˆΣ-1(mt-θ)	▷Compute test statistic
9: K=||ˆΣ-1/2(θ-ˆμ)||2	▷Estimate gradient susceptibility
10: pstep = logF-1χ2(nK)(S)	▷Compute logp-value under H0
11: ptotal←ptotal + pstep
12: end for
13: return Train if ptotal<η, else Test
their access to the gradient distribution (which is standard for membership inference attacks [5, 29] and realistic in federated learning scenarios [20]), to estimate the distribution parameters. In practice, however, the empirical estimates of  $\hat{\pmb{\mu}}$ ,  $\hat{\Sigma}^{-1}$  and thus  $\hat{K}$  will be noisy and therefore we do not expect that the empirical trade-off curves match the analytical curves exactly.
Using our novel Gradient Likelihood Ratio (GLiR) attack we can audit our derived guarantees and their utility. First, we audit our one-step guarantees from Theorem 5.1. To compare the models, we adapt the batch size  $n$  such that all models reach the same level of  $\mu$ -GMIP. In Figure 2a, we use a simulated gradient distribution with known parameters  $\pmb{\mu}, \pmb{\Sigma}^{-1}$  and  $d$ . In this case, we can estimate  $K$  accurately and observe that our bounds are tight when the distribution parameters and thus the respective gradient susceptibilities can be computed accurately. We provide additional ablation studies that gauge the approximation quality of with small values for  $d, n$  and different simulated distributions in Appendix C.2. When the parameters are unknown and we have to estimate the parameters, our attacks become weaker and do not match the analytical prediction (see Figure 2b).
We also audit our composition guarantees. We do five SGD-steps in Figure 2c. While there is a small gain in attack performance on the CIFAR-10 dataset (e.g., at FPR=0.25), the attack performance on the other datasets remains largely unaffected. This mismatch occurs since the theoretical analysis is based on the premise that the attacker gains access to independently sampled gradient means for each step to separate training and non-training points, but in practice we do not gain much new information as the model updates are not statistically independent and too incremental to change the gradient means significantly between two subsequent steps. Therefore, a practical attacker does not gain much additional information through performing several steps instead of one. Future work is required to model these dependencies and potentially arrive at a tighter composition result under incremental parameter updates. We provide results for additional existing membership inference attacks, for instance the recent loss-based likelihood-ratio attack by Carlini et al. [5] in Appendix C.4, which all show weaker success rates than the gradient-based attack that proved most powerful in our setting.
6.2 Comparing Model Utility under  $\mu$ -GDP and  $\mu$ -GMIP
Here we compare the utility under our privacy notion to the utility under differential privacy. We sample 20 different privacy levels ranging from  $\mu \in [0.4,\dots,50]$  and calibrate the noise in the SGD iteration to reach the desired value of  $\mu$ . We can do so both for  $\mu$ -GMIP using the result in Equation (10) and using the result by Dong et al. [11, Corollary 4] for  $\mu$ -GDP, which result in the same
$^{2}$ https://github.com/tleemann/gaussian_mip
(a) CIFAR-10
Figure 3: Utility of DP versus MIP. Model performance on three datasets across different privacy levels  $\mu$  (small  $\mu$  denotes high privacy) using the notions of  $\mu$ -Gaussian Differential Privacy (parametric form of  $f$ -DP, [11]) and  $\mu$ -Gaussian Membership Inference Privacy (parametric form of  $f$ -MIP, ours) on three datasets. GMIP usually allows for substantially increased accuracy over the corresponding GDP guarantee with the same attack success rates controlled by  $\mu$ . However, the attacker under GMIP runs membership inference (MI) attacks while GDP allows for a wider set of privacy threat models. For more details on differences in the underlying threat models see Table 1.
(b) Purchase
Figure 2: Auditing  $f$ -MIP with our gradient attack (GLiR) when  $\tau^2 = 0$ . We show trade-off curves when the gradient distribution is known (a) and when the gradients are obtained from a trained model that was finetuned on various data sets (b, c). The analytical solutions are computed with a value of  $K = d$  and using the composition result for  $k$  steps in Appendix D.3 for (c).
(c) Adult
attack success rates while  $\mu$ -GDP allows for stronger privacy threat models. Due to Theorem 4.2, we never need to add more noise for  $\mu$ -GMIP than for  $\mu$ -DP. Further details are provided in Appendix C.1. Figure 3 shows a comparison of the accuracy that the models obtain. We observe that the model under GMIP results in significantly higher accuracy for most values of  $\mu$ . As  $\mu \to 0$  both privacy notions require excessive amounts of noise such that the utility decreases towards the random guessing accuracy. On the other hand, for higher values of  $\mu$ , there is no need to add any noise to the gradient to obtain  $\mu$ -GMIP, allowing to obtain the full utility of the unconstrained model. This indicates that useful GMIP-bounds do not necessarily require noise. For instance, on the CIFAR-10 model, no noise is required for  $\mu \geq 0.86$  which is a reasonable privacy level [11]. Overall, these results highlight that useful and interpretable privacy guarantees can often be obtained without sacrificing utility.
In the present work, we derived the general notion of  $f$ -Membership Inference Privacy ( $f$ -MIP) by taking a hypothesis testing perspective on membership inference attacks. We then studied the noisy SGD algorithm as a model-agnostic tool to implement  $f$ -Membership Inference Privacy, while maintaining Differential Privacy (DP) as a worst-case guarantee. Our analysis revealed that significantly less noise may be required to obtain  $f$ -MIP compared to DP resulting in increased utility. Future work is required to better model the dependencies when composing subsequent SGD steps which could lead to improved bounds in practice. Furthermore, our analysis shows that when the capacity of the attacker is further restricted, e.g., to API access of predictions, there remains a gap between our theoretical bounds and loss-based membership inference attacks that can be implemented for real models. More work is required to either produce more sophisticated attacks or derive theoretical bounds for even less powerful attackers to close this gap.
[35] Tan, J., Mason, B., Javadi, H., and Baraniuk, R. Parameters or privacy: A provable tradeoff between overparameterization and membership inference. Advances in Neural Information Processing Systems (NeurIPS), 35:17488-17500, 2022. 3, 14
[36] Tan, J., LeJeune, D., Mason, B., Javadi, H., and Baraniuk, R. G. A blessing of dimensionality in membership inference through regularization. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 10968-10993. PMLR, 2023. 3, 14
[37] Thudi, A., Shumailov, I., Boenisch, F., and Papernot, N. Bounding membership inference. arXiv preprint arXiv:2202.12232, 2022. 14
[38] Triastcyn, A. and Faltings, B. Bayesian differential privacy for machine learning. In International Conference on Machine Learning (ICML), pp. 9583-9592. PMLR, 2020. 2, 3
[39] Wasserman, L. and Zhou, S. A statistical framework for differential privacy. Journal of the American Statistical Association, 105(489):375-389, 2010. 4
[40] Ye, J., Maddi, A., Murakonda, S. K., Bindschaedler, V., and Shokri, R. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2022. 3, 4, 14
[41] Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268-282. IEEE, 2018. 2, 3, 4, 14, 15
[42] Zanella-Béguelin, S., Wutschitz, L., Tople, S., Salem, A., Rühle, V., Paverd, A., Naseri, M., Köpf, B., and Jones, D. Bayesian estimation of differential privacy. In International Conference on Machine Learning (ICML). PMLR, 2023. 14