Do Parameters Reveal More than Loss for Membership Inference?
Authors: Anshuman Suri
Xiao Zhang
David Evans
Venue: Accepted by TMLR ()

Abstract: Membership inference attacks aim to infer whether an individual record was used to train a model. They are used as a key tool for disclosure auditing. While such evaluations are useful to demonstrate risk, they are computationally expensive and often make strong assumptions about potential adversaries' access to models and training environments, and thus do not provide very tight bounds on leakage from potential attacks. We show how prior claims around black-box access being sufficient for optimal membership inference do not hold for stochastic gradient descent, and that optimal membership inference indeed requires white-box access.
Our theoretical results suggest a new white-box inference attack IHA (Inverse Hessian Attack) that explicitly uses model parameters by taking advantage of computing inverse-Hessian vector products. Our results show that both auditors and adversaries may be able to benefit from access to model parameters, and we advocate for further research into white-box methods for membership privacy auditing.

Published in Transactions on Machine Learning Research (12/2024)
Do Parameters Reveal More than Loss for Membership Inference?
Anshuman Suri
asOru@virginia.cc edu
University of Virginia
Xiao Zhang CISPA Helmholtz Center for Information Security
zico.zhang@rcispu. de
David Evans University of Virginia
enons@Avirginia.co edu
Reviewed on OpenReview: https: // openreview. form?? id- fmkJJ6GKKC
Abstract
Membership inference allacks are used as a key tool for disclosure auditing. They aim to infer whether an individual record was used train model. While such evaluations are useful to demonstrate risk, they are computationally expensive and often make strong assumptions about potential adversaries' access to models and training environments, and thus do not provide tight bounds leakage from potential attacks. We show how prior claims around black-box access being sufficient for optimal membership inference do not hold for stochastic gradient descent, and that optimal membership inference indoed requires white-box access. Our theoretical results lead to a new white-box inference attack, IHA (Inverse Hessian Attack), that explicitly uses model paramcters by taking advantage of computing inverso- Hessian vector products. Our results show that both auditors and adversaries may be able to benefit from access to model parameters, and we advocate for further research into white- box methods for membership inference.
1 Introduction
Models produced by using machine learning on private training data can leak sonsitive information about data used to train or lune the model (Salem et al., 2023). Researchers study these privacy risks by either designing and evaluating attacks simulate what motivated adversaries may be able infer in particular settings by developing privacy methods that provide strong guarantees, often based on some notion of differential privacy (Dwork et al., 2006), that bounds information disclosure from any attack. Although both developing attacks and formal privacy proofs are important, conducting meaningful privacy audits is different from both approaches. Empirical methods, usually in the form of attack simulations, are inherently limited by the attacks considered and the uncertainty about the possibility of better attacks, while theoretical proofs require many assumptions or result in loose bounds. Further, any claims based on theoretical results depend on careful analysis that the system as implemented consistent with the thcory. If there is a thcoretical result that prescribes an optimal attack, then empirical results with that attack (or approximations of the attack) offer more meaningful estimate of privacy risk than is possible with theory experiments alone. While the theory needs to cover all data distributions, experiments with an optimal attack focus on the actual distribution and given model, resulting in tighter and more relevant privacy evaluations.
Privacy audits can also be important in adversarial contexts, where a regulator or external advocate conducts them test released model. Auditors with elevated model access (such as associated training environments or data) may be able to take advantage of more information produce better estimates of what an adversary could do without that information. Auditing is orthogonal to proofs that establish differential privacy

---

Published in Transactions on Machine Learning Research (12/2024)
bounds other privacy notions. As outlined by Cummings et al. (2024), theoretical bounds may be "too conservative inaccurate in some set settings", and it may not always be possible to come up with proofs or theoretical bounds that ensure models do not "violate disclosure requirements in ways that are not captured by differential privacy". Empirical auditing can provide a more meaningful mcasure of privacy leakage for these siluations.
The most common disclosure auditing approach today is conduct membership inference attacks (Kumar & Shokri, 2020) and related attacks that attempt to extract specific data (Cummings et al., 2024). While membership inference assumes the adversary already knows the full candidate record, it may still constitute direct privacy risk when revealing the inclusion of a known record in the training data itsolf, which leaks sensitive information. In most scenarios, however, membership disclosure by itself is not a serious privacy risk, but rather used as a proxy for understanding information leakage that may result in more serious privacy violations. Membership inference simple to define, relatively easy measure, and aligns well with differential privacy. This has resulted in it being widely used as a method for auditing disclosure risks for machine learning (Kunar & Shokri, 2020; Yeom et al., 2020; Kazmi et al., 2024; Azize & Basu, 2024).
Prior results on membership inference attacks have largely focused on the black-box setting, where the access to the target model. This focus has been reinforced by folklore and
attacker only has results demonstrating negligible gains from parameter access (known as while-bor attacks) (Nasr et al., 2018; Carlini et al., 2022). A well-known theoretical result by Sablayrolles et al. (2019) proves that black- box access is sufficient for optimal membership inference under certain conditions. This result has been the basis of several subsequent works (Ye et al., 2022; Chaudhari et al., 2024). However, the assumptions made in its derivation do not hold for most models, including ones trained with stochastic gradient descent (SGD). This theoretical result has detered researchers from exploring more the potential for inference methods that utilize parameter access, even though the theoretical result does not apply to common machine learning settings.
Contributions. In this work, we revisit previous assumptions surrounding the optimality of membership inference allacks. Utilizing recent advances in discrele-time SGD-dynamics (Liu al., 2021; Ziyin el al., 2021), we provide a more accurate formulation of the optimal membership inference attack that demonstrates the limitations of the rosults from Sablayrolles et al. (2019). In particular, we show that the claim that black- box access is sufficient does not hold for models trained using SGD (Section 3). Our theoretical result also prescribes an attack that exploits white-box access for auditing membership leakage, which we call the Inverse Hessian Attack (IHA) (Section 3.3). We empirically demonstrate its effectiveness in simple settings, showing that it outperforms state-of-tho-art reference- soweoncco-mdll-bbsee and prior white- thite--ox attacks (Section 4). Our analyses suggest that the improved auditing performance can be directly attributed to access to the model's parameters.
2
Preliminaries
The section provides background on membership inference (Section 2.1) and SGD dynamics (Section 2.2).
2.1 Membership Inference
Following the framework established by Sablayrolles et al. (2019), let D be a data distribution from which n records æ1, #2, .2.,........ , En are i.id. sampled with 2; = (x, y:) being the i-th record. Let w E Rd be the model parameters produced by some machine learning algorithm on a training dataset D. Assume mi, m2,..., me follow a Bernoulli distribution with y = Pmmm 1), where m, is the membership indicator of z; (i.e., m; 1 if 2; E D. and m; = 0 otherwise). Given w, a membership inference attack aims to predict the unknown membership m, for any given record z,.
Definition 2.1 (Membership Inference). Let w be the parameters of the target model and z be record. Inferring the membership of ø1 to the training set of w is equivalent to computing:
M(w, 21) P(m1 |
w. 21).
2

---

Published in Transactions on Machine Learning Research (12/2024)
Let P(w| 21,...: Zn: m......... m,) be the posterior distribution of model parameters produced by some ran- domized machine learning algorithm (i.e., stochastic gradient descent). Applying Bayes' theorem, Sablay- rolles et al. (2019) derived the following explicit formula for M(w, z1).
Lemma 2.1 (Sablayrolles et al. (2019)). Let T {=2,.... Zn, m2,,....... m.n) Given model parameters w and a record 21, the optimal membership inference is given by:
( In p(w my = 21, T)
2
M(w, =1) = TE σ
+ In
(1)
p(w m1 0, 21,
where o(u) = (1 exp(-u))- is the Sigmoid function, and 7 - P(m1= = 1).
To use Lemma 2.1, needs to characterize the posterior, P(w 21,..., 2n, m.:..., m,), to make explicit the elfect of the inferred record 21 on the optimal membership inference M(w, z1). Recent advances in discrete-time SGD dynamics (Liu et al., 2021; Ziyin et al., 2021) literature can help provide a connection between the posterior and model parameters.
2.2 Discrete-time SGD Dynamics
A line of theoretical work (Welling & Teh,, 2011; Sato & Nakagawa, 2014; Stephan et al., 2017: Liu et al., 2021; Ziyin et al., 2021) has analyzed the continuous- and discrete-time dynamics of stochastic gradient methods and provided insights for understanding deep learning generalization. Tet (wo = [(w) + g || [ be the lz-regularized total loss that we aim to optimize, where 2 denotes the hyperparameter that controls the regularization strength. Consider an SGD algorithm with the following update rule (for t 1.
g: V (w)-1) + 7h-15
h ph:-1 1 + gti
(2)
Wt = wi-1 Ah2-
Here, P E [0, 1) is the momentum, X 0 is the learning rate, and
(Ve(w,,)) 2;) + aaw)) - V Vioo SEB
Σ
Ve(we, 2:) - VL(w,)
m = S
S REB
represents the unbiased mini-bateh noise, where B, is randomly sampled batch of examples with size S from the training dataset D, and L(w)= = ExED E(w, 2).
Assuming a model is trained using SGD according to the update rule defined by Equation 2 on a guadralic loss and arrives at a stationary state, Liu et al. (2021) established a theoretical connection between the Hessian matrix H, the asymptotic noise covariance C= lim,++o Ew, (cov(n,, m)), and the asymptotic model fluctuation Σ = litty+000 cov(w, w.)) We next lay out the two imposed assumptions.
Assumption (Quadratic Loss). The total loss function (w) is either globally quadratic or locally quadratic close to a local minimum w" Specifically, the loss function can be approximated as:
1 - - w²) (H(("") + ala - w²) + o(llw w 1l1)),.
(w) = (wo)
(3)
where w" is a local minimum, H(w*) denotes the Hessian matrix at w with respect to the unregularized loss function L(w), and I, stands for the d x d identity matrix.
Assumption 2 (Stationary-Stare State).. After a sufficient number of iterations, models trained with SGD defined by Equation 2 arrive at a stationary state, i.c., the asymptotic model fluctuation Σ exists and is finite.
Under the above assumptions, Liu et al. (2021) proved the following theorem that describes model fluctuations of discrete SGD in quadratic potential with connections to the Hessian matrix and the noise covariance: Thcorem 2.2 (SGD Stationary distribution with momentum). Let w be updated with SGD defined by the update rule in Equation 2 with momentum μ E [0, 1). Under Assumptions 1 and 2, we additionally suppose commutes with H(w"), then the asymptotic model fluctuation satisfies
X XHH(W") + alI).
A(H(w)) ) aal))
)1
XCC
Σ =
21
1+ p.
1 + p
1-
3

---

Published in Transactions on Machine Learning Research (12/2024)
Theorem 2.2 requires the existence of finite stationary noise covariance and that the loss function is quadratic close to local minimum, which are both mild assumptions (see Liu et al. (2021) for detailed discussions).
In follow-up work, Ziyin et al. (2021) further derived the explicit dependence of the finite stationary noise covariance C on the loss and Hessian around a local minimum u² under certain assumptions:
Theorem 2.3 (SGD Noise Covariance). Let [wor(w) = I(w)) 2 2| be the total loss with a 2 0. Assume the model w optimized with SGD defined by Equation around a local minimum w". If L(w") + 0, then
2L(ww)) S
a² w w T + + + w w²112) o(T((w))),
C
H(ww))
s
provided that E is proportional to S-1 and [[ww) - f(w, 2;) small (i.e., of order o(L(w))).
The first imposed assumption of Σ = 0(S-1) has been justified by prior works (Liu et al., 2021; Xie et al., 2021; Mori al., 2022); the second assumption assumes that the current total training loss L(w) approximates well the individual loss for each record ((w, 21). Also, note that Theorem 2.3 directly implies that the SGD noise covariance C commutes with the Hessian matrix H(w").
Based on the above two theorems and only considering the leading term in the noise covariance, we can immodiately derive the following formula for the stationary model fluctuation of SGD:
5(1 - x) (2 ((w")) w==w"") ) (((w)) + +als) "(3 21 1+
(H(w") + +Ila)
(4)
We remark that if L(w=) = 0 (i.e., w" is a global minimum), then Σ = 0. In addition, if the Hessian matrix (H(w") + ala) has degenerate rank r < d, then (H(w") + aI,) can be replaced by the corresponding Moore-Penrose pseudo inverse. Accordingly, similar results to Equation 4 can be obtained by considering the projection space spanned by cigenvoctors with non-zero cigenvalues. Soction 5 of Ziyin et al. (2021) provides more detailed discussions of the imposed assumptions and the implications of the results.
3 Black-Box Access is not Sufficient
In this section, we examine previous assertions concerning optimal membership inference (Section 3.1) and show, for models trained with SGD, that optimal membership inference requires parameter access (Sec- tion 3.2). Our theory directly implies an attack (Section 3.3).
3.1 Limitations of Claims of Black- Box Optimality
Sablayrolles et al. (2019) proved the optimality of black-box membership inference under a Bayesian frame- work. They assume (Equation 1 in Sablayrolles et al. (2019)) that the postcrior distribution of model parameters w trained on 21,..., &n with membership my,..., mn follows
P(w , Zn) x exp (-
Σ m; E(w, E(wo,2)) 2.))
-
where T' is temperature parameter that captures the stochasticity of the learning algorithm. This as- sumption makes subsequent derivations of optimal membership inference much easier, but oversimplifies the training dynamics of typical machine learning algorithms such as SGD, Equation 5 assumes that the posterior of w follows Boltzmann distribution that only depends on the training loss. This is desirable for Bayesian posterior inference, where the goal is to provide a sampling strategy for an unknown data distribution given a set of observed data samples. This can be achieved using stochastic gradient Langevin dynamics (SGLD) (Welling & Teh, 2011) with shrinking step size ), (i.c., lim X, = 0) and by injecting carefully-designed Gaussian noise (00 A Ip). However, this special SGLD design differs from the common practice of SGD algorithms used to train neural networks in two key ways:
4

---

Published in Transactions on Machine Learning Research (12/2024)
1. SGLD performs all analyses under mutinuous-timm dynamics whereas actual SGD is performed with discrete steps. While related work such as Stephan et al. (2017) cast the continuous-time dynamics of SGD as a multivariate Ornstein- Uhlenbeck process (similar to SGLD) whose stationary distribution is proven to be Gaussian (Equations 11 and 12 in Stephan et al. (2017)), they make additional assumptions such as the noise covariance matrix being independent of model parameters.
SGLD assumes vanishing learning rate until convergence, whereas SGD is performed with a non- vanishing step size and for finite number of iterations in practice. The learning rale of SGD is often large, which can cause model dynamics to drift even further from SGLD (Ziyin et al., 2023), especially in the discrete-time setting (Liu et al., 2021).
We thus characterize the analytical form of the posterior distribution with respect to model parameters trained with SGD:
Theorem 3.1 (Postcrior for SGD). Assume the same assumptions as used in Theorems 2.2 and 2.3. Let w" be the local minimum that SGD (Equation 2) is converging towards. Then, the (conditional) log- rag-pobbbblityy of observing parameters w is given by (up to constants and negligible terms):
d In L, + Σ In
(2- Itp (H, + (o, + +a)
A
S(1 S(1 (1--M) 11)
(1 da
|| w
2
(H.).
21
1 1+ p
L.
1=1
5(1 u)a
(2 (2- 1 1
Aa
VL(w) (LW)
S(1 n)
L(w)
4A
L.
2(1 + 11) L.
where L.- = L(w"), H. H(w") and o, (H.) denotes the i-th largest eigenvalue of H.
A proof for Theorem 3.1 given in Appendix B. Theorem 3.1 suggests that the posterior distribution of model parameters learned by SGD not only relies on the training loss L(w) but is also crucially dependent on other terms, such as the Hessian structure H the gradient VL(w) and the lo distance ||w =w'l2, ll2. confirming that Equation is insufficient to model the dynamics of discrete-time SGD algorithm.
3.2 Optimal Membership Inference under Discrete-time SGD
We have explained why the assumption imposed by Sablayrolles et al. (2019) about the posterior distribution of w following a Boltzmann distribution (Equation 5) does not hold for stochastic gradient methods typically employed in practice. Next, prove thcorem that gives estimate of the optimal membership inference scoring function by leveraging recent results on discrete-time SGD dynamics (Liu et al., 2021; Ziyin et al., 2021). Our derivation is based on the assumptions that the loss achieved at the local minimum is unaffected by removing a single training record and that the Hessian structure remains unchanged.
Assumption 3 (Similarity at local minimum). For any T and ž1, let Lo(w) 1 Σn= mil(w, 2;) and L (w)) = (C(w,, 21) + Σ myl(w, When the training dataset differs only by a single data point ø1; assumee that the Hessian matrix structure for models trained with and without the differing point share a similar structure, and the loss function also achieves a similar value at the local minimum:
H. = Ho(w)) = Hww)), L. = Lo(w)) = [nww))
(6)
where wy is the local minimum that SGD with Lo is converging towards, and H, denotes the Hessian matrix with respect to Lo (and likewise for wi, L1, and Hj).
As long as the size of the training datasel is sufficient and the excluded training record 2 is not a low- probability outlier from the data distribution D, we expect Assumption 3 generally holds for SGD algorithms. Under Assumption 3 and a few other assumptions imposed prior literature on discrete-time SGD dynamics (Liu et al., 2021; Ziyin et al., 2021), we obtain a theorem (proof in Appendix C) that describes the scoring function for an optimal membership-inferene adversary:
5

---

Published in Transactions on Machine Learning Research (12/2024)
Theorem 3.2 (Optimal Membershipp-Iferrer Inference Score). Given w produced by an SGD algorithm defined by Equation 2 and record 21, the optimal membership inference M(w, 21) is given by:
o (
S(1 ((1-1))
(1(w,x1)
1 (1
12 + 12 12 +1a+11) + In (1
ET
(7)
2nl+
1+
where I, I2, I3, and Is are defined as follows:
1 = 1
1
1 + 1
λa
12 = 2
1
(H.
VH
1 + μ
13 = a
2
λa
(
2n
1 11)
Aa
I4 =
2
(H.'VLo(w)) HH. H. Vew z1)) (V
+ μ
Here, L. and H. are defined in Assumption 3, and are dependent on T. Here, T refers to the set of both member and non-member records along with their corresponding membership indicators, as defined in Lemma 2.1. Note that computing the optimal score requires access to the Hessian and model gradients, both of which require access to the model parameters. In fact, knowledge of the learning rate X, momentum μ., and regularization parameter are also required, thus requiring complete knowledge of the training setup of the target model. Thus, black-box access is not sufficient for optimal membership inference.
The first two additional terms I and I2 can be interpreted as the magnitude and direction, respectively, of a Newtonian step for the given record ø1. The first term J1 characterizes the influence magnitude in f--norm of upweighting 21 on the model parameters close to the local minimum (Koh & Liang, 2017), while the second lerm captures the alignuentt between the influence of 21 and the averaged influence of the remaining training data. larger influence magnitude of z1 or an increased influence aligument suggests a higher risk of membership inference. We remark that the notion of self-influence function introduced in Cohen & Giryes (2024) naturally relates to I1, suggesting a similar insight to ours that better membership inference attacks can be designed by leveraging the influence function of the inferred record. The last two additional terms, 13 and 4, originate from the extra L2 regularization term imposed on the training loss of SGD (Section 2.2). When the regularization parameter a is a very small positive constant, the effects of I3 and I4 on optimal membership inference will be negligible, particularly compared to those of I and Iz.
3.3 Inverse Hessian Attack
While Theorem 3.2 directly prescribes an optimal membership inference adversary, computing the expecta- tion over all possible models trained using the rest of the training data is infeasible. Our definition of optimal membership inference corresponds to the true leakage the model (as defined in Section 3.2 of Ye et al. (2022))- utilizes worst-case vevesary knowledge (membership of all other training records) and white-box access to estimate the influence of the target record, similar to how empirical attacks such as LiRA (Carlini et al., 2022) and RMIA (Zarilzadeh et al., 2023) use reference models to account for alypical examples.
Specifically, making use of the insight of Theorem 3.2, we propose scoring function based on the terms inside the expectation in Equation 7:
IHA(z1) = {(w, 21) 1+
1a
The score, 1HA((1), for some given record æ1, can be used as the probability of æ being a member. This serves directly as a useful attack for privacy auditing, without needing train any reference models. Not having train reference models offers significant advantages. helps auditors avoid additional computational costs and, more importantly, eliminates the need for trainers to reserve hold-out data for reference model training.
6

---

Published in Transactions on Machine Learning Research (12/2024)
This is particularly beneficial when data availability is constraint for privacy auditing methods relying reference models. While the absence of a negative sign with the loss function (like in T.OSS) in (HA may seem counter-intuitive at first glance, it can be rewritten such that it is proportional to the negation of the loss function (Appendix C.2).
While membership leakage typically evaluated on fixed dataset, the theoretical notion of optimal mem- bership inference defined for a much larger space. This space encompasses a broad distribution of possible data (including both member and non-member records) and the corresponding models trained on various splits of these datasets. In practice, it's challenging realize this larger space, but if we could define true positive rates (TPRs) at low false positive rates (FPRs) with respect to this comprehensive space, they would empirically correspond to the optimal membership inference attack. This theoretical framework pro- vides a more robust understanding of membership inference, though its practical implementation remains a significant challenge privacy auditing.
The performance of our audit is also influenced by other factors, such as how efficiently and accurately the unverse-Hessian vector products (iHVPs) can be computed and to what degree assumptions hold (particularly Assumption 3, which requires the Hessian and loss at local minima being unaffected by the exclusion of a single datapoint).
4 Experiments
To evaluate IHA, we efficiently pre-compule VI (w) to facilitate the computation of Vn((w) for any given target record ø1. For accurate Hessian matrix computations, we address the issue of ill-conditioning due to near-zero and small negative cigenvalues by either damping or using low-rank approximations (damping- based conditioning seems to perform best; see Appendix E for details). To support larger models where direct. Hessian computation infeasible, we extend our method to use approximation methods based on Conjugate Gradients for iHVP computation (Koh Liang, 2017). Our implementation for reproducing all the experiments is available open-souree code at https://githu/: wttollocoomaaaaaaeee comm//aaggoot22/audtttooooo
Section 4.1 describes the baseline attacks, datasets, and models we use for our experiments. Section 4.2 summarizes our results, showing that IHA provides a robust privacy auditing baseline, matching or exceeding the performance of current state-of-the-art allacks including attacks that use reference models. This is notable since IHA does ont require training any reference models or the use of hold-out data.
For given false positive rate (FPR), threshold is computed using scores for non-members, which then used to compute the corresponding true positive rate (TPR). This is repeated for multiple FPRs to generate the corresponding ROC curve, which is used to compute the AUC, This experimental design is commonly used for nembership-infeernce evaluations (Ycom et al., 2018; Carlini et al., 2022; Ye et al., 2022).
4.1 Setup
To evaluate IHA, we compare its performance state-of-the-art bascline attacks with representative set of datasets and models.
Baseline Attacks. We include LOSS as a baseline that does not use reference models, SIF as it uses iHVP similar to our audit, and LiRA as it is the current srate-of-the-art for membership inference. While RMIA (Zarifzadeh et al., 2023) uses fewer reference models, it achieves performance comparable to LiRA and thus for the sake of performance comparison, it suffices to use LiRA with large number of reference models. We describe the underlying access assumptions for these attacks in Table 1.
LOSS (Yeom et al., 2018). The negative loss is used in this attack as direct signal for membership inference.
SIF (Cohen & Giryes, 2024). Similar to ours, this attack employs the loss curvature the target model by computing its Hessian, which is then used to compute a self-influence score. The original attack assigns 0-1 scores to target records. classifies given record as a member if its self-influence score is within the specified range and if its predicted class is correct. The latter rule be ruled out having many false
7

---

Published in Transactions on Machine Learning Research (12/2024)
Attack
Model Access
Reference Models Used?
Leave-one-out knowledge?
LOSS
Black-box
No
No
LiRA
Black-box
Yes
No
SIF
White-box
No
No
Black-box
Yes
Yes
LiRA-L
Black-box
Yes
Yes
IHA (Ours)
White-box
No
Yes
Table 1: Comparison attacks based on level of model-access, use of reference models, and knowledge of other training members.
Dositives/negativee Instead of these steps, we choose to use the self-influence as membership scores directly. While the authors usod approximation methods for iHVP, we use the exact Hessian for fair comparison.
LiRA (Carlini et al., 2022). There are two variants, LiRA-Offinee, which uses "offlin" models to estimate Gaussian distribution and then performs one-sided hypothesis testing using loss scores, and LiRA-Online, with additionally employs "online" models, i.c., models whose training data includes the target record. The likelihood ratio for online/ofli /ofline model score distributions is then used as the score for membership inference. We use LiRA-Online since it is the strongest of the two variants.
L-Attack (Ye al., 2022). The L-attack operates in a leave-oneoout setting, training reference models on DV(2) for any given record z. It uses loss the target metric and computes attack thresholds for desired false positive rate (FPR) by leveraging the distribution of losses obtained from reference models.
LiRA-L. We propose combining the LiRA allack for the LOO-setting by utilizing reference models trained under leave-one-out availability, followed by the offline variant of the LiRA attack (Carlini et al., 2022).
Datasets. Since we are limited by the computational constraints of computing iHVPs, we restrict our experiments to datasets where small models can perform adequately.
Purchase-100($) The task for this dataset (Shokri et al., 2017) is to classily given purchase into one of 100 categories, given 600 features. We train 2-layer MLPs (32 hidden neurons) with cross-entropy loss, with an average test accuracy of 84%. Experiments by Zarifzadch et al. (2023) train larger (4-layer MLP) models on 25 samples from Purchase-100, which much smaller than the actual dataset, which why we term (irchase-100(S) (Small). We also demonstrate results with 4-layer MLP that achieves similar task accuracy.
Purchase- 100. For this version, we train models with 80 K samples. We use the same 2-layer MLP archi- tecture as Purchase-100f but achieve higher test accuracy of 90%. Using more data increases the scope for model performance. We report results for Purchase- 100 in Table 2 as the corresponding models are less prone to overfitting. For completeness, we report results for Purchase-100p
in Appendix D.
MNdST-Odd. We consider the MNIST dataset (LeCun al., 1998), with the modified task of classifying a given digit image as odd or even. This modified task allows to train models for binary classification using the regression loss, and is thus highly likely to follow the assumptions made in theory regarding quadratic behavior for the loss function (Assumption 1). We train logistic regression model with mcan-squared error loss, with an average Lest. loss of .078.
MNSST We use the FashionMNIST (Xiao al., 2017) dataset, where the task is classify a given clothing item image into one of ten categories. We train 2-layer MLPs (6 hidden neurons) with cross-entropy loss, with average test accuracy of 83%.

---

Published in Transactions on Machine Learning Research (12/2024)
Table 2: Performance various attacks, reported via attack AUC and true positive rate (TPR) at low false positive rate (FPR). ROC curves for low FPR region are visualized in Figure (Appendix).
Purchase-Ii 100
MNIST-Odd
Fashion
% TPR@FPR
% TPR@FPR
% TPR
Attack
AUC
AUC
AUC
1%
0.1%
1%
0.1%
1%
0.1%
LOSS
.531 2.001
0.97
0.00
.500
4.003
0.97
0.09
.507
+.002
1.00
0.10
SIF
.531 2.001
0.97
0.10
.500
0.00
0.97
0.10
.507
+.002
0.98
0.10
LiRA
.644 1.004
4.70
0.98
.568
1.005
2.77
0.63
.578
1.020
2.98
0.63
IHA (Exact)
.703 1.004
13.69
7.52
.542
4.004
2.61
0.51
.594
1.018
4.06
0.89
Models. We train 128 models in the samn way as done in Carlini et al. (2022), where data from each model is sampled random from the actual dataset with a 50% probability. For each target model and target record, there are thus 127 reference models available, half of which are expected to include the target record in the training data. All of our models trained with momentum (μ 0.9) and regularization (a 5e 5)= with a learning rate X = 0.01. For a given false positive rate (FPR), a threshold is computed using scores for nembees which is then used to compute the corresponding true positive rate (TPR). This is then repeated for multiple FPRs to gencrate the corresponding ROC curve, which is used to compute the AUC. This experimental design is commonly used for menbershhipiinferrncee evaluations (Yeom al., 2018; Carlini et al., 2022; Ye et al., 2022).
4.2 Results
As summarized in Table 2, IHA provides a strong privacy auditing baseline that is competitive with current ttate-of-the-art attacks that require reference models. This is especially useful, considering that IHA does not require training any reference models and thus, does not require any hold-out data to train such reference models. IHA performs much better than the baselines on tabular data (Purchase-100) 100), and is competitive with the baseline for image-based data (MNIST-Odd. Fashion MNIST). By extension, outperforms previous membership inference attacks that specifically utilize parameter access via white-box access (Nast et al., 2018; Cohen & Giryes, 2024), since such methods are outperformed by LiRA (Carlini et al., 2022). While tabular data is a more realistic setting for membership inference and the improved performance on Purchase- 100 promising, we leave to future work further investigation of these factors to better understand the performance discrepancics.
method also
Approximating iHVPs. In order to carry out THA, auditor needs to be able calculate iHVPs and gradients for all training data. While computing gradients is more computationally intensive than simply calculating the loss, the difference is minimal. On the other hand, computing an iHVP involves calculating the Hessian matrix and then inverting it, both of which are computationally expensive processes. Even storing such an inverted Hessian can be problematic (p x p matrix for a model with p parameters). We thus experiment with evaluating IHA using Conjugate Gradients (Koh & Liang, 2017) to approximate iHVP. While such approximation does not require computing the Hessian directly, the time taken to compute this random sample
term for each record is non-trivial. We thus evaluate this approximation-basee method of 10 000 records and find that approximation methods retain most of the attack's 's performance (Table 3).
We emphasize that the purpose of our comparisons is not to claim a better membership inference attack for adversarial use; the thrcat models are not comparable, since our attack requires knowledge of all other records } for inferring a given larget record 2 (relaxing this assumption leads to severe performance degradation, see Appendix F). Instead, THA provides a way empirically audit models for membership leakage without training reference models, which is desirable in avoiding the need reserve hold-out data for training reference models. More importantly, results suggest untapped opportunities in exploring
For direct. comparison, recompute metrics the 10000 samples which use approximate- hased based variants.

---

Published in Transactions on Machine Learning Research (12/2024)
Table 3: Performance exact and approximation- based variants of IHA, reported via attack AUC and true positive rate (TPR) at low false positive rate (FPR). Statistics are computed on 10000 samples.
Exact
CG
Datasel
%TPR@FPR
%TPR@FPR
AUC
AUC
1%
0.1%
1%
0.1%
Purchase-100
.701 =.009
13.74
7.56
.701
000
13.72
7.55
MNIST-Odd
.541 2.0
2.76
0.43
.541
000
2.76
0.43
MNShionMNIST
.593 1.018
4.10
0.85
.592
1.018
4.09
0.86
Table 4: Performance of IHA Purchase100-S (MLP-2) when only some of the terms corresponding to Equation 8 are used. 1 and I seem to be responsible for most of the privacy auditing performance.
%TPR@FPR
Terms Used
AUC
1%
0.1%
.591 000
1.04
0.09
Iz
.704 =.004
2.63
0.70
I1 T2
.779 2.003
17.61
16.65
I1, 12, 13. 14
.779 =.003
17.60
16.64
8(w, 21), 11
.594 -.00
1.12
0.12
8(w, 21), 12
.686 .004
1.97
0.48
[(w, 21), 11,
.791 .003
19.96
19.00
All
.791 .00
20.09
19.09
parameter access for stronger privacy andits as well as the possibility new white-box inference attacks from an adversarial lens.
4.3 Ablating over terms inside IHA
As described in Equation 8, calculating IHA requires computing the loss along with the four additional terms 11, 13 and 4. However, the terms 13 and / are scaled by a (which is usually very small) and involve an iHVP of an iHVP and thus may be much smaller compared to terms like I, Iz and the loss. We explore variants IHA, which ignore the terms I3 and 14, to see how they impact auditing performance. We also consider variants that use only the terms I and Ig to understand the importance of their contributions the performance of IHA, along with the inclusion or not of the loss term to understand the relative importance of baase signals.
The ablation study presented in Table 4 reveals several key insights about the performance of IHA. Excluding la and 14 has negligible impact on auditing performance, even for low-FPR scenarios. The combination of I and Ig alone achieves an AUC of. 779, and performance identical when the terms 13 and 1 are also included. Notably, the addition of the loss term l(w, 21) to Jn and I2 results a marginal improvement, increasing the AUC to .791 and slightly boosting the TPR both 1% and 0.1% FPR. Interestingly, when examined individually, I2 (AUC .704) performs significantly better than I (AUC .591), suggesting that In captures more relevant information for the auditing task. Including the loss term l(w, æ1) has little impact / but harms performance when used wth Iz. These findings indicate that the majority of the attack's effectiveness stems from the inverse Hessian vector products used in I and 12, with I2 being particularly important, while the terms involving weight regularization and nested iHVPs (Ig and 4) contribute minimally to the overall performance. While not as impactful as I2, the loss term still provides valuable information for the auditing process. Based on these results, a simplified version of IHA using only É(w, 21), I, and In could potentially offer favorable trade-off between computational efficiency and auditing effectiveness.
10

---

Published in Transactions on Machine Learning Research (12/2024)
Table 5: Agreement rate between ground truth (GT) membership values, and various attacks for 500 training and 500 testing data points. The upper triangle of the table corresponds to the agreement rates of members, whereas the lower triangle corresponds the agreement rates of non- members. The experimental setup is
Purchasc-100(S), with effective FPR 0.05 (a) and effective FPR = 0.3 (b).
LIRA
L
LiRA-L
IHA
GT
LiRA
L
LiRA-L
IHA
GT
LiRA
.794
.826
.718
.200
LiRA
.732
.658
.436
.652
.930
.908
.712
.234
.724
.766
.492
.668
LIRA-L
.916
.938
.768
.154
LIRA-L
.588
.660
.462
.518
IHA
.912
.26
.166
.230
IHA
.612
.660
.584
.640
GT
.954
.922
.954
.958
GT
.702
.702
.706
.706
(a) Agreement between methods with FPR 0.05
(b) Agreement between methods with FPR 0.3
4.4
Comparison with Leave-One-Our Setting
When targeting a record for inference, IHA assumes knowledge of all the other n - records in an n-sized dataset. It possible that the improved performance of IHA is due to this extra information rather than inherent parameter access. To isolate and analyze these potential sources of increase in leakage, we also assess the performance of leave-one-oul (IOO)) membership inference lest. We evaluate the L-attack (Ye et al., 2022) on 1000 samples, training 100 reference models per record for score calibration. Since targeting cach record requires training multiple reference models for the L-attack, evaluating it on larger sample of data is computationally infeasible.
These results indicate that IHA outperforms the L-attack, achieving an AUC value .791 compared to .737 for the L-attack. 2 This suggests that even when controlling for the additional knowledge of all other records in the dataset, the primary source of IHA's superior performance stems from its access to model parameters rather than just the leave-one-out. setup. In way, THA uses parameter access to obviate the need for reference models, as it directly aims to measure the influence of the given target record instead of relying on reference models for score calibration.
Interestingly, in comparison, LiRA achieves an AUC of .767, which is lower than IHA but still higher than the L-attack. This suggests that LiRA, even without the extensive reference model training, is more effective than the L-attack, possibly due to its utilization of both "in" and "out" models as opposed to just "out" models with the L-attack. To try and devise a stronger black- attack for the LOO setting, we extend LiRA to the LOO sctting by using models trained on LOO data reference models. LiRA-Offlne under the LOO setting achieves AUC of .633, lower than the L-attack (.737). Overall, these results demonstran rate the promise of IHA as a privacy auditing tool. It yields results comparable to that of techniques that train hundrods of reference models (for each target record in the worst case, as in L-attack), without using a single reference model.
4.5 Inter-Attach Agreement
Similar to Ye et al. (2022), we compute the agreement rate between ground-truth membership labels and membership predicted by various attacks to understand the ability of our privacy audit to identify vulnerable data, and demonstrate how it differs from existing attacks. Table 5 presents the agreement rate between ground-truth membership labels and membership predicted by various attacks.
At a low FPR of ~ 0.05, agreement in predictions for non-members is very high between attacks, with agreement rates above 0.91 for all pairs of attacks. On the other hand, agreement rates for member records are expectedly lower. Interestingly, agreement between LiRA and LiRA-L is higher than IHA and any other attack. This difference is especially pronounced for higher FPR (Table 5b), where agreement rates are as
2Our results for the L-attack are lower than those reported by Ye al. (2022). For instance, we observe TPR of .668 0.3 PPR, while it reported be .968 by (Ye al., 2022). This discrepancy arises from setting, which more data and fewer model parameters. We verified implementation through direct. correspondence with the authors and by replicating their results in the original setting, which used smaller dataset and more parameters, resulting a model prone to overfitting.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Attack
Time (8)
Memory (MB)
Precompute
Time/Sample
Precompute
Runtime
max(Precompute, Runtime)
THA
43 60
0.16
4228
1324
4228
THA (Approx)
85
0
1638
1638
LiRA
192 60
0.07
276
1228
1228
LOSS
0.004
0
906
906
Table 6: Runtime and memory statistics for various auditing techniques. Statistics are computed over 100 randomly-solocted members for MLP-2 architocture models trained on Purchaso-100 datasct.
low as z 0.4 compared to 0.766 for LiRA and LiRA-L, This is very interesting becaue the corresponding TPRs for IHA are higher than LiRA and comparable that of the L-attack, thus suggesting that the records identified by IHA being vulnerable are very different from those identified by LiRA or even the L-attack. This also means that combined (classifying record a member only when both attacks classify member) attack would have some true positives with very low FPR.
4.5.1 Runtime Comparison
To compare the computational costs of our proposed audit with existing auditing techniques, we analyze runtime and memory usage statistics across different methods, aiming to evaluate efficioncy and practicality in real-world privacy audits (Table 6).
While the peak memory consumption of IHA is higher than that of LiRA, the approximate version of IHA is not too far from LiRA in terms of memory consumption. Computing the total runtime is function of the number of samples used for the privacy audit, a "precompule" step is required for LiRA (training reference models) and exact IHA (computing Hessian). For instance, computing the audit for 1K samples takes less overall time for IHA ( hour) than it does for LiRA (~ 3 hours). It should be noted that the Hessian is too large to store on our GPU for IHA and thus stored on the CPU, which is also why it slower. Although improvements may reduce the compute costs, the key advantage of such a privacy audit comes from not having reserve hold-out (or auxiliary) data. Our privacy audit, like the most trivial LOSS atlack, only requires member data and some non-member data, whereas other allacks in the literature require shadow shadow/referent /reference models trained on comparable-sized datasets; the limiting factor here is reserving data to train reference models, which real-world model trainer may not want to do since it reduces the amount of data available for training.
5 Conclusion
Our theoretical result proves that model parameter access is indeed necessary for optimal membership inference, contrary to previons results derived under unrealistic assumptions and the common belief that optimal membership inference can be achieved with only black-box model access. We propose the Inverse Hessian Attack inspired by this theory that provides stronger privacy auditing than existing black-box techniques.
Limitations. IHA not yet practically realizable for most settings due to the computational expense of calculating the Hessian, or even approximating iHVPs. This restriction poses challenges for real-world cases where fixed compute budgets may be more crucial than the availability of auxiliary data. An auditor might use subset of parameters to reduce computational costs while performing Hessian--bssed computations. This aligns with model pruning (Liu et al., 2019), but understanding its impact on membership knowledge within parameters is non-trivial (Yuan & Zhang, 2022). We also note that IHA's performance can be sensitive to the choice of the damping factor, which requires further investigation (Appendix D.1).
Our conclusion aligns well with recent calls the literature to consider whitc-box access for rigorous auditing (Casper al., 2024). While our theory shows that parameter access required for optimal membership inference, it remains unclear how much better this is compared to the optimal membership inference attack
12

---

Published in Transactions on Machine Learning Research (12/2024)
restricted to black-box access. Our empirical studies suggest the gap is non-trivial, but further study is required to understand the theoretical limit of black-box attacks, which is a non-trivial but interesting direction to explore. Exploring the accuracy of iHVP approximation methods to extend IHA to larger models, along with multi-record inference, are both promising directions for future research.
Broader Impact Statement
The increasing integration of AT in sensitive domains like healthcare, finance, and personal data manage- ment highlights the critical importance of privacy. Information leakage from Al models can have severe consequences, making effective privacy auditing a necessary safeguard. Our work contributes to this field by theoretically demonstrating that optimal membership inference attacks require white-box access to model parameters, challenging the adequacy of black-box approaches. We also demonstrate with Inverse Hessian Atlack how this theory can be used to design empirical privacy audits that do not rely on reference models.
We advocate for the development of more sophisticated privacy auditing tools that fully leverage the elevated access typically available to auditors, such as model paramcters, to assess privacy Icakage efficiently without extensive data and compute resources. We hope our theoretical ical and empirical results will reinvigorate interest in the privacy research community to explore white-box attacks, for both adversarial and auditing purposes.
References
Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 2017.
Achral Azize and Debabrota Basu. How much does each datapoint leak your privacy? quantilying the per-datum membership leakage. arXiv:2402.1006 2024.
Martin Bertran, Shuai Tang, Aaron Roth, Michael Kearns, Jamie Morgenstern, and Steven Z. Wu. Scalable membership inference attacks via quantile regression. Aduances Neural Information Processing Systems, 2024.
Stella Biderman, USVSN Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 2024.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In IEEE Symposium Security and Privacy, 2022.
Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, et al. Black-box access is insufficient for rigorous ai audits. In ACM Conference on Fairness, Accountahility, and Transparency, 2024.
Harsh Chaudhari, Giorgio Severi, Alina Oprea, and Jonathan Ullman. Chameleon: Increasing label-only membership leakage with adaptive poisoning. In International Conference on Learning Representations, 2024.
Gilad Cohen and Raja Giryes. Membership inference attack using self influence functions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.
Rachel Cummings, Damien Desfontaines, David Evans, Roxana Geambasu, Yangsibo Huang, Matthew Jagiclski, Peter Kairouz, Gautam Kamath, Sewoong Oh, Olga Ohrimenko, Nicolas Papernot, Ryan Rogers, Milan Shen, Shuang Song, Weijie Su, Andreas Terzis, Abhradeep Thakurta, Sergei Vassilvitskii, Yu-Xiang Wang, Li Xiong, Sergey Yekhanin, Da Yu, Huanyu Zhang, and Wanrong Zhang. Advancing differential privacy: Where we are now and future directions for real-world deployment. Harvard Data Science Review, 2024.
Daniel DeAlcala, Aythami Morales, Gonzalo Mancera, Julian Ficrrez, Ruben Tolosana, and Javier Ortega- Garcia. Is my Data in your AI Model? Membership Inference Test with Application to Face Images. 092222 2024.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryplography Conference. Springer, 2006.
Mishaal Kazmi, Hadrien Lautraite, Alireza Akbari, Mauricio Soroco, Qiaoyue Tang, Tao Wang, Sébastien Gambs, and Mathias Lécuyer. PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining. In Advances in Neural Information Processing Systems, 2024.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, 2017.
Sasi Kumar and Reza Shokri. MI privacy meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning. In Workshop on Hot Topies in Privacy Enhancing Technologies (HotPETs)) 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applicd to docu- ment recognition. Proceedings of the IEEE, 86(11), 1998.
Marvin Li, Jason Wang, Jeffrey George Wang, and Seth Neel. Mope: Model perturbation based privacy attacks on language models. In Conference on Empirical Methods in Natural Longuage Processing, 2023.
Kangqiao Liu, Liu Ziyin, and Masahito Ueda. Noise and fluctuation of finite learning rate stochastic gradient descent. In International Conference on Machine Learning, 2021.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value network pruning. In International Conference on Learning Representations, 2019.
Takashi Mori, Liu Ziyin, Kangqiao Liu, and Masahito Ueda. Power-law escape rate of sgd. In International Conference on Machine Learning, 2022.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning. In IEEE Symposium on Security and Privacy, 2018.
Eire Talea Oldewage, Ross M Clarke, and José Miguel Hornández-Lobato. Scries of hessian-vector products for tractable saddle-free newton optimisation of neural networks. Trunsactions on Machine Learning Research, 2024.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs black-box: Bayes optimal strategics for membership inference. In International Conference on Machine Learning, 2019.
Ahmed Salem, Giovanni Cherubin, David Evans, Boris Köpf, Andrew Paverd, Anshuman Suri, Shruti Tople, and Santiago Bénuull-Béguelin. SoK: Let the privacy games begin! A unified treatment of data inference privacy in machine learning. In IEEE Symposium on Security and Privacy, 2023.
Issei Sato and Hiroshi Nakagawa. Approximation analysis of stochastic gradient langevin dynamics by using fokker-planck equation and ito process. In International Conference on Machine Learning, 2014.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In JEEE Symposium on Securily and Privacy, 2017.
Mandt Stephan, Matthew D Hoffman, David M Blei, and others. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research, 2017.
Jasper Tan, Blake Mason, Hamid Javadi, and Richard Baraniuk. Parameters or privacy: A provable tradeoff between overparameterization and membership inference. Advances in Newral Information Processing Systems, 2022.
Max Welling and Yee W Tch. Bayesian learning via stochastic gradient langevin dynamics. In International Conference Machine Learning, 2011.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for benchmarking machine learning algorithms, August 2017. arXiv: cs...G/1708.07777.
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gra- dient descent exponentially favors flat minima. In International Conference on Learning Representations, 2021.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In ACM Conference on Compuler and Communécations Security, 2022.
Jiayuan Ye, Anastasia Borovykh, Soufiane Hayou, and Reza Shokri. Leave-one-oul distinguishabilit in machine learning. In International Conference on Learning Representations, 2024.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Ana- lyzing the connection to overfitting. In Computer Security Foundations Symposium, 2018.
Samuel Yoom, Trene Giacomelli, Alan Menagod, Matt Fredrikson, and Somesh Jha. Overfitting, robustness, and malicious algorithms: A study of potential causes of privacy risk in machine learning. Journal of Computer Securily, 2020.
Soma Yokoi and Issci Sato. Bayesian interpretation of SGD as Ito process. orXim:1911.0000 2019.
Xiaoyong Yuan and Lan Zhang. Membership inference attacks and defenses in neural network pruning. In USENIX Securily Symposium, 2022.
Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference by boosting relativity. arXiv:2812.080 03262,, 2023.
Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength of minibatch noise in SGD. In International Conference on Learning Representations, 2021.
Liu Ziyin, Hongchao Li, and Masahito Ueda. Law of balance and stationary distribution of stochastic gradient descent. ar arXix:2908.000 06677, 2023.
A Related Work
This section reviews methods in membership inference (black-box and white-box), techniques for privacy auditing to predict and mitigate data leakage, and the dynamics of stochastic gradient descent (SGD) along with inverse Hessian vector products (iHVPs).
A.1 Membership Inference
Black-box Membership Inference. Early works on membership inference worked under black-box access, utilizing the model's loss (Shokri et al., 2017) on given datapoint as a signal for membership. Since then there have been several works focusing on different forms of difficulty calibration accounting for the inherent "difficulty" of predicting on a record, irrespective of its presence in train data. This calibration has taken several forms; direct score normalization with reference models (Sablayrolles based score distributions (Carlini al., 2022; Zarifzadeh et al., 2023; Ye et al., 2022), and additional models for predicting difficulty (Bertran et al., 2024).
caabration-accountinn
al., 2019), likelihood tests
White-box Membership Inference. Nasr et al. (2018) explored white-box access to devise a meta- blassifier-based attack that additionally extracts intermediate model activations and gradients to increase Icakage but concluded that layers closer to the model's output are more informative for membership inference and report performance not significantly better than a black-bor loss-based attack. Recent work by DeAlcala et al. (2024), however, makes the opposite observation, with layers closer to the model's input providing
15

---

Published in Transactions on Machine Learning Research (12/2024)
noticeably better performance. Apart from these meta-classifer driven approaches, some works attempt to utilize parameter access much more directly, often utilizing Hessian in one form or another. Cohen & Giryes (2024) defined the self-influence of a datapoint 2; as (g: 9H) as a signal for membership, using LiSSA (Agarwal et al., 2017) to approximate the iHVP. This has similaritics to our result since our optimal membership inference score also involves computing iHVPs. Li et al. (2023) attempted to measure the sharpness for a given model by evaluating fluctuations model predictions after adding zero-mean noise to the parameters, a step that supposed to approximate the trace of the Hessian at the given point.
A.2 Privacy Auditing
Ye et al. (2024) proposed using efficient methods to "predict" memorization by not having to run computa- tionally expensive membership inference attacks, with reported speedups of up to 140r. They showed how their proposed score (LOOD) correlates well with AUC, corresponding to an extremely strong MIA with all-but-onc access to records (L-attack (Ye et al., 2022)). However, it is unclear if this computed LOOD is directly comparable across models, making it hard to calibrate these scores to compare the leakage from a model relative to another (an important aspect of internal privacy auditing). Their derivations also involve connection with the Hessian. Biderman et al. (2024) studied the problem of forccasting memorization in model for specific training data and proposed using partially trained versions of the model (or smaller models) as proxy for their computation. While their results support the need for inexpensive auditing methods, their focus is on predicting memorization early in the training process, while ours relates to audit- ing fully trained models. More recently, Tan et al. (2022) studied the theory behind worst-case membership leakage for the case of linear regression on Gaussian data and derived insights. While this useful to make an intuitive connection with overfitting, it does not provide a realizable attack or insights for the standard case of models trained with SGD.
A.3 SGD Dynamics and iHVPs
SGD Dynamics. Stephan et al. (2017) approximated the SGD dynamics as an Orastein-Uhlenbedd process, while Yokoi & Sato (2019) provided discrete-time weak-order approximation for SGD based on Itō process and finite moment assumption. However, both works rely on strong assumptions about the gradient noises and require a vanishingly small learning rate, largely deviating from the common practice of SGD. To address the limitations of the aforementioned works, Liu et al. (2021) directly analyzed the discrete-time dynamics of SGD and derived the analytic form of the asymptotic model fluctuation with respect to the asymptotic gradient noise covariance and the Hessian matrix. Ziyin et al. (2021) further generalized the results of (Liu et al., 2021) by deriving the exact minibatch noise covariance for discrete-time SGD, which is shown to vary across different kinds of local minima. Our work builds on these advanced theoretical results of discrete-time SGD dynamics but aims to enhance the understanding of optimal membership inference, particularly for models trained with SGD.
vector products relies on one of two hosh conjugate gradients (Koh & Liang, 2017) or LiSSA (Agarwal et al., 2017). Both approximation methods rely efficient computation of exact Hessian- vector products, and use forward and backward propagation sub-routines. While these methods have utility in certain arcas, such as influence functions (Koh & Liang, 2017) and optimization (Oldewage et al., 2024), approximation errors can be non-trivial. For instance, I in the formulation of our attack requires low approximation error in the norm of an iHVP, while I2 simultancously requires a low approximation error in the direction of the iHVP. Recent work on curvature- aware minimization by Oldewage et al. (2024) proposes another method for efficient iHVP approximation as subroutine, but the authors observed high approximation errors based on both norm and direction.
iHVPs. Currently literature on approximating
16

---

Published in Transactions on Machine Learning Research (12/2024)
B Proof for Theorem 3.1
Proof. Recall that H. = H(w")) and L, = L(wx), According to Theorem 2.2 and Theorem 2.3, we obtain
== A
S(1 pd) 21, 2L, - a2w"w") (
(H. (H. H
-1 ) 21 1+ (H, + alaa)
(9)
where we only consider the leading terms in Theorems 2.2 and 2.3. Note that Equation holds when the Hessian matrix H, has full rank and I. 0. rank(H, ala) < d, the following more generalized result can be derived:
When the Hessian has degenerated rank such that
H. a2w"w" wT (H. + +oIl) )
)H.
A (H. + +Iaa) )
P,E
21
5(1 14)
1 1+
where P, = (la 1, 0....0) denotes the projection matrix onto non-zero eigenvalues, and + is the Moore- Penrose inverse operator. If L. = O, meaning w is a global minimum, then the asymptotic model fluctuation
Σ 0. For ease of presentation, we assume the Hessian matrix has full rank in the following proof.
Then, we get:
Σー S(1 u1)
A (H. + ala)
) (H. H. + ala) +
(2L.H.-aawwu (2L.H. a² woTwT)) -
21
(10)
1 p
Using the Sherman Morrison formula, we obtain:
2L,H. - -2w"" "www
-1
1
H.
ww H. H, (H. (H.' ww'w""" "''''H.'' H!!)
a²
2L
(2L.
1 (Ia 01++)
H.
(11)
2L_
Leaving out the sccond-order term O(a²) in Equation 11 (since the regularization parameter a is a typically small constant [0, 1)) and plugging back Equation 10, we get:
Σー S(1 - 14)
A
(H, + ala) ) H. +ald H.
21
2X
+ μ X
S(1 14)
Aa
21,
(H, + + + 2aH
I+ (Ia aH.
2AL.
1 1+ μ
S(1 14)
2 (2(1-
Xa
)
A (1 H. + a
Xa H.
La
(12)
2XLx
1 1+
+
According to Laplace approximation, we can approximate the posterior distribution of w given w" N(w", E). Therefore, making use of Equation 9, can derive the explicit formula of the log-posterior distribution as:
d ln(2T) det (E) / 7(w -ww) w'' (w ww)
1
Inp(w/w") = -
d
(2 Itp (H.) + a)) (o, + a)
In
+ const.
2 i=1
(H.
S(1 - μ1
Aa
Aa
2 1
w|w w 20 +
(w --=ww" H. w w*)
4XL.
1
1 p
1 (w w²) H. (w - w")
(13)
Note that we assume the model parameterizod by w is converging towards some local minima w" using stochastic gradient descent, where the loss landscape around the local minima w" has quadratic structure.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Therefore, we perform second-order Taylor expansion of the loss function at w" follow:
1 L(w) L(w") 7 (w w²) "H (w w") + o(llw w 1l2)
(14)
which further suggests that VL(w) = (w w*) if taking gradient with respect to w for both sides of Equation 14. Plugging Equation 14 into Equation 13, we further obtain:
d
(2
1ta (H.) + (o, ( ) + const.
In (p(www)))))
Σ
In
(H,),
1=1
S(1 11)
Aa
||0 - -wwwl
27L
1 +
5(1 11)
da
2 1 +
(w w " "H,
H. - w")
4AL.
S(1 p1)
(w w")
H.Ww w0)
4(1 + T.)T
(2
d I+#
(H. +)) (o: + +a)
In I.. + 2
Σ
In
+ const.
2
(H.)
1=1
S(1 - 11)
(1
λa
10 w
2A
1 μ
I
S(1 p)aa
(2
da
VL(W) T (((ww
4X
1+ /
L.
5(1 11)
[(w")
+ o(llw- w" w²112)
(15)
2(1 + pd) L.
Omitting the constant and negligible terms, we thus complete the proof of Theorem 3.1. Note that we keep the O(a?) term in Equation 13 and Theorem 3.1 for the sake of completeness but expect it to be negligible compared with other terms, due the fact that typically set as very small constant within [0, [0, 3
C Optimal Membership-Inference Score
C.1 Proof for Theorem 3.2
Proof. To derive the scoring function for an optimal membership inference, we need to compute the ratio between p(w)w() and p(w) wwj), where w (resp. wi) denotes a local minimum (close to w) of the training loss function with respect to (=....., [n} (resp. {z),..., z,)).. Note that we've obtained the posterior distribution of in Theorem 3.1. Therefore, the remaining task is to analyze the following terms:
In p(w/(/ - In ((w)22))
d = =- 2
In LI In Lo(w
1 d Σ in
(2
I+p
A (H,
+ ai) ( (H; (wi)) + a)
os (Ho
2
(2 T+p
d
(Ho((())) + a)) (Ho + a))
o, (H, (w
5(1 12)
Aa
||w0 will
||00 will
S(1 111-1 11)
(i(w) Lo(w)
+
2X
1 + μ
Li (www)
Lo(wE)
2(1 + p)
Li(wi)
Lo(wi)
5(1 - p)0x
Aa
VL (w)
TH -V
[Lo(w) T
Ho(w))
-3 Loo(
2
(16)
4X
1 +
[i
Lo(wx))
where Ho(w)) (resp. (w])) denotes the Hessian of Lo (resp. L2) at w (resp. wi). Since both wi and wi are close to parameters of the observed victim model w, so we can approximate the corresponding loss using second-order Taylor expansion. Also, according to Assumption 3, we know Ho(w)) H, (wx) H.
18

---

Published in Transactions on Machine Learning Research (12/2024)
and Lo(w L1 (ww Lu. Thus, we can simplify Equation 16 and obtain the following form:
In w(wpww(f In (hpp(wuurg (
S(1 - 14) 1
Aae )
w (b) + S(1 - 14) (1|0-wi1 wil2 1 1/w
(Lr(w) Lo(w))
2
1 1+ /
2(1 p.
S(1 1da
Aa
2 1+ 11
(2))(w) H. (w) VLo(w)
H 3 Lo
4XL,
S(1 - 14)
1
Ae
(VI((W)
H. VL)
(Lo(w) H. H. (Lo(w)) (L +
S(1 - 11) 21)
2
1 ++ H
2n(1 + L.
S(1 1daa
1 2 1+ 1 ) (2V H. H
d
-(w. 21) H , VV((,,
AnAL.
n
5(1 - 12)
Aae
S(1 m))(wo,
1
2VLo(10) H H V 21) 2 1+ )
H Vl(w,, (((ww,))11 21)
2nAL.
1/
2n(1 + L.
d S(1 p1)a
(2VLo(()) H. VV((w, + n
flw, x1) H. [V((w,)1)
(17)
4nAL.
where the second equality holds because of the Taylor approximation (L.(w) VLI(w)) H.(w w for EE 10. Morcover, according to Lemma 2.1, we know the optimal membership inference is given by:
M(w, 21) = -E (0(1)
( In p w m= = 71)
Y )]
w m+ = 0, - (121 + In
(18)
where o(u) = (1 +1++ar exp(-u))~ is the Sigmoid function, T = {z Zn. m2, mnJ. and y = P(m, = 1). Plugging Equation 17 into Equation 18, we obtain
M(w, =1) BTTo (0r(oS(110)) 2nLn (1+ À
S(1 [((1-1) 1)) (1(w,22) 1
(I1 12 122 +1 +2 + tt,)]]
(n)
where 11, 12, 13, 14 and ty are defined as:
h = 1
Aa
1 1 1
λa
12 = 2
1
(H. V( 21))
1 + 1
Aa
"H,, (H.' 21)) T (H." (H, VV((,,
13 =
2
2n
1 μ1
Aa
(H,'VLO(W) T (H, (H.'V((w, (((ww,,))),
1 2
1 p
7
by : In
Therefore, we complete the proof of Theorem 3.2.
C.2 Connection with LOSS attack
Note that while there are additional terms in our optimal membershippiinference score, there is another critical difference: the loss function has its sign flipped when compared to existing results (Yeom et al., 2018; Sablayrolles et al., 2019). While this may seem counter-intuitie + 14) terms in Equation 7 are expected to be negatively correlated to the loss function, leading to the proposed scoring function, in fact, aligns with the intuition of existing results. For simplicity, we consider the setting without regularization (i.e., 0) and the Hessian matrix has full rank.
first glance, we show below the addition
+ -4
According to the assumption of quadratic loss around w., we have the following observations:
L(w)= L.
]/( (w -w") H.(w -w") -
1
2 (w w TUT uTTTuudia
w²),
19

---

Published in Transactions on Machine Learning Research (12/2024)
where U diagfor,.... ,...,,, o is the eigenvalue decomposition of H. Let U(w ==) Since U is orthonormal matrix, we know ||e/|2 || - w= w=112- Thus, obtain
da 1/00 w" s og ||a12 2(L(w)) I.,) Σ 5=1
uz S 01 ||l|2 01 |/2 w 112.
which further suggests (provided the Hessian has full rank)
1 < w w (w) - I.,)
1
for any i E {0, 1}.
(19)
Based on Assumption 3, we assume that the Hessian structure and the loss function value remain unchanged with and without a single record ø1. We hypothesize that the ratio E 0 and i 1, where l E 01]. Thercfore, we have
w| ||--w;l8 also remains similar for ([,(w) L.)
28( (2,, 21) $ wlw w 1|w will
20 21) 12
(20)
no1
nod
Note that the derivation from Equation 19 to Equation 20 is not mathematically rigorous, but as long as the record æ1 is not too deviated from the data distribution, we expect the above inequality holds. Plugging Equation 20 into the log- likeliool term inside M(w, #1) (first equality in Equation 17 with a = 0), we get
S(1 p))
1
will l| -ou--w;le. + ((w, 21)
In Inp(w)/f) - In ((w))()))))))))))))))))))))))(
(1120
will)) (1 + p)m
2T.
5(1 - 11)
1
2
f(w,z1) z1)
(21)
2L.
1 1
Al
n
where ls is some real number that falls into [oa, 01). In addition, for the case of full-rank Hessian, is easy see that the i-th eigenvalue of Σ (Equation (4)) can be written as:
S 11)
(2 Ao;
) 1 +
2XL
positive
Since the covariance matrix E positive semi-definite and invertible, it must follow that all of its eigenvalues are positive:
Ao: > 1
1
2 < O, for any i i= 1, 1,2,....4 ...... .dd
1+
λσ;
With the above inequalities in mind, by looking at Equation (21), get:
5(1-H) 11)
2
((w.z1) 21)
In (np(w//?) - in ()w(ww)
(22)
2nd,
1 1+
Ak
n
<0
The upper limit on the score (hence the score itself) corresponding to IHA is thus proportional to the negative of the loss function, aligning with intuition (lower loss indicative of overfitting, and thus the record being a member). The score inside IHA can thus be interpreted (up to some approximation error) as -f(w, 21) 21) for some f(w, 21) > that essentially accounts for SGD training dynamics, and is a function dependent on parameter access to the target model.
D Purchase-100(S) v/s Purchase-100
Model trainers, under practical settings, would not want to produce sub-optimal models. Under the given experimental settings (access to Purchasel00 dataset), is thus crucial to simulate model training setups
20

---

Published in Transactions on Machine Learning Research (12/2024)
Table 7: Performance of various attacks on (00c)) and Purchase- 100. There is a clear drop in performance when shifting from Purchasc-100(S) Purchase- 100. Statistics for IHA (CG) are computed on 10000 samples, not the entire dataset.
Purchase-
Purchase- 100
Attack
MLP-2
MLP-4
% TPRRFPR
% TPR@FPR
% TPRRFPR
AUC
AUC
AUC
1%
0.1%
1%
0.1%
1%
0.1%
LOSS
.529 1.001
0.97
0.00
588
4.003
1.04
0.00
.666
2.005
0.00
0.00
LiRA
.634 + 4.003
4.70
0.98
743
+.006
9.56
2.79
.843 =0044
25.17
9.09
IHA (Ours)
.703 1.004
13.69
7.52
.791
2.003
19.95
18.99
IHA (CG)
.701 +.009
13.72
7.55
.791 1.005
20.09
19.09
.691 2.007
1.14
0.16
that would maximize performance. The switch from Purchase-100(S) Purchase-100 not only improves model performance but also reduces the performance of MIA attacks (TTable 7).
For instance, AUC values for LOSS drop from 0.59 to 0.53, and LiRA-Online from 0.74 to 0.63. While the smaller version of the dataset has recently been argued not to be very relevant Carlini et al. (2022), we believe this larger version is still interesting to study since such large datasets are practically relevant. We hope that rescarchers will aim to use the larger version of the dataset and, in general, train target models maximize performance (as any model trainer would) within the constraints of their experimental design.
D.1 IHA performance on MLP-4
For the MLP-4 architecture the Purchase-100(S) dataset, we observe significant drop in performance when using IHA compared to LiRA. We hypothesize that this performance degradation may be due mismatch between the damping factor used in our experiments (2c-1) and the optimal damping factor for the eigenvalue distribution of this larger model. To test this hypothesis, we increased e to 5e and repeated the ovaluation with IHA.
This adjustment led a substantial improvement performance: the AUC increased to 0.768, with TPRs of 13.11% and 12.12% at 1% and 0.1% FPR, respectively. This result supports our hypothesis and highlights current limitation of IHA. Specifically, the damping factor e should be scaled according to the model architecture ideally determined by some percentile of the eigenvalues. However, identifying the optimal scaling method before conducting the audit remains an open question for future work.
It is important to note that further increasing e does not result in a linear performance improvement; in fact, performance declines across architectures when e becomes too large, which is expected. The remaining performance gap is likely due to the absence of reference models or a violation of our assumptions about Hessian behavior. However, the superior TPR at very low (0.1%) FPRs, compared to LiRA, suggests that the former is more likely the cause.
E Implementing the Inverse Hessian Attack
For some given record 21, (o((w) can be computed by considering all data (except the target record) for which membership is known. To make this step computationally efficient for an audit, we pre-compute (ww. Then, if the test record is indoed a member, we can compute VLo(w) as (Li(w) v((w,=)) Note that this is equivalent to computing V Loww) separately for each larget record. The Hessian H. is also similarly pre- computed using the model's training data.
Conditioning H. While computing Hessian matrices for our experiments, we notice the presence of near- and small, negative eigenvalues (most of which are likely to arise from precision errors). Such eigenvalues
21

---

Published in Transactions on Machine Learning Research (12/2024)
make the Hessian ill-conditioned and thus cannot be inverted directly. We explore two different techniques to mitigate this: damping by adding small constant to all the eigenvalues or low-rank approximation where only eigenvalues (and corresponding eigenvectors) above a certain threshold are used as a low-rank approximation. We ablate over these two techniques for some candidate values c. Our results (Table 8) suggest that damping with e 2e works best across all the datasets we test, which the setting for which we report our main results.
Table 8: Attack AUCs for various techniques to mitigate ill-conditioned Hessian matrix, with corresponding values.
Ron-Rank = le
Damping <= Le
Dataset
== le"
6
2e
6 le" 2
= 2e -1
MNIST-Odd
.521
.530
.500
.513
.542 .535
FashionMNIST
.551
.537
.541
.533
.582
.594
F
Approximating Lo
For auditing purposes, experiments where all but one member known useful, but adversary is unlikely to have this much knowledge of the training data. We experiment with the potential use of IHA where only partial knowledge of the remaining - members may be available to approximate VL. Approximating Lo with a fraction of the actual dataset could be useful in not only reducing the computational cost of the andit, but also potentially enabling adversarial use of the attack in threat models where the attacker has partial knowledge of the training data. We evaluate IHA for versions where Lo approximated using a randomly-sample fraction of the training data and report results in Table
Table 9: Performance of approximation-oon based variant IHA on Purchase100-S (MLP-2), when a fraction of data from D { used approximate Lg. Statistics are computed on 10000 samples.
%TPR@FPR
Fraction
AUC
1%
0.1%
0.2
.577 +.005
0.85
0.11
0.4
.607 1.007
1.79
0.28
0.6
.638 +.010
3.38
0.86
0.8
.692 4.012
6.47
1.59
0.9
.733 +.010
13.53
4.17
1.0
.791 +.005
20.09
19.09
We see a clear degradation in performance when only a subset data is used-this is especially true for lower fractions, where AUC can drop by as much as 0.2. More importantly, even when using 90% of the training data, there is a significant gap in performance. The statistics we compute for IHA thus do completely utilize knowledge of all other training records. While this result suggests that adversarial use of IHA would require very strong adversary (that posseses knowledge of nearly all training records), it also hints at how data poisoning attacks could have large impact on downstream membership inference. A poisoning adversary could hypothetically craft data way that interferes with Lo (when relating to the optimal membership adversary) and thus accease//ecreee inference risk for other records.
22

---

Published in Transactions on Machine Learning Research (12/2024)
G
Additional Results
10
10"
10"1
10"
10
10
TPR
TPR
10
(Qurs)
IHA (Ours)
10"
LiRA
10
LiRA
LOSS
LOSS
SIF
SIF
10
10
10
10"
10
10"
10
10
10
10
10
10"
10"
10
FPR
FPR
(a) Purchase-109
(b) Purchase- 100(S)
10
10
10
10"1
TPR
TPR
10
10
IIAA (Our))
IIAA (Oun)
LOSS
LOSS
SIF
SIF
10
10a
10
10
10"
10"
10"
10"
10
10
10
10"
10
10
FPR
FPR
(c) MNIST-Odd
(d) FMNIST
Figure 1: ROC curves for low-FPR region for various attacks and datasets.
23


Reviews:
[{"review_id": "Id0Hz61Ds4", "reviewer": "Reviewer_heea", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\n[S1] The paper presents a novel idea in a very important topic in the context of ML. I believe the idea is worth being published.\n\n[S2] The experiments show that the method can outperform the baselines. The experiment section empirically manifests the claims that the authors put forward.\n\n[S3] The paper is well written and the ideas are clearly put.\n\nWeaknesses\n\n[W1] The hessian-based component of the method makes it difficult to be used on large models. However, can a simplified version of a large model be used to estimate membership inference?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors show that black-box access is sufficient for optimal membership inference but does not hold for stochastic gradient descent and that optimal membership inference requires white-box access. They also present an IHA method that outperms all other baselines in most experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Although the paper is well written, the structure of the paper makes it difficult to read. I would suggest making the following modifications:\n- Make the related work section shorter / differ it to the appendix. It makes the reading sloppy. \n- Bring section \"D Implementing the Inverse Hessian Attack\" to tbe body of the paper. Clearly, any hessian inverse computation is computationally expensive, and section D explicitly explains how to do it. The fact that it is computationally expensive does not mean this method should not be used, and the details on the computation should not be 'hidden' in the Appendix. This is is a personal opinion.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zhmkZbDbJA", "reviewer": "Reviewer_ycVN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths.**\n\n1. The new membership inference attack proposed in this work is well-motivated by theoretical analysis and the authors show many experimental results that justify its superior performance compared to previous methods.\n\n**Weaknesses.**\n1. The main high-level message of the paper is that white-box approaches have more potential than black-box approaches. However, the authors do not give sufficient justification for the novelty and significance of this message. In particular, in the experiments, the authors compare the proposed attack with many previous methods that also use the white-box approach. It is unclear whether these prior methods already outperformed black-box approaches. It is essential to discuss this to establish the novelty and significance of the main message.\n\n2. The presentation of the paper can be made more organized. For example, in the experiments section, it is hard for me to locate many important details about the attacks. It would be very helpful to include a table that answers the following questions. Which attacks require white-box access? Which attacks require reference models? Which attacks need to know other points in the training set? It is very hard to do a fair comparison without this information.\n\n3. The proposed attack requires access to the model parameters (as expected), but it also requires access to all the other points in the dataset (i.e., points other than the target point in the membership inference task). In addition, the attack requires expensive computation in order to calculate the inverse of the Hessian matrix (though the paper includes an attempt to alleviate this issue using approximation methods). This computational efficiency aspect is not sufficiently tested and compared with previous methods in the experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission introduces a new approach to membership inference, where the goal is to determine whether a given data point is included in the training set of a given model. This is very useful for privacy auditing and has been studied as a standard task in the literature. There are generally two types of membership inference methods (aka membership inference attacks): white-box attacks that require access to the model parameters, and black-box attacks that only need access to the model's output on certain data points. In previous work, there were theoretical arguments suggesting that black-box attacks can perform nearly as well as white-box attacks. This work aims to disprove this claim by introducing a white-box attack with better performance.\n\nThe proposed attack is inspired by a theoretical analysis of the probability distribution of the model parameters trained using SGD with momentum on a given set of data points. This analysis is largely based on results from prior work. While this analysis makes idealized assumptions, it gives a formula for computing a scoring function that can be used to develop an attack which can then be evaluated empirically. The authors test this attack on real-world datasets and demonstrate its superior performance compared to previous methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical.**\n1. Explain why the main claim about the potential of white-box approaches has not been established already in prior work.\n2. Make the experiments section more organized. See comments above.\n\n**Minor.**\n1. At the beginning of Sec 2.1, the authors define $\\alpha = \\Pr[m_i = 1]$, but later this notation switches to $\\gamma$, whereas $\\alpha$ is instead used to denote the regularization strength. It would be helpful to improve the notation consistency here.", "broader_impact_concerns": "The broader impact statement included in the submission seems adequate and satisfactory.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lfZxgiaSFK", "reviewer": "Reviewer_ofku", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. Novel theoretical results analyzing the posterior distribution of parameters, and derivation of an optimal MIA using them.\n2. Analysis of existing assumptions, and a well-written paper which specifies assumptions for the theorems\n\n**Weaknesses**\n\n 1. **Unclear how \"optimal\" the final measured IHA score is after approximations**: After deriving the optimal MIA in Theorem 3.2, the paper makes a wave of assumptions to get to the measured IHA score. Is the final metric really more optimal than the optimal black-box attack based on the results of Sablayrolles et al.?\n2. **More expensive than training reference models**: It seems this MIA is more expensive than training reference models, and requires access to more information (weights, gradients etc.). \n3.  **Little Empirical Improvement**: IHA shows significant improvement in MIA than black-box methods on only one dataset, which is also a subset of Purchase-100 (non-standard). The empirical advantage of using parameters for MIA over black-box is just not convincing.\n4. **Improvement could still be due to stronger threat model**: IHA requires knowledge of all remaining n-1 datapoints, which is a significant advantage of the compared black-box methods. The paper compares it to another method in the same threat model (L-attack), and uses the performance improvement over it to argue the parameter knowledge is what helps IHA. This argument is flawed unless L-attack is the optimal black-box attack in the leave-one-out setting. If it isn't, this control experiment does not prove the advantage of IHA over LiRA is not due to a stronger threat model, rather than having white-box access as claimed.\n\nOverall, I do not think the paper sufficiently defends its claim that white-box access leads to significant advantages for MIA, especially given the costs/risks of providing white-box access.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper argues that an earlier result showing the optimality of black-box membership inference attacks (MIA) is not applicable to real-world use of SGD. Drawing from assumptions of Liu et al. 2021 and their results on discrete-time SGD dynamics, the paper derives a theorem showing that the posterior distribution of parameters learnt by SGD depends on the hessian structure, gradient and weight distance from the local minimum. They use this to derive an \"optimal\" MIA score, under the assumption that Hessian structure, and loss achieved at the local minimum remains unaffected by removing a single training record. This requires access to the hessian, model gradients, and training parameters like learning rate, momentum and regularization. Computing this optimal MIA score is hard, so they propose a proxy scoring function called Inverse Hessian Attack (IHA), working in the setting where all data points except the record whose membership is to be inferred are known to the attacker. IHA is still expensive to compute as it requires inverting hessians, so empirical validation is performed using toy MLPs (< 100 neurons) on a small subset of Purchase-100,  MNIST-Odd, Fashion-MNIST. Results show that IHA performs slightly better than LiRA (state of the art) on the small subset of Purchase 100, and roughly equal on the other 2 datasets. An ablation study shows that only 1 term out of 4 in IHA is responsible for most of the effectiveness, and there might be potential for combining IHA with existing attacks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.  Experiments/Theory showing: a) What information is lost between Equation 7 and 8? b) How much information is lost when approximating iHVPs? c) Comparisons of IHA score with optimal black-box attack\n\n2. Could you please also provide compute and peak-memory-usage comparisons with existing attacks?\n\n3. Could you benchmark on more standard datasets? I understand that IHA is expensive to compute, but right now the empirical evidence for the central claim on the advantage of using parameters is just not clear.\n\n4. Could you design a better control to check whether the advantage is drawn from LOO or white-box access? For example, comparing to an optimal black-box LOO, or devising a variant of IHA which works outside the LOO setting? How far can a method not using the terms that require hessian/gradient information in the derived equation go?\n\n5. Could you provide a discussion on whether the improvement in MIA from parameter access are worth the costs of sharing white-box information for audits? \n\n6. Please tone down the claims made in the introduction. For example: “invalidates the results from Sablayrolles et al. (2019) which claimed black-box access is sufficient” - Does the paper prove contradictions to the results of Sablayrolles et al.? If not, saying the paper ‘invalidates’ them is wrong. It simply studies a different setting, where their results don’t hold.\n\n7. For the concerns you are not able to address, could you please write a clear limitations section mentioning what implications can and cannot be drawn from the paper?", "broader_impact_concerns": "The paper attempts to motivate a stronger threat model for evaluating privacy leakage, which is good. However, since there isn't sufficient evidence to back the claims, the paper could be misused to enable a potentially false-narrative, that white-box access is necessary for audits, leading to potentially unnecessary over-regulation, which can have costs in terms of achieving the positive impacts of innovation to society. A clear limitations section specifying what the paper doesn't prove would help alleviate this.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Some additional concerns still remain among reviewers, and it'd be great to address them in the final version.

* Making it more clear that empirically, the proposed white-box attacks also outperform previous white-box approaches (which are out-performed by black-box attacks), either by providing more experimental results or adding more comprehensive discussions.

* Improve the writing of introduction by making the first few paragraphs focusing on background shorter, and expanding the last two paragraphs which are more relevant to the contributions of this work. It would be helpful to motivate the problem from both theoretical (revisiting previous assumptions) and empirical aspects (proposing stronger attacks).
Decision: Accept (poster)