Curvature Clues: Decoding Deep Learning Privacy with Input Loss Curvature
Authors: Deepak Ravikumar
Efstathia Soufleri
Kaushik Roy
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Input Loss Curvature
Differential Privacy
Membership Inference
Primary area: privacy

Abstract: In this paper, we explore the properties of loss curvature with respect to input data in deep neural networks. Curvature of loss with respect to input (termed input loss curvature) is the trace of the Hessian of the loss with respect to the input. We investigate how input loss curvature varies between train and test sets, and its implications for train-test distinguishability. We develop a theoretical framework that derives an upper bound on the train-test distinguishability based on privacy and the size of the training set. This novel insight fuels the development of a new black box membership inference attack utilizing input loss curvature. We validate our theoretical findings through experiments in computer vision classification tasks, demonstrating that input loss curvature surpasses existing methods in membership inference effectiveness. Our analysis highlights how the performance of membership inference attack (MIA) methods varies with the size of the training set, showing that curvature-based MIA outperforms other methods on sufficiently large datasets. This condition is often met by real datasets, as demonstrated by our results on CIFAR10, CIFAR100, and ImageNet. These findings not only advance our understanding of deep neural network behavior but also improve the ability to test privacy-preserving techniques in machine learning.

Curvature Clues: Decoding Deep Learning Privacy with Input Loss Curvature
Deepak Ravikumar Efstathia Soufleri Kaushik Roy
Department of Electrical and Computer Engineering Purdue University
West Lafayette, IN 47907
(dravikum, esoufler, kaushik
Abstract
In this paper, we explore the properties of loss curvature with respect input data in deep neural networks. Curvature of loss with respect to input (termed input loss curvature) is the trace of the Hessian of the loss with respect the input. We investigate how input loss curvature varies between train and test sets, and its implications for train-test distinguishability. We develop theoretical framework that derives an upper bound on the train-test distinguishability based on privacy and the size of the training set. This novel insight fuels the development of new black box membership inference attack utilizing input loss curvature. We validate our theoretical findings through experiments computer vision classifica- tion tasks, demonstrating that input loss curvature surpasses cxisting methods in membership inference effectiveness. Our analysis highlights how the performance of membership inference attack (MIA) methods varies with the size of the training set, showing that curvature- based MIA outperforms other methods on sufficiently large datasets. This condition is often met by real datasets, as demonstrated by our results on CIFARIO. CIFAR 100, and ImageNet. These findings not only advance our understanding of deep neural network behavior but also improve the ability to test privacy-preservee techniques in machine learning
Introduction
Deep neural networks are being increasingly trained on sensitive datasets; thus ensuring the privacy of these models is paramount, Membership inference attacks (MIA) have become the standard approach test model's privacy [Murakonda and Shokri 2020]. These attacks take trained model and aim to identify if a given example was used in its training. Recent work has linked curvature of loss with respect to input with memorization [Garg al.
2024 and differential privacy [Dwork al.) 2006 Ravikumar et al. 2024). Inspired by this line of research, we investigate the properties of input loss curvature and leverage our insights develop a new membership inference attack.
Curvature of loss with respect to input (termed input loss curvature) is defined as the trace of the Hessian of loss with respect to the input (Moosavi-Dezfooli al. 2019 Garg and Roy 2023 Prior works that study loss curvature have focused on two lines of research. The first line of research has [Keskar et al. 2017 Wu et al. 2020 Jiang et al. 2020) Foret et al. 2021 Kwon et al.) [202]] Andriushchenko and Flammarion 2022 to better understand generalization. The second line of research studied the loss curvature with respect to the input (i.c. data) for gaining insight into adversarial robustness
focused on studying the loss curvature with respect the weights of the deep neural
'Code available at https:: //gith/// www./oo///aa//e PProject website with models and other assets available at at/attps://ennineeriigg pacpeeerrinn.uuuueeee wwwppe..duduueeeeee.e edu/NRI/ project projects/curvature. clues
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Low Curvature Training Examples From ImageNet
Test set examples higher curvature regions
More support training set
Loes
Input Space
Less support in training set
High Curvature Training Examples From ImageNat
Visualizing low and high input curvature samples from ResNet50 trained on ImageNet. Low input curvature training set images are prototypical and have lots support in the trainset, while high input curvature train set examples have less support and are atypical. Test set examples lie around the training set images in higher curvature regions.
Figure I:
Moosavi-Dezfooli al. 2019 Fawzi et al. 2018 coresets [Garg and Roy 2023 and memorization Gargg et al. 2024 Ravikumar et al. 2024 mainly focusing its properties on the train set. Thus, there is a gap in our understanding of input loss curvature on unseen test examples.
Input loss curvature on the train set captures the prototypicality of an image. This is visualized in F (left) which shows high curvature examples from ImagetNet (Russakovsky al.) 2015 train set. Figure [ (right) builds intuition to why this is the case sample with lots of support in the dataset more likely to lie low curvature regions while wwapical/non-proooyyyeo have less support in the training set and thus lie in higher curvature regions [Garg et al. 2024 Ravikumar et al. 2024). With this established, we make a novel observation that on average the input loss curvature scores for test examples are higher than train set examples. This is because test samples were not optimized for, hence they lie slightly off the flat minima, in regions of higher input curvature (also visualized in Figure I]. We leverage this insight and develop theoretical framework that focuses on the distinguishabllity of train-test input loss curvature scores. Studying this distinguishability leads us to understand performance bounds of membership inference attacks
examples
Shokri et al. 2017 Sablayrolles et al., 2019 Song and Mittal
2021 Carlini et al., 2022].
Our theoretical analysis obtains upper bound on the KI. divergence between train-test input loss curvature scores. This provides a ceiling for membership inference attack performance. The analysis also reveals that the upper bound when using input loss curvature scores is dependent of training set examples and the differential privacy parameter e. This insight helps us understand the conditions under which input loss curvature can be more effective at detecting train vs. test examples. Conditions such as the number of training examples and the privacy guarantees of the model.
the number
To test our theoretical results we perform black-box (setting where adversaries have access only to model outputs) membership inference attack (MIA) using input loss curvature scores. However, input loss curvature score calculation needs access to model parameters (which are not available in a black box setting). To resolve this issue we propose using a zero-order input loss curvature estimation. Zero-order estimation can calculate input loss curvature without needing access to model parameters. Our input loss curvature-based black-box membership inference attack results show that real datasets contain enough training examples for curvature scores to outperform current state- of-the-art membership inference methods. Our results show that curvature based MIA outperforms prior state-of-the-arr techniques on 10% or larger subsets of CIFAR100 dataset (about 5000 sample training set).
In summary, our contributions are as follows:
Theoretical Foundation: We provide theoretical analysis understand the train-test distin- guishability with input loss curvature scores, demonstrating that input loss curvature is more

---

sensitive and hence better at detecting train vs. test set examples than current state-of-ther ofrt membership inference attacks.
Adapting Theory To Practice: We propose using zero order input loss curvature estimation to enable black box membership inference attack using input loss curvature scores.
We conduct experiments to validate theoretical results. Specifically, we show that input loss curvature enables more effective black box membership inference attacks when compared existing state-of-the-art techniques.
Better Attack:
2 Related Work
Membership Inference Attacks are used as tool to test privacy (Murakonda and Shokri 2020).
These attacks aim to identify if particular data point was included in a model's training dataset
[Shokri al. 2.17]. Existing techniques often leverage model outputs like loss values (Shokri
et al. 201 Yeom et al. 2018 (abllayolless et al., 2019], confidence scores [Carlini et al. [022] or
modified entropy [Song and Mittal [021]] - (Shokri et al. 2017] proposed the used of shadow models
which are auxíliary models trained on subsets of the target model's data to aid in inference. Several modifications to this approach have been proposed. One important addition is the focus on example hardness, where the authors Sablayrolles et al. (0119) proposed an attack that scaled the loss using per example hardness threshold which is estimated by training shadow models. Watson et al. [202]
proposed similar approach but in the offline case where they calibrated the example hardness using
the average loss of shadow models not trained on the target example. Ye et al. [2022] models various
attacks into four categories and performs differential analysis to explain the gaps between them. Both
et al. [2020] consider the entire loss distribution of samples that are not in the training set. However, they face challenges in extrapolating to low false positive rates (FPRs). To address this issue Carlini et al.] [2022] propose using a parametric model along with shadow models to improve performance. Orthogonal to these approaches Choquette- et al. [2021] suggests the use of input augmentations during evaluation to improve the performance of the attacks. Similarly Jayaraman et al. [ propose the MERLIN attack, which queries the target model multiple times on a sample perturbed with Gaussian noise.
Ye et al. 2022)
Input Loss Curvature is defined as the trace of the Hessian of loss with respect to the input [Moosavi Dezfooli et al., 2019. Garg and Roy] 2023). The aim is to measure the sensitivity of the deep neural network to a specific input. In general, loss curvature with respect to the parameters of deep neural nets has received lots of attention (Keskar et al., 2017 Wu al.] 2020 Jiang et al., 2020 |020|Foret Foret et al., 2021 Kwon et al.] 2021 (Andrushchhenoo and Fla Flammarion 2022]. specifically due to its role in characterizing the sharpness of the learning objective which is closely connected to generalization. However, loss curvature with respect to the input data has received much less attention. It has been studied in the context of adversarial robustness [Fawzi et al. 2018 Moosavi-Dezfool) et al. 2019),
coresets (Garg and Roy 2023]. It has recently been linked with memorization [Garg al. 2024
Ravikumar et al. 2024 and privacy [Ravikumar et al., 2024). The authors in MMosavii-Dzzion
et al.
[2019] showed that adversarial training decreases the input loss curvature and provided thcoretical
link between robustness and curvature. In an orthogonal direction, Garg and Roy [2023) proposed the
use of low input loss eurvature examples as training dataset sets called coresets, which they showed to be more data-efficient. However, all of these works have focused on input loss curvature on the trainset. In this paper we focus on input loss curvature and its behavior test unseen examples to understand and improve the performance of membership inference attacks. Before we discuss our contributions, we present few preliminaries, notation, and background needed for this paper
3 Notation and Background
Notation. Let us consider a supervised learning problem, where the goal is to learn a mapping from an input space X C dd to an output space y C R. The learning is performed using randomized algorithm A on a training set S. Note that a randomized algorithm employs a degree of randomness as part of its logic. The training set S contains m elements denoted as 21,
2m. Each element 2; = (It, y:) is drawn from an unknown distribution D, where 2; E 2, x4 E X, y1 E y and 2 X' x V. Thus, define the training set S E 2n as S {21. , 2m }. Another related
concept is that of adjacent datasets, which are obtained when the set's 7th element is removed and
3

---

defined as
sW {21.- 1 2i-1, 2,+1," 1,
2m }.
Additionally, the concept of adjacent (or neighboring) datasets is linked to the distance between datasets. The distance between two datasets S and S", denoted by || SiIl1, measures the number of samples that differ between them. The notation |||]]| represents the size of dataset S. When a randomized learning algorithm A is applied to a dataset S, it produces a hypothesis denoted by ng = A(,, S), where é ~ & is the random variable associated with the algorithm's randomness. A cost function e : y y - R+ used to measure the hypothesis's 's performance. The cost of the hypothesis h at a sample 2; is also referred to as the loss l at 2, defined as E(h, 2;) = c(h(z;), yz). The performance of a hypothesis h is measured using risk R(h) = Ez, [e(h, 2;]] and approximated using empirical risk Remp(h) = (1/m) >x= {(h, 2,).
Differential Privacy. Introduced by Dwork et al. [2006), differential privacy is defined as follows: A randomized algorithm A with domain nd is considered 6-differentially private if for any subset R C (ange(A) and for any datasets S, E mm differing in at most one element (i.e. ||S S' S 1):
Pr[Mg E R] I cf Pr(h R.,,
where the probability is taken over the randomness of the algorithm A, with á ~ Φ
Error Stability of possibly randomized algorithm A for some B > 0 defined as Kearns and Ron 1997|
Vi E (1, m)) [eh= 2)] E =]] S 8,
where 2 ~~ D and @ ~ Φ.
Generalization. A randomized algorithm A is considered to generalize with confidence 8 and at rate of ((m) if:
Pemp (h. S) R(h) < 7/(m) 2 6.
Uniform Model Bias. The hypothesis h produced by algorithm A to learn the true conditional h" Ely|a] from a dataset Dmm has uniform bound on model bias denoted by A if:
AS D
Ep Ex(R(h)) R(hh)] S..
p-Lipschitz Hessian. The Hessian of is Lipschitz continuous on E, if V21, E 2, and vh RAnge(A), there exists some p >> such that:
V2 (1, V2, (h, 22)1 $ p||21 -211.
(5)
Input Loss Curvature. As defined by Dezsavi-Dezzooli et al. [2019]. Garg et al. [222]] input loss curvature is the sum of the eigenvalues of the Hessian H of the loss with respect to input 2,. This is conveniently expressed using the trace
Curv o S) = tr(H) = tr tr(v3, ((ig, 2))
(6)
e-adjacency. A dataset S is said to contain u-adjacent (read as upsilon-adjacent) elements if contains two elements 2,, Rj such that 2j = 2, + a for some a E B,(v) (read as v-Ball). This condition can be ensured through construction. Consider dataset S' which has no 2j s.t
2j = 2, + a; Zj,2 E S'. We can construct 5 such that S = {z 2 E 5") U {4 + a for some E S', (p))), thereby ensuring u-adjacency.
Membership Inference Threat Model. a membership inference security game Yeom et al. 2018 Jayaraman et al. 2021 Carlini et al.) 2022), challenger and an adversary interact to test the privacy of machine learning model. The process begins with the challenger sampling a training dataset from a distribution and training a model on this data. The challenger then flips coin to decide whether to select a fresh data point from the distribution, which is not part of the training set, or to choose a data point from the training set. This selected data point is given to the adversary, who has access to the same data distribution and the trained model. The adversary's 's task is to determine whether the given data point part of the training successful membership inference attack. Such a game is said to be a black-box setting when the adversary has access to only the challenger's 's output.
not. If the adversary's guess is correct, the game indicates

---

Marginal Curvature Score Distribution
Conditional Curvature Score Distribution
Train Examples
TTain
Test Examples
Density (Normlized Count)
Test
KDE Train
40000
2.0
KDE Test
Count
30000
1.5
20000
1.0
10000
0.5
25
-0
-15
0.5
1.0
2.0
log(Curve)
Curv
Figure 2: Marginal curvature score histogram for all images in ImageNet when samples train vs test set.
Figure 3: Conditional curvature scores for sin- gle image from ImageNet when using 56 shadow models.
4 Theoretical Analysis
In this section, we analyze the distinguishability of train-test samples using KL divergence. We do so for two cases, the first case when using network's 's output probability. and second when using input loss curvature. The importance of analyzing network output probabilities stems from its utilization in the current state-of-the-arr attack, LiRA [Carlini et al.) 2022]. Thus, studying both cases will let us theoretically analyze and compare their performance with input loss curvature.
Before we begin the analysis, we briefly discuss the conditional and marginal distributions of input loss curvature for train and test examples. Discussing this is important to build intuition for the analysis. Figure S visualizes the histogram (proxy for distribution) of input loss curvature for train and test examples from ImageNet [Russakovsky et al., 2015]. Specifically, it plots the log of the input loss curvature log Curv obtained on pre-trained ResNet50 (He et al. 2016) models from Feldman and Zhang [2020). It plots histogram of these scores, showing the frequency of specific log curvature values.
naive membership inference attack would apply threshold to separate these distributions. However, it is common practice consider sample specific scores (i.e., conditioned on the target sample). This visualized in Figure which plots the distribution of curvature scores for a single ImageNet sample, indicating differences when the sample part of the training set versus when is test or unseen sample. The data was generated using models from Feldman and Zhang 2020)] The figure also includes kernel density estimate (KDE, shown as a dashed line) better to visualize the underlying distribution. Figure 3 suggests that, similar to Carlini et al. (2022) sample conditional curvature scores can be modeled using a Gaussian parametric model. If we represent the curvature score Curv as a random variable, then Curv, ~ N ((,, o). The probability density function of Curv is function of the randomness of the algorithm Φ, the dataset S and the jth sample 2; and be denoted by Po(0, S, 2,). Similarly, let p(Ø, S, 2,) denote the probability density function of the neural network's output probability, which is also parameterized by the randomness of the algorithm Ø, the dataset S and the tth sample 2, With this setup we present the following theoretical results on the upper bound on the KL divergence between train and test distribution for the two cases. Theorem 4.1 presents the upper bound on the KI. divergence when using the neural network's 's output probability scores, and Theorem 4.2 presents the upper bound on the KI. divergence when using input loss curvature.
Theorem 4.1 (Privacy bounds Train-Tee KL Divergence). Assume e-differential private algorithm, then the divergence between train-test distributions of the neural network's output probability upper hound by the differential privacy parameter given by:
DKL (p(o, 8, 1| S't S'V ,2)) Se
Sketch of Proof. Given that the algorithm A is e-differentially private, we know that the probability of the output hg R on dataset S is bounded by et times the probability of the same event
5

---

on the neighboring dataset SV, i.e. Prolig E R e² Prolhs Sr E R/. From this inequality it follows that the KL divergence between the output distributions of A on S and S is bounded by (L(p(Ø, 5,2)) II p(ó, S' 21)) S The full proof for Theorem 4.1 is provided in Appendix A.3
Theorem 4.2 (Dataset Size and Privacy bound Curvature KL Divergence), Let the assumptions of generalization 3 and uniform model bias 3 hold. Further, assume 0 S E S L. Let the conditional distribution be parameterized with variance o. Then, any e-differential private satisfiex:
error stabiliry 2
algorithm using dataset of size n with a probability at least
[Lm - - + cc²
(p (1 1 ,/,,)))
(8)
20²
c=m 10m--11 (1m 2(m 1)A
Billa| 1
Sketch of Proof. We begin by assuming Gaussian model for the curvature distribution and expand the formula for the KL divergence, which leads to expressions involving the mean and standard deviation of the curvature scores. To establish an upper bound for this expression. we prove lemma concerning the upper bound of the mean curvature, utilizing results from Ravikumar et al.) (2024). This enables us to express the upper bound on the mean curvature in terms of the privacy parameter c. Subsequently, this result is employed to derive an upper bound on the KL divergence of the curvature scores. The proof for Theorem 4.2 provided in Appendix A.3
Discussion. Theorem 4. and Theorem provide the upper bound on KI. Divergence between train and test distributions of the probability and input loss curvature scores, respectively. They imply the upper limit of MIA performance, and interestingly, the result of Theorem [42 suggests the role of the dataset size on MIA performance. However, the relation not as straightforward as suggested by Equation 8 This is because, in real applications, parameter a depends on and so does the loss bound L. In fact, the loss bound is also dependent on the number of samples m [Bousquet and
Elisseeff 2002].
Theorem 4.3 (Dataset Size and Curvature MIA Performance). Let the assumptions of error stability 2 generalization 3 and uniform model bias holld Further, assume 0 < l S. L, and the bounds of T Theorem 4.1 and 4.2 are tight. Then, the performance of MIA using curvature scores exceeds that of confidence scores with a probability at least 1 - when:
m (V202e)
(10)
Theorem 4.3 suggests that the performance of curvature based MIA will exceed that of probability score based methods when the size of the dataset used to train the target model exceed a certain threshold. Indeed, this is what we observe in practice (see section 6.4).
The proof compares the upper bounds from Theorem 4.1 and Theorem 4.2 followed by series of algebraic manipulations. This identifies the conditions under which the performance of curvature scores exceeds that of confidence scores. The full proof can be found in Appendix(A.0)
Sketch of Proof.
On the validity of the assumptions. Before presenting our experiments to validate our theory, we briefly discuss the validity of our assumptions in practical settings. Research by Hardt al.) [2016) shows that stochastic gradient methods, such as stochastic gradient descent, achieve small generalization error and exhibit uniform stability. Thus, the assumptions of stability (Equation 2 and generalization (Equation 3) are justified. Model bias is property of the model, and a uniform bound across different datasets seems reasonable. Note, uniform loss bound (and L independent of m as used by Theorem 4.3) holds true for certain losses and for statistical models and is often assumed in learning theory [Wang et al. 2016). Lastly, the v-adjacency can be ensured through construction. For large datasets this may not be needed, because the size of the ball (v) is unconstrained. Hence two samples from the same class that are similar may suffice. Given the size of modern datasets, this assumption is also reasonable.
5 Zero Order Input Loss Curvature MIA
To test if input loss curvature based membership inference performs better than existing methods need an efficient technique to estimate curvature. We are interested in black-box membership

---

inference attacks, where one does not have access to the target network's 's parameters. However, current techniques use Hutchinson's 's trace estimator [Hutchinson 1989 to measure input loss curvature such as from Garg and Roy 2023]. Garg et al. [024 or Ravikumar et al. [2024]. This approach needs to evaluate the gradient and hence requires access to model parameters. To solve this issue, we propose using a zero-order estimation technique. Zero-order curvature estimation starts with
finite-difference estimation. Consider a function f g + IR, then the Hessian at a given point ž; can be estimated as follows:
f(z))) = n² 5 he + hu) f(2; hv + ha) - f(2; hv hu) + f(2; hw hu)
we
4h2
(11)
where h is small increment (a hyper parameter in out case), and u, are vectors in R In case, to get the input loss curvature, we have f + ((g(z;). y:), where is the neural network, l is the loss function and 2, = (æ U:) are the image, label pair. The pseudo-code for obtaining input loss curvature score using zero order estimation shown in Algorithm [ in Appendix |A.] To execute membership inference attack using input loss curvature scores, we propose the following methodology. First, we begin by training shadow models, similar to Shokri et al. (017)) Carlini et al. 22022). These shadow models are used to obtain empirical estimates of parametric model for the curvature score described in Section 4 During the inference phase, we employ likelihood ratio between the sample being in the train set vs test set parametric models identify the membership status of given sample (see Appendix AA. A..2 for pseudo-code). In addition, we perform negative log likelihood ratio test. We denote the results of likelihood test as 'LR' and the results of the negative log likelihood ratio test as 'NLL'.
6 Experiments
6.1 Experimental Setup
Datasets. To evaluate our theory, we consider the classification task using standard vision datasets. Specifically, we use the CIFARIO, CIFARI00 KKizhevsky et al. 2009 and ImageNet [Russakovsky al. 2015) datasets.
Architectures. For experiments on ImageNet we use the ResNet50 architecture [He et al. 2016). For CIFARIO and CIFAR100, used the ResNet18 8 architecture. Details regrading hyperparameters are provided in Appendix 13 To improve reproducibility, we have open sourced the code at https://g: httpbb//o///eepaaaaaaaaaaaaaa//a www.chhhaaaarraaauueeee.e -Cluue
Training. For experiments using private models, we trained ResNet18 models with the Opacus library YYousefpour et al. 2021 using DP-SGD with maximum gradient norm of 1.0 and privacy parameter of 0 = x 10 Shadow models for CIFARIO and CIFARI00 were trained on a 50% subset of the data for 300 epochs. For ImageNet, used pre-trained models from Fellman and Zhang 2020). trained on a 70% subset of ImageNet. More details about training and compute resources are provided in Appendix(A.13
Metrics. To evaluate curvature scores, we use AUROC and balanced accuracy. The Receiver Operating Characteristic (ROC) is the plot of the True Positive Rate (TPR) against the False Positive Rate (FPR). The area under the ROC called Area Under the Receiver Operating Characteristic (AUROC). AUROC of denotes an ideal detection scheme, since the ideal detection algorithm results in false positive and false negative samples.
6.2 Membership Inference
In this subsection, we compare the performance of the proposed input loss curvature based member- ship inference against prior MIA techniques.
Setup: We use CIFAR10, CIFAR100 and ImageNet datasets to test the MIA performance. We consider a black-box MIA setup similar to Carlini et al. [2022]. We use ResNet18 for CIFARIO and CIFAR 100, for ImageNet we use the ResNer50 architecture. For all the membership inference attacks, we compute a full ROC curve and report the results. When using shadow models we 64
for CIFAR10 and CIFAR 100 and 52 for ImageNet. The AUROC plot for CIFARIO0 100 for various
7

---

ImageNet
CIFARI00
CIFARIO
Method
Bal. Ace.
AUROC
Bal. Acc.
AUROC
Bal. Acc.
AUROC
Curv 0O NLL (Ours)
69.16
77.45
84.47
93.49
61.92
68.82
Curv 20 LR (Ours)
68.76 004
72.28 1 0.00
80.48
0.10
90.15 0.04
55.00 1: 1.117
58.89 0.38
Carlini et al.
66.14 001
73.46 = 0.22
81.55 0013
88.89 0.16
58.23 0.20
61.73 = 0.32
Yoom et al.
2018
58.50 002
63.23 = 0.0
76.29 0.30
82.11 0.31
55.57 = 0.52
60.44 = 0.75
Sablayrolles ct al.
2019
66,93
76.50 0u04
70.22
81.11 0.39
56.65 0.56
61.50 0.79
Watson 2022
61.40
69.44
62.71
71.66
54.86 +
58.58
et 2022
66.16 0.02
75.79 # 0.05
80.73
0.24
90.88 0.19
59.62 # 0.84
67.30 + 1.25
Song and Mittal
2021
57.88 0.08
63.29 1: 0.08
75.58
0.29
82.28 1: 0.27
55.63 1: 0.611
60.42 1: 0.85
Table 1: Comparison of the proposed curvature score based MIA with prior methods tested on ImageNet, CIFARI00, and CIFAR10 datasets. Results reported are the mean + std obtained over seeds. For CIFAR10 and CIFAR100 100 64 shadow models were used and 52 for ImagNet.
methods shown in Figure 4 Table T reports the average balanced accuracy and AUROC values over three seeds on various MIA methods on the three datasets. The plot in Figure 4 is a log-log plot to emphasize performance of the proposed method at very low false positive rates (see the orange line y-intercept). We also studied the effect of augmentations and the results can be found in Appendix A.7 the takeaway presented in Table | used 2 augmentations (image + mirror) for our technique Carlini et al. 2022 for fair comparison.
that adding more augmentations improved performance. Note that the results
From Table L we see that the proposed method performs better than all existing MIA techniques both ImageNet and CIFAR datasets. Apart from AUROC and balanced accuracy results, the log-log plot emphasizes the performance at really low FPR (i.e. the intercept, high TPR at low FPR in Figure 4 Additional results low FPR available in Appendix A.8
Results:
Takeaways: As predicted by Theorem 4.2 the higher KL divergence between train and test curvature score distributions results in superior MIA performance. Further, we observe that while using negative log likelihood ratio test does better in AUROC and balanced accuracy, the parametric likelihood ratio test does better at low false positive rates as shown in Figure 4 Thus, the proposed use of curvature should be tailored based on use case. Applications that demand high AUROC can use NLL approach, while those that demand high TPR at very low FPR should use the LR technique.
6.3 Effect of Privacy
In this section, we study the effect of differential privacy on MIA performance and test the relation predicted by Theorem 4.2
Setup: To study the effect of privacy on MIA attack performance, we use DP-SGD trained models. We use privacy e values of 2, 3, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 with 1 1 10 We
100
0.575
0.550
True Positive Rate
10-1
Cary ZO NLL
Cury
AUROC
0.525
10
Carlini
0.500
Yeom et al.
Sablayrolles et al.
0.475
10-3
Warson et al.
Ye al.
0.450
Best Fit
Song et
10-3
10-2
10)
100
0
20
40
False Positive Rate
Privacy Parameter e
Figure Comparing our method against existing techniques at low FPR. The proposed parametric Curv LR technique has the highest TPR at very low FPR.
Figure 5: Validating the upper bound from The- orem 4.2 by fitting the MIA performance (AU- ROC Curv LR) on DP-SGD trained models for various privacy parameters values.

---

1.0
1.00
MIA AUROC /Bal. Ace
0.9
0.95
MIA AUROC
0.8
0.7
0.90
Curv ZO NLL
0.6
Curv Zo LR
AUROC
0.5
Carlini et al.
0.85
Balanced Accuracy
Yeom et al.
0.4
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Fraction of dataset used for training
Fraction of dataset used for training
Figure 6: Visualizing MIA performance as a function of the size of the train set, which is randomly sampled.
Figure 7: MIA performance size of the train set when subsets contain the low- est curvature examples from the entire set.
function of the
plot the result of the proposed curvature based MIA in Figure 5 In Figure 5wee we also plots the best fit using s (L E c²)², where sy, Ly and ey are fit the data.
Results and Takeaways: We see that the theoretical prediction from Theorem 4.2 about MIA performance is well matched. Since Theorem 4.2 provides an upper bound, the results validate the theory.
6.4 Effect of Dataset Size
In this section, we study the effect of dataset size on MIA attack performance. This lets us test the relationship of MIA performance m as predicted by Theorem 4.2 and Theorem 4.3
Setup: For this experiment, we train models on increasing dataset size on CIFARI00. Specifically, train multiple seeds various subsets randomly chosen from the CIFARI00 training We also repeated the same by choosing the lowest curvature samples from CIFAR 100 and train with the same subset sizes. Next, for each of the models we perform MIA attack and we present the results in Figure/6l (randomly chosen) and Figure 7 (lowest curvature samples).
that when samples randomly chosen, the MIA performance decreases as we add more samples. This result is consistent with prior literature [Abadi et al., 2016).
Results: From Figure
Further as predicted by Theorem 4.3 beyond 30 40% subset Curv ZO LR out perform prior works and Curv ZO NLL outperforms prior works beyond 10% subset size.
To extend this analysis, we train models on curvature based coresets [Garg and Roy 2023 These coresets of low input loss curvature samples have been shown to memorize less [Garg al. 2024 Ravikumar al. 2024). Thus expect MIA accuracy increase is exactly what happens and shown Figure 7 which plots the AUROC and accuracy of the NLL curvature attack as we increase curvature coreset size. However, the MIA performance is higher for comparable size in Figure 6 and 7 This suggests that curvature based coresets result in more susceptible models, which is also supported by results in (Sng et al. 2019
increase coreset size. This
Takeaways: We validate Theorem 4.3 We note that beyond certain dataset size (of about 40% subset of the training set Curv ZO LR out perform prior works and Curv ZO NLL outperforms prior works beyond 10% subset size) curvature scores outperform probability score based MIA method. While Theorem 4.3 predicts that survature-based MIA outperforms other methods on sufficiently large datasets. This condition is often met by real datasets as evident from the results presented.
Conclusion
In this paper, we explored the properties of input loss curvature on the test set. Specifically, focus on using input loss curvature to distinguish between train and test examples. We established theoretical framework deriving upper bound on train-test KL Divergence based on privacy and

---

training set size. To extend the applicability of input loss curvature computation to black-box setting, we propose a novel zero-order curvature estimation method. This enables the development of a new cutting-edge black-box membership inference attack (MIA) methodology that leverages input loss curvature. Through extensive experiments on the ImageNet and CIFAR datasets, we demonstrate that our input loss curvature-based MIA method outperforms existing state-of-the-art techniques. Our results corroborate our theoretical predictions regarding the relationship between MIA performance and dataset size. Specifically, we show that beyond certain dataset size, the effectiveness of curvature scores surpasses other methods. Importantly, we observe that this dataset size condition is frequently met in practical scenarios, evidenced by results on the CIFARIOS dataset. These findings advance our understanding of input loss curvature in the context of privacy and help build more secure deep learning models.
Acknowledgment
This work was supported in part by, the Center for the Co-Design of Cognitive Systems (COCOSYS), a -ARPA--ponsored JUMP 2.0 center, the Semiconductor Research Corporation (SRC), and the National Science Foundation.
References
M. Abadi, A. Chu, Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308 318- 2016.
M. Andriushchenko and N. Flammarion. Towards understanding sharpness- aware minimization. In International Conference Machine Learning, pages 639-668. PMLR, 2022.
Ansel, E. Yang, He, Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burovski, G. Chauhan, A. Chourdia, W. Constable, A. Desmaison, Z. Detito, E. Ellison, W. Feng, J. Gong, M. Gschwind, B. Hirsh, S. Huang, K. Kalambarkar, L. Kirsch, M. Lazos, M. Lezcano, Y. Liang, J. Liang, Y. Lu, C. Luk, B. Maher, Y. Pan, C. Puhrsch, M. Reso, M. Saroufim, M. Siraichi, Suk, M. Suo, Tillet, E. Wang, X. Wang, W. Wen, Zhang, X. Zhao, K. Zhou, R. Zou, A. Mathews, G. Chanan, P. Wu, and S. Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume (ASPLOS '24). ACM, Apr. 2024. doi: 10.1145/3620665.3666 https://pytorchh org/ httg///asstts/pt/crr//2222
URL
O. Bousquet and A. Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526 526, 2002.
N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914. IEEE, 2022.
C. A. Choquette-Choo, F. Tramer, N. Carlini, and N. Papernot. Label-only membership inference attacks. In International conference on machine learning. pages 1964-1974. PMLR, 2021.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265-284, Springer, 2006.
A. Fawzi, S.-M. -ozsavi-Dezfooli, P. Frossard, and S. Soatto. Empirical study of the topology and geometry deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3762-3770, 2018.
10

---

V. Feldman and C. Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation, Advances in Neural Information Processing Systems, 33:2881-2891, 2020.
Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference Learning Representations, 2021. URL
net f oorrmmmmm - -eriiidd-6-m -
I. Garg and K. Roy. Samples with low loss curvature improve data efficiency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20290-20300, 2023,
I. Garg, D. Ravikumar, and K. Roy. Memorization through the lens of curvature of loss function around samples. In Forty-first International Conference Machine Learning, 2024. URL https: htttp://opperevieee nwt/ weet/forumiiddWOaae d=
M. Hardt, B. Recht, and Y. Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pages 1225-12344 PMLR, 2016.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
M. F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications Skatistics-Simllant and Computation, 18(3):1059-1076 1989.
B. Jayaraman, L. Wang, K. Knipmeyer, Q. Gu, and D. Evans. Revisiting membership inference under realistic assumptions. Proceedings on Privacy Enhancing Technologies, 2021(2), 2021.
Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and Bengio. Fantastic generalization measures and where to find them. In International Conference Learning Representations, 2020. URL
wwt. www/foorumiiieemm http://ormmmm--SlgPPPPPPBPB-P
M. Kearns and D. Ron. Algorithmic stability and sanity-check bounds for leave-one-out cross- validation. In Proceedings of the tenth annual conference on Computational learning theory, pages 152-162, 1997.
N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL htttps://operrev////
net. orum Mer-ddd-Hooo HoddiooHHogggg
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images, 2009.
J. Kwon, J. Kim, H. Park, and I. K. Choi. Asam: Adaptive sharpness-aware minimization for scale- invariant learning of deep neural networks. In International Conference on Machine Learning, pages 5905-5914, PMLR, 2021.
Y. Long, Wang, D. Bu, V. Bindschaedler, X. Wang, H. Tang, C. A. Gunter, and K. Chen. A pragmatic approach to membership inferences on machine learning models. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pages 521-534. IEEE, 2020.
H. Mehta, A. Thakurta, A. Kurakin, and A. Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2205. Xiix:2205.00973, 2022.
S.-M. Moosavi-Dezfooli,, A. Fawzi, Uesato, and Frossard. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9078 9086, 2019.
K. Murakonda and R. Shokri. MI privacy meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning. arXiv preprint (1XXi:2000700333, 2020.
Y. Nesterov and B. T. Polyak. Cubic regularization of newton method and its global performance. Mathematical Programming. 108(1):177-205, 2006.
D. Ravikumar, E. Soufleri, A. Hashemi, and K. Roy. Unveiling privacy, memorization, and input curvature links. Forty-first International Conference on Machine Learning, 2024. URL. https://openreviee net./Eorum?..--4t88//aaa

---

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recog- nition Challenge. International Journal of Computer Vision (IJCV), 115(3):111-55, 2015. doi:
A. Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. Jégou. White-box vs black-box box: Bayes optimal strategies for membership inference. In International Conference Machine Learning. pages 5558-5505 PMLR, 2019.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15555 2014.
L. Song and P. Mittal, Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21). pages 2615-2632, 2021.
L. Song, R. Shokri, and P. Mittal. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC Conference Computer and Communications Security, pages 241-257, 2019.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention all you need. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http:::://ppocceeediiee newrips cttuurips.ccppppppe. cc/paper.
Y.-.. Wang. J. Lei, and S. E. Fienberg. Learning with differential privacy: Stability, learnability and the sufficiency and necessity of erm principle. Journal of Machine Learning Research, 17(183): 1 1-40, 2016.
L. Watson, C. Guo, G. Cormode, and Sablayrolles. On the importance of difficulty calibration in membership inference attacks. In International Conference on Learning Representations, 2022. URL
D. Wu, S.-T. Xia, and Wang. Adversarial weight perturbation helps robust generalization, Advances in Neural Information Processing Systems, 33:2958-2966 2020.
J. Ye, A. Maddi, S. K. Murakonda, V. Bindschaedler, and R. Shokri. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093-3106, 2022.
S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. 2018 IEEE 3 Ist computer security foundations symposium (CSF), pages 268 282. IEEE, 2018.
A. Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, J. Nguyen, S. Ghosh, A. Bharadwaj. J. Zhao, G. Cormode, and I. Mironov. Opacus: User-friendly differential privacy library in PyTorch arXiv preprint arXiv:2109, 12298, 2021.
12

---

A Appendix
A.1 Zero Order Curvature Estimation
We present the pseudo-code for zero-order curvature score calculation below. Please note that Algorithm 1 shows the input loss curvature calculation for one example; however, it can be easily implemented for batch data.
Algorithm Pseudo-code for Zero Order Input Loss Curvature Estimation
Input: 2, = Us) (image, label data), & (loss functionn, Pster (hyperparameter: number of iterations), (hyperparameter: small increment), (neural network)
2: Output: cure_ cure_estimate (zero order input loss curvature estimate of 2;)
3: curv f- 0
4: for +- to Nater do Rademacher
5:
6:
u ~ Rademacher
7:
f ((g(2))) ys)
8:
V f(z) + Use Equation 11
9: curv
tr (r(.)(((.)))))
10: end for
curv_estimate t curv Niter
12: return curv _er__sstimatee
A.2 Curvature based MIA
In this section, we present the detailed step by step description and pseudo code the curvature based MIA (refer to Algorithm 2
1. Initialization (Lines 3-4): Two empty sets, curuin and cur are initialized. These sets will store the curvature scores for the shadow models trained with and without the target example ze.
2. Shadow Model Training (Lines 5-12): For a specified number of iterations N, the algorithm performs the following steps:
Sampling Shadow Dataset (Line 6): A shadow dataset Dattock is sampled from the data distribution DD
Training IN Shadow Model (Line 7): shadow model gh trained on the shadow dataset augmented with the target example, Dattack U {2/). The curvature score for this model, denoted as curei is computed using Algorithm [
Collecting IN Curvature Scores (Line 8): The curvature score for the training set Dattack U { of this model curet, is added to the set curtin.
Training OUT Shadow Model (Line 9): Another shadow model your is trained on the shadow dataset excluding the target example, Dattack \ {zz). The curvature score for this model, denoted as curra is also computed using Algorithm[1 [
Collecting OUT Curvature Scores (Line 10): The curvature score for the training set
Dattack ) of this model curv is added to the set curtout-
3. Model Parameter Estimation (Lines 13-16): After training the shadow models, the algorithm calculates the mean (Hins Pour) and variance (o in. aa of the collected curvature scores for both the 'IN' and 'OUT" models.
4. Target Model Query (Line 17): the target example 2, denoted as curutarget, is computed using Algorithm I
The curvature score for the target model Starget with respect
5. Likelihood Ratio Test (Line 18-19): The likelihood ratio P... is calculated. This ratio compares the probability of the observed curvature score under the 'IN' distribution against the 'OUT" distribution and is returned. For the 'NLL' version P.,. us given by:
PN log [P (curUtarget | [Hout, ( o.r)))] log [P (curttarget (Ppin (.))
(12)
13

---

Algorithm 2 Membership Inference Attack using Input Loss Curvature Scores
Require: Target model Stargets larget example 2; (2i Ui), data distribution D
1: curvin f {
2: curvout < [
3: for n in N do
4: Dattock <- D
Sample shadow dataset D Train in shadow model
5:
Din < Train on Dattack V { 2
6:
curvi < Use Algorithm
7:
curvmn < curvin U curv
8:
Pout f Train on Dattack
Train out shadow model
9: curvout Use Algorithm
10: curvout f curvout U curo out
11: end for
12: Pin < mean mean(curuaa)
13: Mout mean( curvout)
14: o² f- var(cu))
15: out var(cuuVout)
16: curetarget <- Use Algorithm with Starget and 2;.
P (curUtarget
[ pin))
17: Pm.
P (currUtarget (pout )
18: return Pmm.
A.3 Proof of Theorem 4.1
Consider an cDDP algorithm and S, SV such that ||S- | = that R = {h h(x;) y4 Since A is e-differentially private, then it follows from the definition of differential privacy in Equation T that
Next let R C (a))e(AA) such
Prihi E R] Se Pr E
(13)
plo. S. 2;) < ( 2;)
(14)
Since p(ò, 2,),) Pr Ihg R. and denotes the probability density function of the neural network's output probability. Now, we use the definition of KL divergence
S,0,
DKL (p(e. 2)) p(o Si z.) = Es log
p(0, SV,,
E. [loge"]
From Equation
DKL pp(, S,2) (Vo. Se
A.4 Lemma A.1
Lemma A.1 (Upper bound on Input Loss Curvature). Let A be a randomized algorithm which
is d-differentially private and the assumptions of error stability 2 generalization 3 and uniform
model bias / hold. Further, assume 0 S l S L. Then for two adjacent datasets S,S 'D with a
probability at least 1 - 8 we have:
Eş|Curvo (V,,SV S < L(m(1 - +1
(15)
(4m- 1)y 2(m 1)4
l 6
E|||
(16)
Proof of Lemma A.1 For this proof, we use results from Nesterov and Polyak (0066). we restate
here for convenience
Lemma A.2. If Lipschitz assumption on the Hessian of holds from Nesterov and Polyak (0006)
we have
[e(h, 21) E(h, 22) (7((h,2), -2) - ( 22) 22), - 22) < 0
3
(17)
14

---

From Lemma A.2 have
G -21 s ((h,22) 21) - - 22) - (Ve(h, 22), 21 22) - (VV - -22),21 22)
(h,,21) {(h, 22) (Vl(h, 22), 22) (v2e(h, (2) 22), =2) g 22 |
This gives us a lower bound on ((h, =1)
1 11-zy 2212 + e(h, (Ve(h, 22): 21 22) + (2)((1)22)(11 -2)) 21 22) (h),21)
21)
(18)
Consider 2j E S such that 2; + a for some j * where a E By(w) such that E(a]
0
and Bia T a| = 1. Using the lower bound in Lemma A.2 from Ravikumar et al. [0024] with
21 24, 22 2j get
Esle(h sur 23)] Egl((hg. x.]l S m8 + (4m 1)y + 2(m 1)A
(19)
Using the result from Lemma
A.2
L Ilaa| & + Esle(h s 2 4 Es V
sir &),
+ E Sur x)a, at} E, ((hh =:]] mB (4m 1)7 + +(m 1)A (20)
Taking Expectation over and since the mean of we have
Eple(h 2;]] - Ea, ( (Ve(h gyr =), at) + E., (V E(h & g 2i a)]
E2 E 5: 2,] S m3 + (4m 1)y + 2(m 1) + Glall
Note that we can change the order of expectation using Fubini's theorem
Egle(he sic. 2:) + + (V 2)a, - x)] < mB + (4m 1)7 + 2(m - 1)A + - ||ll
Egle(h S 2 + R. |t V 2)) ( 2; S mB + (4m 1)y + 2(m 1) A ||a!* 3
}] S mB + (4m 1)7 + 2(m 1)A-
0
B||la|| |8 + E. (e(h 2:]] - Eppe(hh 2/)
6
Ee(tr(V x))] < mB + (4m - 1)7 + 2(m - 1)4 + f
B[|la|]*] L
6
From Ravikumar et
2024 Lemma 5.2 have 8 < L(1 e e). Thus we have the upper bound:
Eu(Cv S < [mm(1 -) 1) + c1
(21)
C1 (4m 1)7 2(m 1) g EB|L||19
0
(22)
A.5 Proof of Theorem 4.2
This proof uses results from Lemma A. provided above. For the proof of Theorem 4.2 let (=(ó, (,, S, 2i) be the curvature probability mass function, then using the Gaussian model discussed in Section 4 we model it as a Gaussian. Thus, we can write
(Ø 22
(e( 5,2: =
1
20
(23)
( - Mgrt) )2
1
20 3
Pe(
e
(24)
T /2T
15

---

Now consider the (pL 2,2 |l Po SV,2))
(po(pe S, 2:) (l)lllleel po SV, ,2)) = Eø log Po(d,, 21)
(25)
Pol0, Si ,1,,2))
(0 - us))
(0 - Mg
Es
Assume ggy.
20²
20
(26)
Φ² Mg + 20ps + 6² + 1 201
Es
(27)
20²
Eg
8/
P
(28)
20
| S\
1
(29)
20
To upper bound Equation 29 we need an upper bound on May and lower bound on ps. The lower bound on 2
For ease of notation let's define Sman == 9 + 1). Notice that the upper bound of per
sample (Ag Es (2, Using the result from Lemma A.1 in Equation 29 we have
(Bmaz + c2)² 02
DRI(Pe S, 2)) (o(,, SV,, s1), s
(30)
20²
(Bmaz + c1)²
(31)
20
[L(m(1 e + + ca
(32)
20²
A.6 Proof of Theorem 4.3
Assuming the bounds of Theorem 4.1 and Theorem 4.2 are tight, then using curvature score will outperform neural network probability scores if:
S,, po(o, s 2)) (KL(P(0, S, 2:) pl
(33)
[L(m(1 -c [[(m(1---)) 1) cal
>6
202
[Lm(1 -e c)2
> :0= c +L
20
( - 9mmm + - 70mmmmmm---- + 6² 20 20" >0
Using the quadratic formula have the roots given by:
= -2Lc(1- - - 4L' c)3 4L2 1 c 20"e)
+ VAL262 - (c2 - 2
212 e-
= Lot - 20
(
c 122 - (62 20
L(1 -e-))
-ct V20eE
-(1-
16

---

V20º L(1 E
V20 - Equation L - -
Assuming both roots real, the larger root is
Thus when
satisfied.
A.7 Input Augmentations Ablation
In this section we present the results of using augmentation (only the image) Vs 2 augmentations (image + mirror) in Tables 2 3 and 4 for ImageNet, CIFAR 100 and CIFARIO respectively. For these results, we used 64 shadow models for CIFARIO and CIFAR100 and 52 shadow models for ImageNet.
ImageNet
Method
Augmentation
2 Augmentations
Bal. Acc.
AUROC
Bal. Acc.
AUROC
Curv ZO NLL (Ours)
68.09 +: 0.07
75.84 + 0.08
69. 0.08
77.45 + 0.09
Curv ZO LR (Ours)
67.59 + 0,05
70.62
0.04
68.76 0.01
72.28 0.04
Carlini et al. 2022
65.11 0.03
71.96 0.04
66.14 0.01
73.46
Table 2: Comparison of the proposed curvature score-based MIA with prior methods on ImageNet dataset with and augmentations. Results reported are the mean + std obtained over seeds.
CIFAR 100
Method
Augmentation
2 Augmentations
Bal. Acc.
AUROC
Bal. Acc.
AUROC
Curv ZO NLL (Ours)
83.97
92.72
84.47
93.49
Curv Z0 LR (Ours)
77.88 + 0.21
88.10 = 0.12
80.48
90.15
Carlini et al. 2022
80.73 + 0.16
87.69 + 0.18
81.55 0.13
88.89 = 0.16
Table Comparison the proposed curvature score-based MIA with prior methods on CIFAR100 dataset with and 2 augmentations, Results reported are the mean + std obtained over 3 seeds.
CIFARIO
Method
I Augmentation
2 Augmentations
Bal. Acc.
AUROC
Bal. Acc.
AUROC
Curv NLL (Ours)
61.03 +: 0.84
67.56 +: 1.23
61.92 +
68.82 ::2 1.30
Cury 70 LR (Ours)
54.54 +: 0.19
58.06 +: 0.1
55.00 0.17
58.89 +:2 0.38
Carlini et al. 2022
56.96 + 0.27
60.05 0.34
58.23
0.29
61.73 0.32
Table 4: Comparison of the proposed curvature score-based MIA with prior methods on CIFARIO dataset with and augmentations. Results reported are the mean + std obtained over seeds.
A.8 Low FPR Results
In this section, we present the performance of curvature based MIA compared with prior works using TPR (true positive rate) at very low FPR percentages. The results are presented when using I augmentation (only the image) and augmentations (image + mirror) in Table and Figure For these results, we use 64 shadow models for all the results in 3 (for methods that use shadow models). The results for Carlini al.) [2022) are little lower than the one reported by Carlini et al.) [2022) in Table of their paper. However, note that the authors of Carlini et al.) [2022] report the results with 256 shadow models while we report using 64 shadow models. This highlights benefits of curvature based approach, it achieves similar performance to Carlini et al. [2022) with 256 shadow models with only 64 shadow models thus needs significantly less shadow models to achieve similar performance.
and
17

---

The parametric curvature LR method has the best TPR at very low FPR and makes efficient use of shadow models.
Takeaways:
Augmentation
Augmentations
Method
TPR @ 0.1% FPR
TPR 0.01% FPR
TPR 0.1% FPR
TPR @ 0.01% FPR
Curv ZO LR (Ours)
21.07 = 0,80
15.74 1 M1
23.92 :: 0.92
17.17 1.87
Curv 00 NLL (Ours)
6.42
0.05
8.29
0.10
Carlini et al.
2022
15,52 = 0.54
5.02 1.70
15.80 0.49
6.56 1.13
Sablayrolles
2019
10.50 1: 0.59
4,30 0.33
0.39
4.30 0.3
Watson ct al.
2022
6.62 0.33
2.95 0.18
6.62 0.23
2.95
Ye al.
2022
6,63 0.52
2,86 1 0.42
6.63 0.52
2,86 1: 0.42
Song Mittal
2021
1.21
1.21
1.21
1.21
Yeom et al.
2018
0.07 0.01
0.01 0.00
0.07 0.01
0.01 0.00
Table 5: Comparison of TPR @ 0.1% FPR and TPR @ 0.01% FPR for CIFAR100 dataset with and 2 augmentations. Results reported are the mean + std obtained over 3 seeds and using 64 shadow models.
Comparing ZO estimation performance
Curv ZO NLL
10P
MI 0.1% FPR
Cury 70 IRR
40
Carlini al.
True Positive Rate
10-1
Yeom et al.
10-2
Carv Z0 NLL
10
Curv ZO LR
Curv NLL
Carv LR
0.0
0.2
0.4
0.6
0.8
10
Fraction of datasct used for training
10
10
10-2
10
10P
False Positive Rate
Figure 8: TPR at low FPR visualizing MIA performance function of the size of the train set, which is randomly sampled.
Figure Low TPR performance comparison on CIFAR 100 of ZO estimation (Curv ZO, black- box) vs. Hutchinson (Cury, white-box).
A.9 ZO Estimation Performance
In this section, we present the results of comparing the proposed zero-order curvature estimation with Hutchinson's 's trace estimator-bais based curvature calculation. It is important to note that the trace estimator proposed by Garg et al. 2024] requires access to the model's parameters, thus it is white-box attack. In contrast, the proposed zero-order (70) estimation operates black-box attack.
The comparative results are presented in Table Additionally, Figure
9 illustrates the True Positive Rate (TPR) vs. False Positive Rate (FPR) for the ZO estimation and Hutchinson's trace estimator, utilizing a log-log plot to emphasize performance at low FPR.
Takeaways. From Table we see that zero-order approach to curvature estimation results in notable differences in performance. Specifically, when examining AUROC and balanced accuracy, particularly on the CIFAR- 10 dataset, gap of several percentage points can be observed between the two methods. From Figure 2 we observe that the proposed LR technique performs similarly between Hutchinson's 's trace estimator [Garg et al. 2024] and the proposed Z0 estimation. However, the NLL method exhibits much greater sensitivity to estimation errors, as demonstrated by significant performance drop at low FPR levels.
18

---

Dataset
Method
AUROC
Balanced Accuracy
Curv ZO NLL
93.49 + 0.18
84.47 + 0.21
Curv NLL
95.01 + 0.19
85.66 + 0.30
CIFARI00
Curv ZO LR
90.15 + 0.04
80.48 + 0.10
Curv LR
92.87 + 0.02
83.75 + 0.16
Curv ZO NLL
68.82 + 1.30
61.92 + 0.87
Curv NLL
74.20 + 1.42
64.96 + 1.05
CIFARIO
Curv ZO LR
58.89 + 0.38
55.00 + 0.17
Curv LR
68.53 + 0.27
61.82 + 0.11
Table 6: Performance Metrics for CIFARIO0 and CIFARI0 of zero order estimation (Cury ZO,
black-box) vs. Hutchinson trace estimation (Cury, white-box) [Garg et al. 2024
A.10 Performance Across Architectures
In this section, we present the results of evaluating the proposed zero-order MIA on different neural network architectures. We selected four architectures: Inception (Szegedy al. 20155 (specifically, the smaller Inception architecture used by Feldman and Zhang [2020]) VGGII [Simonyan and Zisserman 2014), VGGII with batch normalization (denoted as VGGIIBN), and finally, a ViT-BI6 vision transformer [Vaswani al.) 2017 Dosovitskiy al.) 2021]. The results are provided in Table Table Please note that for all of the results we used ResNet18 shadow models and 2 augmentations.
Takeaways. The proposed techniques outperform the previous state-of-the-ur method by Carlini al. [2022] across all architectures. Additionally, the lower performance on the Vision Transformer (ViT) model can be attributed to transfer learning. Since ViT's do not perform well when trained from scratch on CIFAR-100. we used an ImageNet-pretrainee model, which was fine-tuned on CIFAR-100.
This led to significant privacy benefits, which aligns with the findings Mehta et al. [2022]
Arch.
Method
AUROC
Bal. Acc.
TPR 1% FPR
Curv ZO NLL
87.77 0.61
79.04 0.65
15.33 1.38
Inception
Curv LR
81.94 0.79
73.46 =
19.46
Carlini et al. 2022
83.77 0.63
76.70 = 0.66
0.82
Curv 70 NLI.
73.68 2.99
66.53 + 3.35
6.19 + 1.15
VGGI1
Curv ZO LR
68.29 1.38
64.30 = 1.13
8.14 = 1.85
Carlini et al. 2022
62.85 0.71
61.08 = 0.87
2.07 = 0.19
Cury ZO NLL
87.76 0.17
79.12 + 0.11
15.56 1.34
VGGIIBN
Curv ZO LR
76.17 + 0.65
70.98 = 0.26
22.80 0.40
Carlini et al. 2022
70.57 + 0.39
67.13 = 0.43
3.22 + 0.34
Curv NLL
58.10 0.20
55.40 - 0.18
1.60 + 0.01
ViT-B16
Curv LR
53.98 0.30
52.55 - 0.18
1.91 + 0.25
Carlini et al. 2022
54.59 0.05
53.22 = 0.10
1.44 - 0.14
Table 7: MIA performance metrics for different architectures and methods on CIFAR 100 dataset.
19

---

A.11 Broader Impact
The development and analysis of input loss curvature in deep neural networks presented in this work have significant implications for both the academic and practical fields of machine learning and privacy. By introducing novel black-box membership inference attack that leverages input loss curvature, this research advances the state-of-the-un in privacy testing for machine learning models.
From a societal perspective, the ability to detect and mitigate membership inference attacks is crucial for maintaining user privacy in machine learning applications. This work helps pave the way for more robust unauthorized inference. Furthermore, the insights gained from the relationship between input loss curvature and memorization can guide the development of more secure machine learning models, which is particularly important as these models are increasingly deployed in sensitive domains such as healthcare, finance, and personal data processing.
techniques, ensuring that sensitive information is protected against
Additionally, this research highlights the potential of using subsets of training data as defense mechanism against membership inference attacks. By identifying that training on certain sized subsets can improve resistance to these attacks, this work offers practical guidance for model training practices that can enhance privacy without significantly compromising performance.
A.12 Limitations
In this paper we presented the a theoretical analysis and experimental evidence for improved MIA using input loss curvature. The theoretical and empirical evidence also shows that certain sized subsets of the training set may provide defense against membership inference attacks. However, as mentioned in the paper, the MIA performance improvement occurs only above certain training dataset size below which the non-parametric model of 'Curv NLL' works better and is not explained by the theoretical analysis. Similar to techniques that use shadow models such by /Shokri et al. (2017). Carlini et al. (2022) we need to train shadow models, which can be computationally expensive. Further, the method described requires more queries at minimum 4x more than Carlini et al. [2022] We believe these limitations can be addressed by follow up research.
A.13 Reproducibility Details
In this section we present additional details for reproducing our results. Additionally we have released our code here htttps://githuut com/ TheppaKTatachar//rraaaaa
Clues
Training. For experiments that use private models, we use the Opacus library YYuueefoorr et al. 2021] to train ResNet18 8 models for 20 epochs till the privacy budget is reached. We use DP-SGD Abadi et al. 2016) with the maximum gradient norm set 1.0 and privacy parameter d = 1 10-5 The initial learning rate was set to 0.001. The learning rate is decreased by 10 at epochs 12 and 16 with batch size of 128.
For shadow model training on CIFAR1O and CIFAR100 we trained on 50% randomly sampled subset of the data for 300 epochs with batch size of 512 for CIFAR100 and 256 for CIFAR10. We used SGD optimizer with the initial learning rate set to 0.1, weight decay of x 10 and momenutm of 0.9. The learning rate was decayed by 0.1 at 180 and 240"h epoch. For ImageNet pre-trained models from Feldman and Zhang 2020 as shadow models which were trained on a 70% subset of ImageNet. For both CIFAR TO and CIFAR 100 datasets, we used the following sequence of data augmentations for training: resize (32 x 32), random crop, and random horizontal flip, this is followed by normalization.
used
Testing. During testing we used resize followed by normalization. We used two augmentations the original image and its mirror. The number of augmentations used are specified in the corresponding experiment section. When using pre-trained models from Feldman and Zhang 2020) we validated the accuracy of the models before performing experiments.
Compute Resources. All of the experiments were performed on heterogeneous compute cluster consisting of 9 1080Ti's, 6 2080TT's and 4 A40 NVIDIA GPUs, with a total of 100 CPU cores and a combined 1.2 TB of main system memory. However, the results can be replicated with a single GPU with 11GB of VRAM.
20

---

Hyperparameters. Our code uses 2 hyper parameters for zero-order input loss curvature estimation. The nater and Algorithm ! We used nste 10 and h 0.001. To improve reproducibility, we have provided the code in the supplementary material.
A.14 Licenses for Assets Used
For each of the assets used we present the licenses below also have provided the correct citation in the main paper as well as here for convenience.
1. ImageNet [Russkkosky et al. org/dowmlodd... php
2015
Terms of access available at hthttpss//immage
2. CIFARIO [Krizhessky et al.
2009
Unknown / No license provided
3. CIFARI00 Krizhevsky et al. 2009 Unknown No license provided
4. Pre-trained ImageNet models and code: We used pre-trained ImageNet models from
Feldman and Zhang (2020) which is licensed under Apache 2.0 https://githbb. com/
researe-reeeacchhhie www.rreeeeeeecceeetottttt-e---ee innuutt-illluneeeeeeee luence-
5. Baseline methods:
We re-implemented the baseline methods hence provided with along with our code which is distributed under the MIT License.
6. Opacus Yousefpour et al. 2021] Licensed under Apache 2.0,
com/
pytorchhopacuss//iiii blob/ http///aaannnn////.///
7. Pytorch (Ansel al. 2024 Custom BSD-style license available at pettps//githnn
8. ResNet Model Architecture [He 2016] MIT license available at btttps://githb
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract clearly outlines the exploration of input loss curvature in deep neural networks and the development of a theoretical framework for train-test distinguisha- bility. introduces novel black-box membership inference attack (MIA) utilizing input loss curvature and the use of subsets of training data as a mechanism to defend against shadow model based MIA. These claims are substantiated through theoretical analysis and experimental validation, presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the work in Appendix A.12 where we note the increased computation requirement imposed by zero-order curvature calculation and other limitations of the proposed methods.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
22

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All the assumptions are stated the Theorems and discussed prior to their utilization in the main paper. The complete proof has also been provided in the Appendix of the paper, specifically Appendix A.3 A.5 and A.6
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We include all the training details for the models in Appendix A.13 We also provide the pseudo-code for the proposed technique in Appendix A.2 along with all the hyper parameter details (see Appendix 13). Further, we also provide the code in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
23

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets used publicly available and are the standard datasets used for vision tasks. Our code is included in the supplementary material and will be released publicly after the conference deadline. The code replicates the results of our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cr
Beallic/guides/Coo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
swi.iilicoyyycccooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training details reported in Section 6. the paper and more details are provided in Appendix A.13]
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All reported results are obtained by averaging over seeds and reported with mean and standard deviation values.
24

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: For the experiments, we used a heterogeneous cluster whose details are mentioned in Appendix A.13
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'ttt make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The paper conforms to the code of ethics, we discus potential impacts of this research paper in Appendix A.11
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
25

---

Answer: [Yes]
Justification: We provide a detailed discussion about the broader impact of the research presented in the paper in Appendix A.I1
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data models thus this paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We provide the detailed list assets used and the terms under which they were licensed in Appendix A.14 We have respected the licensing terms of the original owners.
Guidelines:
The answer NA means that the paper does not use existing assets.
26

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessitthheeeocc thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pase re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the code for reproducing the results under the MIT license and details of its usage are documented in Appendix A.13 and the supplementary material README file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
27

---

Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "GNUa9h7aPK", "reviewer": "Reviewer_Vex3", "summary": "The paper explores the use of loss curvature, under the hypothesis that test samples lie in high curvature regions, to improve the performance of membership inference attacks. It estimated the curvature by measuring the trace of the Hessian of the loss values. It demonstrates the viability of the approach in both white-box and black-box settings. It empirically validates the success of the approach on multiple datasets.", "questions": "Why not use zero-order loss estimation in black-box method as it would be significantly more sample efficient than block-box curvature estimation. \n\nWould using the first-order statistics, gradients of the loss w.r.t input, be as helpful as curvature in MIA attacks?", "limitations": "The paper proposes using second order statistics (loss curvature) over using loss values in membership attacks. However it will fail to generalize to networks that are piecewise linear, thus having zero curvature for all samples. A simple example of it would be the resnet models with convolution layer, ReLU activation, and Batchnorm normalization. While the paper does consider the resnet models, it doesn’t clarify the architecture uses. In the piecewise linear architecture, only the cross-entropy loss on output logits would contribute to loss curvature.\n\nThus it’s critical to ablate the choice of network architecture and demonstrate that despite the diversity in architectural designs, the approach succeeds in MIA attacks. A few simple variations would be changing the normalization layers, activation functions, and broadly the class of models (CNNs vs transformers).", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a simple but effective improvement over the canonical baseline of using loss estimation on train and test samples for membership inference attacks. It tests its effectiveness on three datasets and compares it with key baselines in the previous works. It also validates its effectiveness against differentially private models that provide provable privacy guarantees to a certain degree.", "weaknesses": "While the loss value based MIA evaluation generalizes to pretty much all class of models, it’s unclear if loss curvature based MIA attack would enable a similar generalization, i.e., going beyond the small-scale classification setup to self-supervised models, contrastive models, and other foundation models [1]. \n\nAs fig. 6 itself highlights, the robustness of attack against a simple defense that subsamples the training data for shadow models is very low. Just using ½ of the training data renders the attack ineffective (albeit Carlini et al. suffers from the same limitation). Since shadow models are only a proxy of the true black-box model with training data information not publicly available, in practice they will likely only be trained on a subset of original training data of the block-box model, thus significantly reducing the attack success in black-box threat model. \n\n1. Niu, Jun, Xiaoyan Zhu, Moxuan Zeng, Ge Zhang, Qingyang Zhao, Chunhui Huang, Yangming Zhang et al. \"SoK: Comparing Different Membership Inference Attacks with a Comprehensive Benchmark.\" arXiv preprint arXiv:2307.06123 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, we address the weaknesses and questions below:\n\n1.\tWhile the loss value based MIA evaluation generalizes to pretty much all class of models, it’s unclear if loss curvature based MIA attack would enable a similar generalization, i.e., going beyond the small-scale classification setup to self-supervised models, contrastive models, and other foundation models [1]. \n\nA:  The proposed curvature score is generic and only requires a valid loss function to measure model performance. Therefore, it can theoretically be applied to any machine learning model. The only limitation is with auto-regressive models, where defining input loss curvature is non-trivial. To test the generalizability of the method, we evaluated it across additional architectures. The results are presented in Table 3 in the rebuttal PDF. We appreciate the reviewer's input and will revise the paper to include these additional results.\n\n2.\tAs fig. 6 itself highlights, the robustness of attack against a simple defense that subsamples the training data for shadow models is very low. Just using ½ of the training data renders the attack ineffective (albeit Carlini et al. suffers from the same limitation). Since shadow models are only a proxy of the true black-box model with training data information not publicly available, in practice they will likely only be trained on a subset of original training data of the block-box model, thus significantly reducing the attack success in black-box threat model. \n\nA:    Further investigation and code audit revealed that the performance degradation at 50% dataset size in Figure 6 was caused by a bug in the experiment. This one-line bug (see the diff in the rebuttal pdf) resulted in under-representing the membership inference attack (MIA) performance of our method and that of Carlini et al. After fixing this bug, the updated graph for Figure 6 in the rebuttal PDF (Figures 1 and 2 in the global response pdf) shows that the curvature-based method performs better than before. Specifically, as predicted by Theorem 4.3 beyond 20 − 30% (Figure 2 and 1 in the rebuttal pdf) subset Curv ZO LR matches or outperforms prior works in AUROC (better in TPR at low FPR) and Curv ZO NLL (Figure 1 in the rebuttal pdf) outperforms prior works beyond 10% subset size (compared to 40% previously). Both our method and Carlini et al.'s do not show a performance drop at 50% dataset size. We thank the reviewer for their input and will revise the paper to include these additional results.\n\n3.\tWhy not use zero-order loss estimation in black-box method as it would be significantly more sample efficient than block-box curvature estimation.\n\nA:   This is indeed possible; however, our empirical results suggest curvature outperforms loss-based attack. We use the true loss (not estimated) for LOSS attack [as described Carlini et al.], which performs poorer than LIRA [Carilli et al] and our method (supported by Table 1 and Figure 4 in the main paper, and Table 2 in the rebuttal pdf). The main reason being loss-based MIA is non-parametric. Non-parametric MIA methods struggle at low FPR performance (also shown by Carlini et al.). Since the loss distribution is non-gaussian Carlini et al. suggest logit scaling to build parametric MIA i.e. LiRA. Our experimental results show that we outperform both of these methods (Carlini et al. and LOSS).\n\n4.\tWould using the first-order statistics, gradients of the loss w.r.t input, be as helpful as curvature in MIA attacks?\n\nA:  Indeed, first and zero-order statistics can be useful. To evaluate this, we compare these metrics to the memorization score from Feldman and Zhang, which is an independent metric of memorization in deep models. We find that curvature aligns best with memorization at the end of training and thus outperforms zero and first-order (i.e., loss and loss gradient) metrics. See the Table 4 in the rebuttal pdf. Table 4 shows the cosine similarity of Feldman Zhang (FZ) memorization scores (independent metric) with the different order statistics, for all the samples in CIFAR100 and top 5000 most memorized samples in CIFAR100 (according to Feldman and Zhang). Further, the loss-based attack [LOSS from Carlini et al.] is provided as reference in Table 2 (rebuttal pdf) which fails to outperform curvature. We thank the reviewer for their input and will revise the paper to include these additional results.\n\n5.\tHowever .... Thus it’s critical to ablate the choice of network architecture and demonstrate that despite the diversity in architectural designs, the approach succeeds in MIA attacks. A few simple variations would be changing the normalization layers, activation functions, and broadly the class of models (CNNs vs transformers).\n\nA: We consider the second derivative of the loss with respect to the input. Curvature estimation involves sampling the loss landscape at specific points to determine curvature. This method assumes that the function is smooth. If the function is piecewise linear, the zero-order (zo) method estimates the curvature of the smoothest function that intersects all sampled points, thereby smoothing the piecewise linear function. This approach is validated by our membership inference attack (MIA) results across various architectures, including ResNet18 with batch normalization and ReLU (as detailed in the main paper), and additional results for VGG11, VGG11_BN, Inception, and Vision Transformers (ViT-B16) as presented in the rebuttal document. The results demonstrate that zo curvature outperforms prior methods (see Table 3 in rebuttal PDF).\n\nFeldman, V. and Zhang, C. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881–2891, 2020.\n\nN. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UFtWbITwox", "reviewer": "Reviewer_mnxr", "summary": "This paper develop a theoretical framework to derive an upper bound on train-test distinguishability based on input loss curvature. The authors propose a new black-box membership inference attack utilizing input loss curvature, which surpasses existing methods in membership inference effectiveness. The paper also explores the potential of using subsets of training data as a defense mechanism against shadow model-based membership inference attacks.", "questions": "What is the effect of the error in the zeroth-order curvature estimation? How the estimated curvature deviates Gaussian and can we still model the distribution with it?", "limitations": "The authors have addressed the limitations of their work by acknowledging that their analysis only holds when the dataset size is large enough. They emphasize the need for further exploration of the method's applicability to different models and datasets and the potential impact of computational complexity.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Membership inference attack is an important and interesting problem and this paper discussed it from a theoretical perspective. The attacking method based on theoretical guarantees makes the method more solid. In the experiment, the curvature based attack proposed by this paper also exceeds other existing methods.", "weaknesses": "1. In the theoretical analysis, the authors propose the train-test distinguishability distribution upper bound for both data and curvature. However, they compare the upper bounds directly and suggest that curvature based MIA is a better method when the size of dataset is large enough. The comparison between upper bounds may not reflect the real relationship. To validate this result, one may compare between an upper bound and a lower bound.\n2. The assumption that the curvature score can be modeled with Gaussian distribution has not been discussed and validated in the paper, which is a very important assumtion in theoretical analysis. Figure 3 seems that the distribution is not very similar to Gaussian.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, we address the weaknesses and questions below:\n\n1.\tIn the theoretical analysis, the authors propose the train-test distinguishability distribution upper bound for both data and curvature. However, they compare the upper bounds directly and suggest that curvature based MIA is a better method when the size of dataset is large enough. The comparison between upper bounds may not reflect the real relationship. To validate this result, one may compare between an upper bound and a lower bound.\n\nA: We agree with the reviewer that comparing the upper bound with a lower bound would be a more elegant approach. However, to account for the fact the we compare upper bounds, we provide empirical evidence to validate this result in Figure 6 (please see the updated plot in the rebuttal pdf, specifically Figures 1 and 2). Additionally, please see the global response bullet 2 and the pdf with revised dataset size vs performance plots validating the theoretical result. In summary, we use empirical results to bolster and check the validity of the theoretical result. The results show we perform better than other methods and as predicted by theory.\n\n2. How the estimated curvature deviates Gaussian and can we still model the distribution with it?\n\nA: The validity of the Gaussian assumption is difficult to assess using a single plot, such as Figure 3 in the main paper, which depicts one image from ImageNet. It was provided as a visual aid for the reader. Parametric MIA methods demonstrate the 'goodness' of parametric estimation in the low FPR region, as shown by Carlini et al. Specifically, accurate parametric estimation results in good low FPR performance [Carlini et al]. Our results in Figure 4 and Table 1 empirically validate that the Gaussian assumption is a good parametric estimate. That is we see the Curv LR method which is the parametric model outperforms the Curv NLL method and others in low FPR performance.\n\n3. What is the effect of the error in the zeroth-order curvature estimation? \n\nA: We provide additional results on the effect of zero-order estimation. We run the proposed MIA attack on CIFAR100 and CIFAR10 datasets with zero order (ZO) estimation (black-box) and compare it against white-box Hutchinson curvature calculation [Garg et al.]. The results are provided in Table 1 and Figure 3 in the rebuttal pdf. Please note this comparison is a black-box and white-box comparison hence is provided to illustrate the effect of the error in the zeroth-order curvature estimation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J6tr8f4BNw", "reviewer": "Reviewer_1uCf", "summary": "This paper builds the connection between privacy and input loss curvature.  Core observation is that test samples often lie in higher curvature areas than those prototypical samples, so the input loss curvature can be a good metric to distinguish between train and test sets. Utilizing this observation, authors build a stronger black-box MIA, outperforming previous MIAs, and they also reveal a previously unknown limitation of shadow model based MIAs. The paper provides both theoretical analysis and experimental results.", "questions": "Q1: What is $s_f$, $L_f$ and $c_f$ in Line 300? The notation seems to be missing from the context.\n\nQ2: How accurate is the zeroth-order estimation of input loss curvature? Can you provide a comparison of this estimation to the real values   (assuming we have white-box access to target model)", "limitations": "The limitations of this work is properly discussed in Appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The introduction of input loss curvature into the area of privacy attacks are original and novel. Previous works have revealed the utility of input loss curvature on coreset selection, adversarial robustness and model memorization, this work has further broadened its application. \n\nS2. This work proposed an advanced black-box MIA, which is more effective than previous attacks. Moreover, the study reveals one limitation of shadow-model based MIAs: when the dataset size used to train shadow models is similar to dataset size used to train target model, shadow-model based MIAs perform poorly.", "weaknesses": "Line 117 missing a `.`", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, we address the weaknesses and questions below:\n\n1.\tWhat is $s_f$, $L_f$ and $c_f$ in Line 300? The notation seems to be missing from the context. Line 117 missing a .\n\nA: $s_f$, $L_f$ and $c_f$ are the fitting coefficients for equation 8, we will clarify this in the revised version of the paper and include missing punctuation in line 117.\n\n2. How accurate is the zeroth-order estimation of input loss curvature? Can you provide a comparison of this estimation to the real values (assuming we have white-box access to target model)\n\nA: We provide additional results on the effect of zero-order estimation. We run the proposed MIA attack on CIFAR100 and CIFAR10 datasets with zero order (ZO) estimation (black-box) and compare it against white-box Hutchinson curvature calculation [Garg et al.]. The results are provided in Table 1 and Figure 3 in the rebuttal pdf (see global response). Please note this comparison is a black-box (ours) vs white-box (Garg et al) comparison hence is provided for a complete analysis. Additionally, please see our global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vgq3bHIAo4", "reviewer": "Reviewer_2dr3", "summary": "This paper conducts a theoretical and empirical investigation of the use of input loss curvatures in MIAs. They study the train-test input loss curvature scores and use this to develop a zero order input loss curvature black-box MIA.", "questions": "1. Other authors have used the Hutchinson trace estimator to estimate information about input loss curvature. How does the performance of this method compare to the zero-order method presented in the paper?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides an interesting theoretical explanation of MIAs based on input loss curvature that have performed well empirically but lacks theoretical understanding. \n2. The new MIA is simple and motivated by theory, and the evaluation of it is thorough and clearly shows that this method is more performant that existing baselines. \n3. The paper is well-written and clear.", "weaknesses": "1. Missing some references, e.g. Li et al. 2023 or the original DetectGPT paper (Mitchell et al. 2023).\n2. I believe the paper would benefit from including some sort of proof sketch of the theorems in the main body.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, we address the weaknesses and questions below:\n\n1.\tMissing some references, e.g. Li et al. 2023 or the original DetectGPT paper (Mitchell et al. 2023).\n\nA: We thank the reviewer for the feedback, we will include these references in the revised version of the paper.\n\n2.\tI believe the paper would benefit from including some sort of proof sketch of the theorems in the main body.\n\nA: We thank the reviewer for the feedback, we will include a brief sketch of proof in the main body of the paper in the revised version of the paper.\n\n3.\tOther authors have used the Hutchinson trace estimator to estimate information about input loss curvature. How does the performance of this method compare to the zero-order method presented in the paper?\n\nA: We provide additional results on the effect of zero-order estimation. We run the proposed MIA attack on CIFAR100 and CIFAR10 datasets with zero order (ZO) estimation (black-box) and compare it against white-box Hutchinson curvature calculation [Garg et al.]. The results are provided in Table 1 and Figure 3 in the rebuttal pdf (please see the global response). Please note this comparison is a black-box (ours) vs white-box (Hutchinson) comparison hence is provided for a complete analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their input, there were a few common questions and weaknesses that we discuss and address below

1.	Performance of zero-order curvature estimation vs curvature.

A: We provide additional results on the effect of zero-order estimation. We run the proposed MIA attack on CIFAR100 and CIFAR10 datasets with zero order (ZO) estimation (black-box) and compare it against white-box Hutchinson curvature calculation [Garg et al.]. The results are provided in Table 1 and Figure 3 in the rebuttal pdf. Please note this comparison is a black-box (ZO) vs white-box (Hutchinson) comparison hence is provided only to illustrate the effect of ZO estimation.

2.	Improved MIA performance.

A: Further investigation and a code audit revealed that the performance degradation at the 50% dataset size in Figure 6 was due to a minor bug (see diff in Figure 4) in the experiment. This issue resulted in under-representing the membership inference attack (MIA) performance of our method and that of Carlini et al. This under-representation was in Figure 6 only, tables and other figures are unaffected. After correcting this bug, the updated graph for Figure 6 in the rebuttal PDF (Figures 1 and 2 in the global response PDF) shows that the curvature-based method performs significantly better than before. Specifically, as predicted by Theorem 4.3, beyond 20-30% (Figure 2 and 1 in the rebuttal pdf) subset, Curv ZO LR matches or outperforms prior works in AUROC and exhibits better TPR at low FPR, while Curv ZO NLL (Figure 1 in the rebuttal pdf) outperforms prior works beyond 10% subset size (compared to 40% previously). Neither our method nor Carlini et al.'s shows a performance drop at the 50% dataset size. We thank the reviewers for their input and will revise the paper to include and correctly represent these results, i.e. remove this specific weakness described in the paper.

3.	Performance on different architectures

A: We validated our membership inference attack (MIA) results across various architectures, including ResNet18 with batch normalization and ReLU (in the main paper), and additional results for VGG11 (without batch norm), VGG11_BN (with batch norm), Inception, and Vision Transformers (ViT-B16) as presented in the rebuttal pdf. The results demonstrate that ZO curvature outperforms prior methods (see Table 3 in rebuttal PDF) and is consistent with the takeaways in section 6.2 of the paper “Applications that demand high AUROC can use NLL approach, while those that demand high TPR at very low FPR should use the LR technique”

N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE, 2022.

I. Garg, D. Ravikumar, and K. Roy. Memorization through the lens of curvature of loss function around samples. arXiv preprint arXiv:2307.05831, 2023

Meta-review: This papers studies how the second-order derivative of the loss w.r.t. the input data vector is revealing information about whether the data was used for training.  

Reviewers generally like the work how it contributes to membership inference attack,  privacy auditing as well as that the paper provided both a theoretical analysis and experiments on a variety of settings.

All reviewers are supportive of the paper and I am happy to recommend accepting the paper to neurips.
Decision: Accept (spotlight)