Curvature Clues: Decoding Deep Learning Privacy with Input Loss Curvature

Authors: deepak  ravikumar
efstathia  soufleri
kaushik  roy
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 21:51:44

Abstract: In this paper, we explore the properties of loss curvature with respect to input data in deep neural networks. Curvature of loss with respect to input (termed input loss curvature) is the trace of the Hessian of the loss with respect to the input. We investigate how input loss curvature varies between train and test sets, and its implications for train-test distinguishability. We develop a theoretical framework that derives an upper bound on the train-test distinguishability based on privacy and the size of the training set. This novel insight fuels the development of a new black box membership inference attack utilizing input loss curvature. We validate our theoretical findings through experiments in computer vision classification tasks, demonstrating that input loss curvature surpasses existing methods in membership inference effectiveness. Our analysis highlights how the performance of membership inference attack (MIA) methods varies with the size of the training set, showing that curvature-based MIA outperforms other methods on sufficiently large datasets. This condition is often met by real datasets, as demonstrated by our results on CIFAR10, CIFAR100, and ImageNet. These findings not only advance our understanding of deep neural network behavior but also improve the ability to test privacy-preserving techniques in machine learning.

Main paper: 
 introduction: 
 Deep neural networks are being increasingly trained on sensitive datasets; thus ensuring the privacy of these models is paramount. Membership inference attacks (MIA) have become the standard approach to test a model's privacy [Murakonda and Shokri, 2020] . These attacks take a trained model and aim to identify if a given example was used in its training. Recent work has linked curvature of loss with respect to input with memorization [Garg et al., 2024] and differential privacy [Dwork et al., 2006 , Ravikumar et al., 2024] . Inspired by this line of research, we investigate the properties of input loss curvature and leverage our insights to develop a new membership inference attack.Curvature of loss with respect to input (termed input loss curvature) is defined as the trace of the Hessian of loss with respect to the input [Moosavi-Dezfooli et al., 2019, Garg and Roy, 2023] . Prior works that study loss curvature have focused on two lines of research. The first line of research has focused on studying the loss curvature with respect to the weights of the deep neural net [Keskar et al., 2017 , Wu et al., 2020 , Jiang et al., 2020 , Foret et al., 2021 , Kwon et al., 2021 , Andriushchenko and Flammarion, 2022] to better understand generalization. The second line of research studied the loss curvature with respect to the input (i.e. data) for gaining insight into adversarial robustness [ Moosavi-Dezfooli et al., 2019 , Fawzi et al., 2018] , coresets [Garg and Roy, 2023] and memorization [Garg et al., 2024 , Ravikumar et al., 2024] , mainly focusing on its properties on the train set. Thus, there is a gap in our understanding of input loss curvature on unseen test examples.Input loss curvature on the train set captures the prototypicality of an image. This is visualized in Figure 1 (left) which shows high curvature examples from ImagetNet [Russakovsky et al., 2015] train set. Figure 1 (right) builds intuition as to why this is the case -a sample with lots of support in the dataset is more likely to lie in low curvature regions while atypical/non-prototypical examples have less support in the training set and thus lie in higher curvature regions [Garg et al., 2024 , Ravikumar et al., 2024] . With this established, we make a novel observation that on average the input loss curvature scores for test set examples are higher than train set examples. This is because test samples were not optimized for, hence they lie slightly off the flat minima, in regions of higher input curvature (also visualized in Figure 1 ). We leverage this insight and develop a theoretical framework that focuses on the distinguishability of train-test input loss curvature scores. Studying this distinguishability leads us to understand performance bounds of membership inference attacks [Shokri et al., 2017 , Sablayrolles et al., 2019 , Song and Mittal, 2021 , Carlini et al., 2022] .Our theoretical analysis obtains an upper bound on the KL divergence between train-test input loss curvature scores. This provides a ceiling for membership inference attack performance. The analysis also reveals that the upper bound when using input loss curvature scores is dependent on the number of training set examples and the differential privacy parameter ϵ. This insight helps us understand the conditions under which input loss curvature can be more effective at detecting train vs. test examples. Conditions such as the number of training examples and the privacy guarantees of the model.To test our theoretical results we perform black-box (setting where adversaries have access only to model outputs) membership inference attack (MIA) using input loss curvature scores. However, input loss curvature score calculation needs access to model parameters (which are not available in a black box setting). To resolve this issue we propose using a zero-order input loss curvature estimation. Zero-order estimation can calculate input loss curvature without needing access to model parameters. Our input loss curvature-based black-box membership inference attack results show that real datasets contain enough training examples for curvature scores to outperform current stateof-the-art membership inference methods. Our results show that curvature based MIA outperforms prior state-of-the-art techniques on 10% or larger subsets of CIFAR100 dataset (about 5000 sample training set).In summary, our contributions are as follows:• Theoretical Foundation: We provide theoretical analysis to understand the train-test distinguishability with input loss curvature scores, demonstrating that input loss curvature is more sensitive and hence better at detecting train vs. test set examples than current state-of-the-art membership inference attacks.• Adapting Theory To Practice: We propose using zero order input loss curvature estimation to enable black box membership inference attack using input loss curvature scores.• Better Attack: We conduct experiments to validate our theoretical results. Specifically, we show that input loss curvature enables more effective black box membership inference attacks when compared to existing state-of-the-art techniques. 
related work: 
 Membership Inference Attacks are used as a tool to test privacy [Murakonda and Shokri, 2020] . These attacks aim to identify if a particular data point was included in a model's training dataset [Shokri et al., 2017] . Existing techniques often leverage model outputs like loss values [Shokri et al., 2017 , Yeom et al., 2018 , Sablayrolles et al., 2019] , confidence scores [Carlini et al., 2022] or modified entropy [Song and Mittal, 2021] . Shokri et al. [2017] proposed the used of shadow models which are auxiliary models trained on subsets of the target model's data to aid in inference. Several modifications to this approach have been proposed. One important addition is the focus on example hardness, where the authors Sablayrolles et al. [2019] proposed an attack that scaled the loss using per example hardness threshold which is estimated by training shadow models. Watson et al. [2022] proposed a similar approach but in the offline case where they calibrated the example hardness using the average loss of shadow models not trained on the target example. Ye et al. [2022] models various attacks into four categories and performs differential analysis to explain the gaps between them. Both Ye et al. [2022] and Long et al. [2020] consider the entire loss distribution of samples that are not in the training set. However, they face challenges in extrapolating to low false positive rates (FPRs). To address this issue Carlini et al. [2022] propose using a parametric model along with shadow models to improve performance. Orthogonal to these approaches Choquette-Choo et al. [2021] suggests the use of input augmentations during evaluation to improve the performance of the attacks. Similarly Jayaraman et al. [2021] propose the MERLIN attack, which queries the target model multiple times on a sample perturbed with Gaussian noise.Input Loss Curvature is defined as the trace of the Hessian of loss with respect to the input [Moosavi-Dezfooli et al., 2019, Garg and Roy, 2023] . The aim is to measure the sensitivity of the deep neural network to a specific input. In general, loss curvature with respect to the parameters of deep neural nets has received lots of attention [Keskar et al., 2017 , Wu et al., 2020 , Jiang et al., 2020 , Foret et al., 2021 , Kwon et al., 2021 , Andriushchenko and Flammarion, 2022] , specifically due to its role in characterizing the sharpness of the learning objective which is closely connected to generalization. However, loss curvature with respect to the input data has received much less attention. It has been studied in the context of adversarial robustness [Fawzi et al., 2018 , Moosavi-Dezfooli et al., 2019] , coresets [Garg and Roy, 2023] . It has recently been linked with memorization [Garg et al., 2024 , Ravikumar et al., 2024] and privacy [Ravikumar et al., 2024] . The authors in Moosavi-Dezfooli et al. [2019] showed that adversarial training decreases the input loss curvature and provided a theoretical link between robustness and curvature. In an orthogonal direction, Garg and Roy [2023] proposed the use of low input loss curvature examples as training dataset sets called coresets, which they showed to be more data-efficient. However, all of these works have focused on input loss curvature on the trainset. In this paper we focus on input loss curvature and its behavior on test or unseen examples to understand and improve the performance of membership inference attacks. Before we discuss our contributions, we present a few preliminaries, notation, and background needed for this paperQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of the work in Appendix A.12, where we note the increased computation requirement imposed by zero-order curvature calculation and other limitations of the proposed methods. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
notation and background: 
 Notation. Let us consider a supervised learning problem, where the goal is to learn a mapping from an input space X ⊂ R d to an output space Y ⊂ R. The learning is performed using a randomized algorithm A on a training set S. Note that a randomized algorithm employs a degree of randomness as a part of its logic. The training set S contains m elements denoted asz 1 , • • • , z m . Each element z i = (x i , y i ) is drawn from an unknown distribution D, where z i ∈ Z, x i ∈ X , y i ∈ Y and Z = X × Y. Thus, we define the training set S ∈ Z m as S = {z 1 , • • • , z m }.Another related concept is that of adjacent datasets, which are obtained when the set's i th element is removed and defined asS \i = {z 1 , • • • , z i-1 , z i+1 , • • • , z m }.Additionally, the concept of adjacent (or neighboring) datasets is linked to the distance between datasets. The distance between two datasets S and S ′ , denoted by ∥S -S ′ ∥ 1 , measures the number of samples that differ between them. The notation ∥S∥ 1 represents the size of a dataset S. When a randomized learning algorithm A is applied to a dataset S, it produces a hypothesis denoted by h ϕ S = A(ϕ, S), where ϕ ∼ Φ is the random variable associated with the algorithm's randomness. A cost function c : Y × Y → R + is used to measure the hypothesis's performance. The cost of the hypothesis h at a sample z i is also referred to as the loss ℓ at z i , defined as ℓ(h, z i ) = c(h(x i ), y i ). The performance of a hypothesis h is measured using risk R(h) = Ez i [ℓ(h, z i )] and approximated using empirical risk R emp (h) = (1/m) m i=1 ℓ(h, z i ). Differential Privacy. Introduced by Dwork et al. [2006] , differential privacy is defined as follows: A randomized algorithm A with domain Z m is considered ϵ-differentially private if for any subset R ⊂ Range(A) and for any datasets S, S ′ ∈ Z m differing in at most one element (i.e. ||S -EQUATIONwhere the probability is taken over the randomness of the algorithm A, with ϕ ∼ Φ.Error Stability of a possibly randomized algorithm A for some β > 0 is defined as Kearns and Ron [1997] :EQUATIONwhere z ∼ D and ϕ ∼ Φ.Generalization. A randomized algorithm A is considered to generalize with confidence δ and at a rate of γ ′ (m) if:EQUATIONUniform Model Bias. The hypothesis h produced by algorithm A to learn the true conditional h * = E[y|x] from a dataset S ∼ D m has uniform bound on model bias denoted by ∆ if:EQUATIONρ-Lipschitz Hessian. The Hessian of ℓ is Lipschitz continuous on Z, if ∀z 1 , z 2 ∈ Z, and ∀h ∈ Range(A), there exists some ρ > 0 such that:EQUATIONInput Loss Curvature. As defined by Moosavi-Dezfooli et al. [2019] , Garg et al. [2024] , input loss curvature is the sum of the eigenvalues of the Hessian H of the loss with respect to input z i . This is conveniently expressed using the trace as:EQUATIONυ-adjacency. A dataset S is said to contain υ-adjacent (read as upsilon-adjacent) elements if it contains two elements z i , z j such that z j = z i + α for some α ∈ B p (υ) (read as υ-Ball). This condition can be ensured through construction. Consider a dataset S ′ which has no z j s.tz j = z i + α; z j , z i ∈ S ′ . We can construct S such that S = {z | z ∈ S ′ } ∪ {z i + α} for some z i ∈ S ′ , α ∈ B p (υ), thereby ensuring υ-adjacency.Membership Inference Threat Model. In a membership inference security game [Yeom et al., 2018 , Jayaraman et al., 2021 , Carlini et al., 2022] , a challenger and an adversary interact to test the privacy of a machine learning model. The process begins with the challenger sampling a training dataset from a distribution and training a model on this data. The challenger then flips a coin to decide whether to select a fresh data point from the distribution, which is not part of the training set, or to choose a data point from the training set. This selected data point is given to the adversary, who has access to the same data distribution and the trained model. The adversary's task is to determine whether the given data point was part of the training set or not. If the adversary's guess is correct, the game indicates a successful membership inference attack. Such a game is said to be in a black-box setting when the adversary has access to only the challenger's output. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: All the assumptions are stated in the Theorems and discussed prior to their utilization in the main paper. The complete proof has also been provided in the Appendix of the paper, specifically Appendix A. 3, A.5 and A.6 . Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
theoretical analysis: 
 In this section, we analyze the distinguishability of train-test samples using KL divergence. We do so for two cases, the first case when using network's output probability, and second when using input loss curvature. The importance of analyzing network output probabilities stems from its utilization in the current state-of-the-art attack, LiRA [Carlini et al., 2022] . Thus, studying both cases will let us theoretically analyze and compare their performance with input loss curvature.Before we begin the analysis, we briefly discuss the conditional and marginal distributions of input loss curvature for train and test examples. Discussing this is important to build intuition for the analysis. Figure 2 visualizes the histogram (proxy for distribution) of input loss curvature for train and test examples from ImageNet [Russakovsky et al., 2015] . Specifically, it plots the log of the input loss curvature log(Curv ϕ ) obtained on pre-trained ResNet50 [He et al., 2016 ] models from Feldman and Zhang [2020] . It plots a histogram of these scores, showing the frequency of specific log curvature values.A naive membership inference attack would apply a threshold to separate these distributions. However, it is common practice to consider sample specific scores (i.e., conditioned on the target sample). This is visualized in Figure 3 which plots the distribution of curvature scores for a single ImageNet sample, indicating differences when the sample is part of the training set versus when it is a test or unseen sample. The data was generated using models from Feldman and Zhang [2020] . The figure also includes a kernel density estimate (KDE, shown as a dashed line) to better to visualize the underlying distribution. Figure 3 suggests that, similar to Carlini et al. [2022] sample conditional curvature scores can be modeled using a Gaussian parametric model. If we represent the curvature score Curv ϕ as a random variable, then Curv ϕ ∼ N (µ, σ). The probability density function of Curv ϕ is a function of the randomness of the algorithm ϕ, the dataset S and the i th sample z i and be denoted by p c (ϕ, S, z i ). Similarly, let p(ϕ, S, z i ) denote the probability density function of the neural network's output probability, which is also parameterized by the randomness of the algorithm ϕ, the dataset S and the i th sample z i . With this setup we present the following theoretical results on the upper bound on the KL divergence between train and test distribution for the two cases. Theorem 4.1 presents the upper bound on the KL divergence when using the neural network's output probability scores, and Theorem 4.2 presents the upper bound on the KL divergence when using input loss curvature. Theorem 4.1 (Privacy bounds Train-Test KL Divergence). Assume ϵ-differential private algorithm, then the KL divergence between train-test distributions of the neural network's output probability is upper bound by the differential privacy parameter ϵ given by:EQUATIONSketch of Proof. Given that the algorithm A is ϵ-differentially private, we know that the probability of the output h ϕ S ∈ R on dataset S is bounded by e ϵ times the probability of the same event on the neighboring dataset S \i , i.e.Pr ϕ [h ϕ S ∈ R] ≤ e ϵ Pr ϕ [h ϕ S \i ∈ R].From this inequality it follows that the KL divergence between the output distributions of A on S and S \i is bounded by KL(p(ϕ, S, z i ) || p(ϕ, S \i , z i )) ≤ ϵ. The full proof for Theorem 4.1 is provided in Appendix A.3. Theorem 4.2 (Dataset Size and Privacy bound Curvature KL Divergence). Let the assumptions of error stability 2, generalization 3, and uniform model bias 4 hold. Further, assume 0 ≤ ℓ ≤ L. Let the conditional distribution be parameterized with variance σ. Then, any ϵ-differential private algorithm using a dataset of size m with a probability at least 1 -δ satisfies:D KL (p c (ϕ ϕ ϕ, S, z i ) || p c (ϕ ϕ ϕ, S \i , z i )) ≤ [Lm(1 -e -ϵ ) + c] 2 2σ 2 (8) c = (4m -1)γ + 2(m -1)∆ + ρ 6 E[∥α∥ 3 ] + L (9)Sketch of Proof. We begin by assuming a Gaussian model for the curvature distribution and expand the formula for the KL divergence, which leads to expressions involving the mean and standard deviation of the curvature scores. To establish an upper bound for this expression, we prove a lemma concerning the upper bound of the mean curvature, utilizing results from Ravikumar et al. [2024] . This enables us to express the upper bound on the mean curvature in terms of the privacy parameter ϵ.Subsequently, this result is employed to derive an upper bound on the KL divergence of the curvature scores. The proof for Theorem 4.2 is provided in Appendix A.5.Discussion. Theorem 4.1 and Theorem 4.2 provide the upper bound on KL Divergence between train and test distributions of the probability and input loss curvature scores, respectively. They imply the upper limit of MIA performance, and interestingly, the result of Theorem 4.2 suggests the role of the dataset size on MIA performance. However, the relation is not as straightforward as suggested by Equation 8. This is because, in real applications, parameter σ depends on ϵ, and so does the loss bound L. In fact, the loss bound is also dependent on the number of samples m [Bousquet and Elisseeff, 2002] . Then, the performance of MIA using curvature scores exceeds that of confidence scores with a probability at least 1 -δ when:EQUATIONTheorem 4.3 suggests that the performance of curvature based MIA will exceed that of probability score based methods when the size of the dataset used to train the target model exceed a certain threshold. Indeed, this is what we observe in practice (see section 6.4).The proof compares the upper bounds from Theorem 4.1 and Theorem 4.2, followed by a series of algebraic manipulations. This identifies the conditions under which the performance of curvature scores exceeds that of confidence scores. The full proof can be found in Appendix A.6.On the validity of the assumptions. Before presenting our experiments to validate our theory, we briefly discuss the validity of our assumptions in practical settings. Research by Hardt et al. [2016] shows that stochastic gradient methods, such as stochastic gradient descent, achieve small generalization error and exhibit uniform stability. Thus, the assumptions of stability (Equation 2) and generalization (Equation 3) are justified. Model bias is a property of the model, and a uniform bound across different datasets seems reasonable. Note, uniform loss bound (and L independent of m as used by Theorem 4.3) holds true for certain losses and for statistical models and is often assumed in learning theory [Wang et al., 2016] . Lastly, the υ-adjacency can be ensured through construction. For large datasets this may not be needed, because the size of the ball B p (υ) is unconstrained. Hence two samples from the same class that are similar may suffice. Given the size of modern datasets, this assumption is also reasonable.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We include all the training details for the models in Appendix A.13. We also provide the pseudo-code for the proposed technique in Appendix A.2 along with all the hyper parameter details (see Appendix A.13). Further, we also provide the code in the supplementary material. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide a detailed discussion about the broader impact of the research presented in the paper in Appendix A.11.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
zero order input loss curvature mia: 
 To test if input loss curvature based membership inference performs better than existing methods we need an efficient technique to estimate curvature. We are interested in black-box membership inference attacks, where one does not have access to the target network's parameters. However, current techniques use Hutchinson's trace estimator [Hutchinson, 1989] to measure input loss curvature such as from Garg and Roy [2023] , Garg et al. [2024] or Ravikumar et al. [2024] . This approach needs to evaluate the gradient and hence requires access to model parameters. To solve this issue, we propose using a zero-order estimation technique. Zero-order curvature estimation starts with a finite-difference estimation. Consider a function f : R n → R, then the Hessian at a given point z i can be estimated as follows:∇ 2 f (z i ) = n 2 f (z i + hv + hu) -f (z i -hv + hu) -f (z i + hv -hu) + f (z i -hv -hu) 4h 2 uv ⊤ (11)where h is a small increment (a hyper parameter in out case), and u, v are vectors in R n . In our case, to get the input loss curvature, we have f ← ℓ(g(x i ), y i ), where g is the neural network, ℓ is the loss function and z i = (x i , y i ) are the image, label pair. The pseudo-code for obtaining input loss curvature score using zero order estimation shown in Algorithm 1 in Appendix A.1. To execute membership inference attack using input loss curvature scores, we propose the following methodology. First, we begin by training shadow models, similar to Shokri et al. [2017] , Carlini et al. [2022] . These shadow models are used to obtain empirical estimates of parametric model for the curvature score described in Section 4. During the inference phase, we employ a likelihood ratio between the sample being in the train set vs test set parametric models to identify the membership status of a given sample (see Appendix A.2 for pseudo-code). In addition, we perform a negative log likelihood ratio test. We denote the results of likelihood test as 'LR' and the results of the negative log likelihood ratio test as 'NLL'.6 Experiments 
experimental setup: 
 Datasets. To evaluate our theory, we consider the classification task using standard vision datasets. Specifically, we use the CIFAR10, CIFAR100 [Krizhevsky et al., 2009] and ImageNet [Russakovsky et al., 2015] datasets.For experiments on ImageNet we use the ResNet50 architecture [He et al., 2016] . For CIFAR10 and CIFAR100, we used the ResNet18 architecture. Details regrading hyperparameters are provided in Appendix A.13. To improve reproducibility, we have open sourced the code at https://github.com/DeepakTatachar/Curvature-Clues.Training. For experiments using private models, we trained ResNet18 models with the Opacus library [Yousefpour et al., 2021] using DP-SGD with a maximum gradient norm of 1.0 and a privacy parameter of δ = 1 × 10 -5 . Shadow models for CIFAR10 and CIFAR100 were trained on a 50% subset of the data for 300 epochs. For ImageNet, we used pre-trained models from Feldman and Zhang [2020] , trained on a 70% subset of ImageNet. More details about training and compute resources are provided in Appendix A.13.Metrics. To evaluate curvature scores, we use AUROC and balanced accuracy. The Receiver Operating Characteristic (ROC) is the plot of the True Positive Rate (TPR) against the False Positive Rate (FPR). The area under the ROC is called Area Under the Receiver Operating Characteristic (AUROC). AUROC of 1 denotes an ideal detection scheme, since the ideal detection algorithm results in 0 false positive and false negative samples.In this subsection, we compare the performance of the proposed input loss curvature based membership inference against prior MIA techniques.Setup: We use CIFAR10, CIFAR100 and ImageNet datasets to test the MIA performance. We consider a black-box MIA setup similar to Carlini et al. [2022] . We use ResNet18 for CIFAR10 and CIFAR100, for ImageNet we use the ResNet50 architecture. For all the membership inference attacks, we compute a full ROC curve and report the results. When using shadow models we 64 for CIFAR10 and CIFAR100 and 52 for ImageNet. The AUROC plot for CIFAR100 for various methods are shown in Figure 4 . Table 1 reports the average balanced accuracy and AUROC values over three seeds on various MIA methods on the three datasets. The plot in Figure 4 is a log-log plot to emphasize performance of the proposed method at very low false positive rates (see the orange line y-intercept). We also studied the effect of augmentations and the results can be found in Appendix A.7, the takeaway was that adding more augmentations improved performance. Note that the results presented in Table 1 used 2 augmentations (image + mirror) for our technique and Carlini et al. [2022] for fair comparison.Results: From Table 1 , we see that the proposed method performs better than all existing MIA techniques on both ImageNet and CIFAR datasets. Apart from AUROC and balanced accuracy results, the log-log plot emphasizes the performance at really low FPR (i.e. the y intercept, high TPR at low FPR in Figure 4 ). Additional results at low FPR are available in Appendix A.8.Takeaways: As predicted by Theorem 4.2, the higher KL divergence between train and test curvature score distributions results in superior MIA performance. Further, we observe that while using a negative log likelihood ratio test does better in AUROC and balanced accuracy, the parametric likelihood ratio test does better at low false positive rates as shown in Figure 4 . Thus, the proposed use of curvature should be tailored based on use case. Applications that demand high AUROC can use NLL approach, while those that demand high TPR at very low FPR should use the LR technique.In this section, we study the effect of differential privacy on MIA performance and test the ϵ relation predicted by Theorem 4.2.Setup: To study the effect of privacy on MIA attack performance, we use DP-SGD trained models. We use privacy ϵ values of 1, 2, 3, 4, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 with δ = 1 × 10 -5 . We plot the result of the proposed curvature based MIA in Figure 5 . In Figure 5 we also plots the best fit using s f (L f (1 -e -ϵ ) + c f ) 2 , where s f , L f and c f are fit to the data.Results and Takeaways: We see that the theoretical prediction from Theorem 4.2 about MIA performance is well matched. Since Theorem 4.2 provides an upper bound, the results validate the theory.In this section, we study the effect of dataset size on MIA attack performance. This lets us test the relationship of MIA performance to m as predicted by Theorem 4.2 and Theorem 4.3.Setup: For this experiment, we train models on increasing dataset size on CIFAR100. Specifically, we train multiple seeds on various subsets randomly chosen from the CIFAR100 training set. We also repeated the same by choosing the lowest curvature samples from CIFAR100 and train with the same subset sizes. Next, for each of the models we perform MIA attack and we present the results in Figure 6 (randomly chosen) and Figure 7 (lowest curvature samples).Results: From Figure 6 , we see that when samples are randomly chosen, the MIA performance decreases as we add more samples. This result is consistent with prior literature [Abadi et al., 2016] . Further as predicted by Theorem 4.3 beyond 30 -40% subset Curv ZO LR out perform prior works and Curv ZO NLL outperforms prior works beyond 10% subset size.To extend this analysis, we train models on curvature based coresets [Garg and Roy, 2023] . These coresets of low input loss curvature samples have been shown to memorize less [Garg et al., 2024 , Ravikumar et al., 2024] . Thus we expect MIA accuracy to increase as we increase coreset size. This is exactly what happens and is shown in Figure 7 which plots the AUROC and accuracy of the NLL curvature attack as we increase curvature coreset size. However, the MIA performance is higher for comparable size in Figure 6 and 7 . This suggests that curvature based coresets result in more susceptible models, which is also supported by results in Song et al. [2019] .Takeaways: We validate Theorem 4.3. We note that beyond a certain dataset size (of about 30 -40% subset of the training set Curv ZO LR out perform prior works and Curv ZO NLL outperforms prior works beyond 10% subset size) curvature scores outperform probability score based MIA method. While Theorem 4.3 predicts that curvature-based MIA outperforms other methods on sufficiently large datasets. This condition is often met by real datasets as evident from the results presented. 
conclusion: 
 In this paper, we explored the properties of input loss curvature on the test set. Specifically, we focus on using input loss curvature to distinguish between train and test examples. We established a theoretical framework deriving an upper bound on train-test KL Divergence based on privacy and training set size. To extend the applicability of input loss curvature computation to a black-box setting, we propose a novel zero-order curvature estimation method. This enables the development of a new cutting-edge black-box membership inference attack (MIA) methodology that leverages input loss curvature. Through extensive experiments on the ImageNet and CIFAR datasets, we demonstrate that our input loss curvature-based MIA method outperforms existing state-of-the-art techniques. Our results corroborate our theoretical predictions regarding the relationship between MIA performance and dataset size. Specifically, we show that beyond a certain dataset size, the effectiveness of curvature scores surpasses other methods. Importantly, we observe that this dataset size condition is frequently met in practical scenarios, as evidenced by our results on the CIFAR100 dataset. These findings advance our understanding of input loss curvature in the context of privacy and help build more secure deep learning models. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release data or models thus this paper does not pose such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We provide the detailed list of assets used and the terms under which they were licensed in Appendix A.14. We have respected the licensing terms of the original owners.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We release the code for reproducing the results under the MIT license and details of its usage are documented in Appendix A.13 and the supplementary material README file. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used.  
 
 Appendix: A.1 Zero Order Curvature EstimationWe present the pseudo-code for zero-order curvature score calculation below. Please note that Algorithm 1 shows the input loss curvature calculation for one example; however, it can be easily implemented for batch data.Algorithm 1 Pseudo-code for Zero Order Input Loss Curvature Estimation 1: Input: z i = (x i , y i ) (image, label data), ℓ (loss function), n iter (hyperparameter: number of iterations), h (hyperparameter: small increment), g (neural network) 2: Output: curv_estimate (zero order input loss curvature estimate of z i ) 3: curv ← 0 4: for i ← 1 to n iter do 119:5: v ∼ Rademacher 6: u ∼ Rademacher 7: f ← ℓ(g(x i ), y i ) 8: ∇ 2 f (z i ) ← Use Equationcurv ← curv + tr(∇ 2 f (z i )) 10: end for 11: curv_estimate ← curv/n iter 12: return curv_estimateIn this section, we present the detailed step by step description and pseudo code for the curvature based MIA (refer to Algorithm 2) 1. Initialization (Lines 3-4): Two empty sets, curv in and curv out , are initialized. These sets will store the curvature scores for the shadow models trained with and without the target example z i . 2. Shadow Model Training (Lines 5-12): For a specified number of iterations N , the algorithm performs the following steps: S \i || = 1. Next let R ⊂ Range(A) such that R = {h | h(x i ) = y i }.Since A is ϵ-differentially private, then it follows from the definition of differential privacy in Equation 1thatEQUATION=⇒ p(ϕ, S, z i ) ≤ e ϵ p(ϕ, S \i , z i )Since p(ϕ, S, z i ) = Pr ϕ [h ϕ S ∈ R] and denotes the probability density function of the neural network's output probability. Now, we use the definition of KL divergenceD KL p(ϕ, S, z i ) || p(ϕ, S \i , z i ) = Eϕ log p(ϕ, S, z i ) p(ϕ, S \i , z i ) ≤ Eϕ [log e ϵ ] From Equation 14 D KL p(ϕ, S, z i ) || p(ϕ, S \i , z i ) ≤ ϵ ■ A.4 Lemma A.1Lemma A.1 (Upper bound on Input Loss Curvature). Let A be a randomized algorithm which is ϵ-differentially private and the assumptions of error stability 2, generalization 3, and uniform model bias 4 hold. Further, assume 0 ≤ ℓ ≤ L. Then for two adjacent datasets S, S \i ∼ D with a probability at least 1 -δ we have:EQUATIONProof of Lemma A.1 For this proof, we use results from Nesterov and Polyak [2006] , we restate it here for convenience Lemma A.2. If Lipschitz assumption 5 on the Hessian of ℓ holds from Nesterov and Polyak [2006] we haveEQUATIONFrom Lemma A.2 we have- ρ 6 |z 1 -z 2 | 3 ≤ ℓ(h, z 1 ) -ℓ(h, z 2 ) -⟨∇ℓ(h, z 2 ), z 1 -z 2 ⟩ -⟨∇ 2 ℓ(h, z 2 )(z 1 -z 2 ), z 1 -z 2 ⟩ ℓ(h, z 1 ) -ℓ(h, z 2 ) -⟨∇ℓ(h, z 2 ), z 1 -z 2 ⟩ -⟨∇ 2 ℓ(h, z 2 )(z 1 -z 2 ), z 1 -z 2 ⟩ ≤ ρ 6 |z 1 -z 2 | 3This gives us a lower bound on ℓ(h, z 1 )EQUATIONConsider z j ∈ S such that z i = z j + α for some j ̸ = i where α ∈ B p (υ) such that E[α] = 0 and E[α T α] = 1. Using the lower bound in Lemma A.2 from Ravikumar et al. [2024] withEQUATIONUsing the result from Lemma A.2- ρ 6 ∥α∥ 3 + Eϕ[ℓ(h ϕ S \i , z i )] + Eϕ[⟨∇ℓ(h ϕ S \i , z i ), α⟩] + Eϕ[⟨∇ 2 ℓ(h ϕ S \i , z i )α, α⟩] -Eϕ[ℓ(h ϕ S , z i )] ≤ mβ + (4m -1)γ + 2(m -1)∆ (20)Taking Expectation over α and since the mean of α = 0 we haveEϕ[ℓ(h ϕ S \i , z i )] + Eα,ϕ[⟨∇ℓ(h ϕ S \i , z i ), α⟩] + Eα,ϕ[⟨∇ 2 ℓ(h ϕ S \i , z i )α, α⟩] -Eϕ[ℓ(h ϕ S , z i )] ≤ mβ + (4m -1)γ + 2(m -1)∆ + ρ 6 ∥α∥ 3Note that we can change the order of expectation using Fubini's theoremEϕ[ℓ(h ϕ S \i , z i )] + Eϕ,α[⟨∇ 2 ℓ(h ϕ S \i , z i )α, α⟩] -Eϕ[ℓ(h ϕ S , z i )] ≤ mβ + (4m -1)γ + 2(m -1)∆ + ρ 6 ∥α∥ 3 Eϕ[ℓ(h ϕ S \i , z i )] + Eϕ[tr(∇ 2 ℓ(h ϕ S \i , z i ))] -Eϕ[ℓ(h ϕ S , z i )] ≤ mβ + (4m -1)γ + 2(m -1)∆ + ρ 6 ∥α∥ 3 Eϕ[tr(∇ 2 ℓ(h ϕ S \i , z i ))] ≤ mβ + (4m -1)γ + 2(m -1)∆ + ρ 6 E[∥α∥ 3 ] + Eϕ[ℓ(h ϕ S , z i )] -Eϕ[ℓ(h ϕ S \i , z j )] Eϕ[tr(∇ 2 ℓ(h ϕ S , z i ))] ≤ mβ + (4m -1)γ + 2(m -1)∆ + ρ 6 E[∥α∥ 3 ] + L From Ravikumar et al. [2024] Lemma 5.2 we have β ≤ L(1 -e -ϵ). Thus we have the upper bound:Eϕ[Curvϕ(z, S \i )] ≤ L(m(1 -e -ϵ ) + 1) + c 1 (21) c 1 = (4m -1)γ + 2(m -1)∆ + ρ 6 E[∥α∥ 3 ] ■ (22) A.5 Proof of Theorem 4.2This proof uses results from Lemma A.1 provided above. For the proof of Theorem 4.2 let p c (ϕ, S, z i ) be the curvature probability mass function, then using the Gaussian model discussed in Section 4 we model it as a Gaussian. Thus, we can writeEQUATION)p c (ϕ, S \i , z i ) = 1 σ S \i √ 2π e - (ϕ -µ S \i ) 2 2σ 2 S \i (24) Now consider the D KL (p c (ϕ, S, z i ) || p c (ϕ, S \i , z i )) D KL (p c (ϕ, S, z i ) || p c (ϕ, S \i , z i )) = Eϕ log p c (ϕ, S, z i ) p c (ϕ, S \i , z i )) (25) = Eϕ - (ϕ -µ S ) 2 2σ 2 + (ϕ -µ S \i ) 2 2σ 2 Assume σ = σ S \i = σ S (26) = Eϕ -ϕ 2 -µ 2 S + 2ϕµ S + ϕ 2 + µ 2 S \i -2ϕµ S \i 2σ 2 (27) = Eϕ µ 2 S \i -µ 2 S 2σ 2 (28) = µ 2 S \i -µ 2 S 2σ 2 (29)To upper bound Equation 29, we need an upper bound on µ S \i and a lower bound on µ S . The lower bound on µ 2 S = 0 For ease of notation let's define β max := L(m(1 -e -ϵ ) + 1). Notice that the upper bound of per sample µ S \i = Eϕ[Curvϕ(z, S \i )]. Using the result from Lemma A.1 in Equation 29we haveD KL (p c (ϕ, S, z i ) || p c (ϕ, S \i , z i )) ≤ (β max + c 1 ) 2 -0 2 2σ 2 (30) ≤ (β max + c 1 ) 2 2σ 2 (31) ≤ [L(m(1 -e -ϵ ) + 1) + c 1 ] 2 2σ 2 ■ (32) A.6 Proof of Theorem 4.3Assuming the bounds of Theorem 4.1 and Theorem 4.2 are tight, then using curvature score will outperform neural network probability scores if:D KL (p c (ϕ, S, z i ) || p c (ϕ, S \i , z i )) > D KL (p(ϕ, S, z i ) || p(ϕ, S \i , z i )) (33) [L(m(1 -e -ϵ ) + 1) + c 1 ] 2 2σ 2 > ϵ [Lm(1 -e -ϵ ) + c] 2 2σ 2 > ϵ ; c = c 1 + L L 2 (1 -e -ϵ ) 2 m 2 + 2Lc(1 -e -ϵ )m + c 2 -2σ 2 ϵ > 0Using the quadratic formula we have the roots given by:= -2Lc(1 -e -ϵ ) ± 4L 2 c 2 (1 -e -ϵ ) 2 -4L 2 (1 -e -ϵ ) 2 (c 2 -2σ 2 ϵ) 2L 2 (1 -e -ϵ ) 2 = -2Lc ± 4L 2 c 2 -4L 2 (c 2 -2σ 2 ϵ) 2L 2 (1 -e -ϵ ) = -Lc ± L 2 c 2 -L 2 (c 2 -2σ 2 ϵ) L 2 (1 -e -ϵ ) = -c ± c 2 -(c 2 -2σ 2 ϵ) L(1 -e -ϵ ) = -c ± √ 2σ 2 ϵ L(1 -e -ϵ )Assuming both roots are real, the larger root is√ 2σ 2 ϵ -c L(1 -e -ϵ )Thus when m >√ 2σ 2 ϵ -c L(1 -e -ϵ )Equation 33 is satisfied. ■In this section we present the results of using 1 augmentation (only the image) vs 2 augmentations (image + mirror) in Tables 2, 3 and 4 for ImageNet, CIFAR100 and CIFAR10 respectively. For these results, we used 64 shadow models for CIFAR10 and CIFAR100 and 52 shadow models for ImageNet. In this section, we present the performance of curvature based MIA compared with prior works using TPR (true positive rate) at very low FPR percentages. The results are presented when using 1 augmentation (only the image) and 2 augmentations (image + mirror) in Table 5 and Figure 8 . For these results, we use 64 shadow models for all the results in Table 5 (for methods that use shadow models). The results for Carlini et al. [2022] are a little lower than the one reported by Carlini et al. [2022] in Table I of their paper. However, note that the authors of Carlini et al. [2022] report the results with 256 shadow models while we report it using 64 shadow models. This highlights the benefits of curvature based approach, it achieves similar performance to Carlini et al. [2022] with 256 shadow models but with only 64 shadow models thus needs significantly less shadow models to achieve similar performance.Takeaways: The parametric curvature LR method has the best TPR at very low FPR and makes efficient use of shadow models.TPR @ 0.1% FPR TPR @ 0.01% FPR TPR @ 0.1% FPR TPR @ 0.01% FPR Curv ZO LR (Ours) 21.07 ± 0.80 15.74 ± 3.12 23.92 ± 0.92 17.17 ± 1.87 Curv ZO NLL (Ours)6.42 ± 0.62 0.05 ± 0.04 8.29 ± 1.52 0.10 ± 0.14 Carlini et al. [2022] 15.52 ± 0.54 5.02 ± 1.70 15.80 ± 0.49 6.56 ± 1.13 Sablayrolles et al. [2019] 10.50 ± 0.59 4.30 ± 0.53 10.50 ± 0.59 4.30 ± 0.53 Watson et al. [2022] 6.62 ± 0.23 2.95 ± 0.18 6.62 ± 0.23 2.95 ± 0.18 Ye et al. [2022] 6.63 ± 0.52 2.86 ± 0.42 6.63 ± 0.52 2.86 ± 0.42 Song and Mittal [2021] 1.21 ± 0.37 1.21 ± 0.37 1.21 ± 0.37 1.21 ± 0.37 Yeom et al. [2018] 0.07 ± 0.01 0.01 ± 0.00 0.07 ± 0.01 0.01 ± 0.00Table 5 : Comparison of TPR @ 0.1% FPR and TPR @ 0.01% FPR for CIFAR100 dataset with 1 and 2 augmentations. Results reported are the mean ± std obtained over 3 seeds and using 64 shadow models. A.9 ZO Estimation PerformanceIn this section, we present the results of comparing the proposed zero-order curvature estimation with Hutchinson's trace estimator-based curvature calculation. It is important to note that the trace estimator proposed by Garg et al. [2024] requires access to the model's parameters, thus it is a white-box attack. In contrast, the proposed zero-order (ZO) estimation operates as a black-box attack.The comparative results are presented in Table 6 . Additionally, Figure 9 illustrates the True Positive Rate (TPR) vs. False Positive Rate (FPR) for the ZO estimation and Hutchinson's trace estimator, utilizing a log-log plot to emphasize performance at low FPR.Takeaways. From Table 6 we see that zero-order approach to curvature estimation results in notable differences in performance. Specifically, when examining AUROC and balanced accuracy, particularly on the CIFAR-10 dataset, a gap of several percentage points can be observed between the two methods. From Figure 9 , we observe that the proposed LR technique performs similarly between Hutchinson's trace estimator [Garg et al., 2024] and the proposed ZO estimation. However, the NLL method exhibits much greater sensitivity to estimation errors, as demonstrated by a significant performance drop at low FPR levels. [Garg et al., 2024] .In this section, we present the results of evaluating the proposed zero-order MIA on different neural network architectures. We selected four architectures: Inception [Szegedy et al., 2015] (specifically, the smaller Inception architecture used by Feldman and Zhang [2020] ), VGG11 [Simonyan and Zisserman, 2014] , VGG11 with batch normalization (denoted as VGG11BN), and finally, a ViT-B16 vision transformer [Vaswani et al., 2017 , Dosovitskiy et al., 2021] . The results are provided in Table Table 7 . Please note that for all of the results we used ResNet18 shadow models and 2 augmentations.Takeaways. The proposed techniques outperform the previous state-of-the-art method by Carlini et al. [2022] across all architectures. Additionally, the lower performance on the Vision Transformer (ViT) model can be attributed to transfer learning. Since ViTs do not perform well when trained from scratch on CIFAR-100, we used an ImageNet-pretrained model, which was fine-tuned on CIFAR-100. This led to significant privacy benefits, which aligns with the findings of Mehta et al. [2022] . Code available at https://github.com/DeepakTatachar/Curvature-CluesProject website with models and other assets is available at https://engineering.purdue.edu/NRL/ projects/curvature-clues38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: This papers studies how the second-order derivative of the loss w.r.t. the input data vector is revealing information about whether the data was used for training.  

Reviewers generally like the work how it contributes to membership inference attack,  privacy auditing as well as that the paper provided both a theoretical analysis and experiments on a variety of settings.

All reviewers are supportive of the paper and I am happy to recommend accepting the paper to neurips.

Reviews: review: paper_summary: paper_summary: The paper explores the use of loss curvature, under the hypothesis that test samples lie in high curvature regions, to improve the performance of membership inference attacks. It estimated the curvature by measuring the trace of the Hessian of the loss values. It demonstrates the viability of the approach in both white-box and black-box settings. It empirically validates the success of the approach on multiple datasets.
strength_weakness: strength_weakness: While the loss value based MIA evaluation generalizes to pretty much all class of models, it’s unclear if loss curvature based MIA attack would enable a similar generalization, i.e., going beyond the small-scale classification setup to self-supervised models, contrastive models, and other foundation models [1]. 

As fig. 6 itself highlights, the robustness of attack against a simple defense that subsamples the training data for shadow models is very low. Just using ½ of the training data renders the attack ineffective (albeit Carlini et al. suffers from the same limitation). Since shadow models are only a proxy of the true black-box model with training data information not publicly available, in practice they will likely only be trained on a subset of original training data of the block-box model, thus significantly reducing the attack success in black-box threat model. 

1. Niu, Jun, Xiaoyan Zhu, Moxuan Zeng, Ge Zhang, Qingyang Zhao, Chunhui Huang, Yangming Zhang et al. "SoK: Comparing Different Membership Inference Attacks with a Comprehensive Benchmark." arXiv preprint arXiv:2307.06123 (2023).
questions: questions: Why not use zero-order loss estimation in black-box method as it would be significantly more sample efficient than block-box curvature estimation. 

Would using the first-order statistics, gradients of the loss w.r.t input, be as helpful as curvature in MIA attacks?
limitations: limitations: The paper proposes using second order statistics (loss curvature) over using loss values in membership attacks. However it will fail to generalize to networks that are piecewise linear, thus having zero curvature for all samples. A simple example of it would be the resnet models with convolution layer, ReLU activation, and Batchnorm normalization. While the paper does consider the resnet models, it doesn’t clarify the architecture uses. In the piecewise linear architecture, only the cross-entropy loss on output logits would contribute to loss curvature.

Thus it’s critical to ablate the choice of network architecture and demonstrate that despite the diversity in architectural designs, the approach succeeds in MIA attacks. A few simple variations would be changing the normalization layers, activation functions, and broadly the class of models (CNNs vs transformers).
paper_summary: paper_summary: This paper develop a theoretical framework to derive an upper bound on train-test distinguishability based on input loss curvature. The authors propose a new black-box membership inference attack utilizing input loss curvature, which surpasses existing methods in membership inference effectiveness. The paper also explores the potential of using subsets of training data as a defense mechanism against shadow model-based membership inference attacks.
strength_weakness: strength_weakness: 1. In the theoretical analysis, the authors propose the train-test distinguishability distribution upper bound for both data and curvature. However, they compare the upper bounds directly and suggest that curvature based MIA is a better method when the size of dataset is large enough. The comparison between upper bounds may not reflect the real relationship. To validate this result, one may compare between an upper bound and a lower bound.
2. The assumption that the curvature score can be modeled with Gaussian distribution has not been discussed and validated in the paper, which is a very important assumtion in theoretical analysis. Figure 3 seems that the distribution is not very similar to Gaussian.
questions: questions: What is the effect of the error in the zeroth-order curvature estimation? How the estimated curvature deviates Gaussian and can we still model the distribution with it?
limitations: limitations: The authors have addressed the limitations of their work by acknowledging that their analysis only holds when the dataset size is large enough. They emphasize the need for further exploration of the method's applicability to different models and datasets and the potential impact of computational complexity.
paper_summary: paper_summary: This paper builds the connection between privacy and input loss curvature.  Core observation is that test samples often lie in higher curvature areas than those prototypical samples, so the input loss curvature can be a good metric to distinguish between train and test sets. Utilizing this observation, authors build a stronger black-box MIA, outperforming previous MIAs, and they also reveal a previously unknown limitation of shadow model based MIAs. The paper provides both theoretical analysis and experimental results.
strength_weakness: strength_weakness: Line 117 missing a `.`
questions: questions: Q1: What is $s_f$, $L_f$ and $c_f$ in Line 300? The notation seems to be missing from the context.

Q2: How accurate is the zeroth-order estimation of input loss curvature? Can you provide a comparison of this estimation to the real values   (assuming we have white-box access to target model)
limitations: limitations: The limitations of this work is properly discussed in Appendix.
paper_summary: paper_summary: This paper conducts a theoretical and empirical investigation of the use of input loss curvatures in MIAs. They study the train-test input loss curvature scores and use this to develop a zero order input loss curvature black-box MIA.
strength_weakness: strength_weakness: 1. Missing some references, e.g. Li et al. 2023 or the original DetectGPT paper (Mitchell et al. 2023).
2. I believe the paper would benefit from including some sort of proof sketch of the theorems in the main body.
questions: questions: 1. Other authors have used the Hutchinson trace estimator to estimate information about input loss curvature. How does the performance of this method compare to the zero-order method presented in the paper?
limitations: limitations: Yes

Comments: title: Previous concerns addressed in the rebuttal
comment: I thank the authors for their rebuttal. The detailed response and the additional results addressed my concerns. I will keep the score.
Thanks for posting a detailed rebuttal to the review. I believe that it had addressed the concerns raised in the review, thus I've increased my score to accept.
Thanks for the rebuttal. I will keep my score.
I have read the rebuttal and will increase my score.