Leave-one-out Distinguishability in Machine Learning
Authors: Jiayuan Ye
Anastasia Borovykh
Soufiane Hayou
Reza Shokri
Venue: ICLR 2024 poster (2024)
Keywords: memorization
influence estimation
information leakage
neural network Gaussian process
Primary area: societal considerations including fairness, safety, privacy

Abstract: We introduce an analytical framework to quantify the changes in a machine learning algorithm's output distribution following the inclusion of a few data points in its training set, a notion we define as leave-one-out distinguishability (LOOD).  This is key to measuring data **memorization** and information **leakage** as well as the **influence** of training data points in machine learning. We illustrate how our method broadens and refines existing empirical measures of memorization and privacy risks associated with training data. We use Gaussian processes to model the randomness of machine learning algorithms, and validate LOOD with extensive empirical analysis of leakage using membership inference attacks. Our analytical framework enables us to investigate the causes of leakage and where the leakage is high.  For example, we analyze the influence of activation functions, on data memorization.  Additionally, our method allows us to identify queries that disclose the most information about the training data in the leave-one-out setting.  We illustrate how optimal queries can be used for accurate **reconstruction** of training data.

Published as a conference paper at ICLR 2024
LEAVE-ONE-OUR DISTINGUISHABILITY IN MACHINE LEARNING
Jiayuan Yel, Anastasia Borovykh', Soufiane Hayou', and Reza Shokri National University of Singapore, Imperial College London
ABSTRACT
We introduce analytical framework to quantify the changes in machine learn- ing algorithm's 's output distribution following the inclusion of a few data points its training set, notion we define leave-one-out distinguishability (LOOD). This is key to measuring data memorization and information leakage as well as the influence of training data points in machine learning. We illustrate how method broadens and refines existing empirical measures of memorization and pri- vacy risks associated with training data. We use Gaussian processes to model the randomness of machine learning algorithms, and validate LOOD with extensive empirical analysis of leakage using membership inference attacks. Our analytical framework enables us to investigate the causes of leakage and where the leakage is high. For example, we analyze the influence of activation functions, on data mem- orization. Additionally, our method allows us to identify queries that disclose the most information about the training data in the leave-one-out setting. We illustrate how optimal queries can be used for accurate reconstruction of training data [
INTRODUCTION
A key question in interpreting a model involves identifying which members of the training set have inftuenced the model's predictions for particular query (Koh & Liang, 2017: Koh et al., 2019 Pruthi et al, 2020 Our goal is to understand the impact of the presence of a specific data point in the training set of a model on its predictions. This becomes important, for example, when model's prediction on a data point is primarily due to its own presence (or that of points similar to it) in the training set phenomenon referred to as memorization (Feldman) 2020) and can cause leakage of sensitive information about the training data. This enables an adversary to infer which data points were included in the training set (using membership inference attacks), by observing the model's predictions (Shokri et al. 2017 Zarifzadeh et al.) 2023). One way to measure such influence, memorization, and information leakage, and thus addressing related questions, by (re)training models with various combinations of data points. This empirical approach, however, is computa- tionally expensive (Feldman & Zhang 2020), and does not enable efficient analysis of, for example, what data points are influential, which queries they influence the most, and what properties of model (architectures) can impact their influence. These limitations demand an efficient and constructive modeling approach to influence and leakage analysis. This is the focus of the current paper.
We generalize the above-mentioned closely-related concepts (Section 2). and unify them as the statistical divergence between the output distribution of models, trained using (stochastic) machine learning algorithms, when one or more data points are added to (or removed from) the training set. We refer to this measure leave-one-out distinguishability (LOOD). A larger LOOD of training data S in relation to query data Q suggests a larger potential for information leakage about S, when an adversary observes the prediction of models trained S and queried Q. As special variant of LOOD, if we measure the gap between the mean of prediction distributions over Icave-one-out models on a query, the resultant mean distance LOOD recovers existing definitions for influence (function) (Steinhardt et al.] 2017 Koh & Liang 2017 Koh et al. 2019) and memorization (self- influence) (Feldman Zhang] 2020 Feldmann (020) (i.e., a greater mean distance LOOD suggests
larger influence of training data S on query data Q).
Authors AB, SH and R$ are ordered alphabetically.
'The code is available through this link

---

Published as a conference paper at ICLR 2024
We propose analytical method that estimates LOOD accurately and efficiently (Section by using Gaussian Processes (GPs) to model the output distribution of neural networks in response to querying on any data point(s) We empirically validate that LOOD under NNGP models correlates well with the performance of membership inference attacks (Ye et al.] 2022 Carlini et al. 2023) against neural networks in the leave-one-out setting for benchmark datasets (Figure lb]. We also experimentally show that mean distance LOOD under NNGP agrees with the prediction differences under leave-one-out retraining (Figure of deep neural networks. This demonstrates that our ap- proach allows an accurate assessment of information leakage and memorization (elffinfluence for deep neural networks, while significantly reducing the required computation time (by more than two orders of magnitude Footnote 6 as we compute LOOD analytically.
Our method offers theoretical explanations as well as efficient algorithms (Section 4 to examine which query points are most influenced by any training data S (i.e., to quantify the leakage of mod- els in black-box settings). This further allows one to gauge the severity of this leakage by examining to what extent the most-influenced query can help reconstruct the training data. We prove that, for GPs under weak regularity conditions (Theorem 4.T). the differing data itself is a stationary point of LOOD. Experiments image datasets further show that, LOOD under NNGP model is maxi- mized when the query equals the differing datapoint (4b). These conclusions (for NNGP models) experimentally generalize to deep neural networks trained via SGD, where the empirical leakage is maximized when query data is within a small perturbation of the differing data (Figure 44. Addi- tionally, LOOD under NNGP remains almost unchanged even as the number of queries increases, even when the queries are optimized to have high LOOD (...e. In other words, for NNGP models, all the information related the differing point in the leave-one-out setting is contained within the model's prediction distribution solely on that differing point. We show how to exploit this to reconstruct training data in the leave-one-our or leave-one group-out settings. We present samples of reconstructed images in Figure 10 and 15 19 which strikingly resemble the training data, even when our optimization lands on sub-optimal queries.
Finally, the analytical structure of LOOD guides us in theoretically understanding how the model architecture can affect information leakage. In Section 6 we investigate how the analytical LOOD for Neural Network Gaussian Processes (NNGPs) is affected by activation functions. We prove that a low-rank kernel matrix implies low LOOD (Proposition 6. while existing literature (Proposi- tion [G.) prove that the NNGP kernel for fully connected networks is closer to low-rank matrix under ReLU activation (thus lead to less leakage) compared to smooth activation functions such as GeLU. Thus, smooth activation functions induce higher LOOD (information leakage) than non- smooth activation functions in deep neural networks. We validate that this conclusion aligns with empirical experiments for both NNGPs and for deep neural network trained via SGD (Figure 5)
LOOD DEFINITION AND ITS CONNECTIONS TO RELATED CONCEPTS
Definition 2.1 (LOOD). Let D and D' be two training datasets that only differ in the set S Let fo (Q) be the distribution of model predictions given training dataset D and query dataset Q, where the probability is taken over the randomness of the training algorithm. We define leave-one-out distinguishability (LOOD) as the statistical distance between fo(Q) and (Q):
LOOD(Q: D, D') = dist( (Q))
In the paper, mainly measure LOOD using the KL divergence Kullback & Leibler (151 to capture information leakage. However, as we explain below, different choices of dist() and f( allow us to recover established definitions of memorization, information leakage, and influence.
Information Leakage as LOOD. Information Leakage refers to how much a model's ' predictions can reveal information about specific data records in its training set, which would not be extractable if those entries were removed from the training set. In the black-box setting, where adversary observes model's prediction (distribution)
query set Q, this is exactly LOOD. This is closely related
2 NNNGPs are widely used in the literature standalone models Neal
2012
modeling tools for different
types neural networks (Lee et al. [207 Novak et al. [018 HHon et al. 22200] Yangl 2019), 9 and as building
blocks for model architecture search (Park et al. 2020) or data distillation (Lo0 ct al. 2022 *s s= D A D' and D' D.
2

---

Published as a conference paper at ICLR 2024
to Differential privacy (DP) (Dwork et al. 2006) setting, for specific datasets, and DP determines the maximum (white-box box) privacy loss across all possible datasets. | In this paper, we measure LOOD using KL divergence, which is different from Hocky-stick divergence used for equivalent definitions of (e, 6)DP (Balle et al. 2018). The closest DP definition LOOD is Rényi DP (Mironov 2017) (consider Rényí divergence with order a ++ 1).
LOOD measures privacy loss in black-box
LOOD gives useful information about leakage in DP ML. algorithms. DP guarantees that the pre- diction distributions cannot change beyond a certain bound under the change of any training data. However, such a guarantee reflects the worst-case scenario and the same bound holds for any possi- ble training data and query. DP bound does not help measure the privacy risk of specific data points in certain training sets and queries. LOOD, however, helps compute this risk and explore how in- formation leakage varies with different predictions, which prediction leads to the most leakage, and Section 4 and Appendix I.II.
how leakage evolves with more predictions (See Figure 4
LOOD controls the leakage quantified by inference attacks. One established method for empiri- cally assessing information leakage is through the performance of a membership inference attack (MIA) (Shokri et al. 2017), where an attacker attempts to guess whether a specific data point was part of the model's s training dataset. The power of any MIA (true positive rate given a fixed false positive rate) is controlled by the likelihood ratio test of observing the prediction given D versus D' as the training set (Neyman & Pearson, 1933). Thus, the leakage quantified by MIA is upper bounded by statistical distance relates to likelihood ratio, e.g., KI. divergence (Dong et al. 2019 See Figure Ta for the high correlation between LOOD and MIA performance, and see Appendix C
for why KL divergence is better suited than mean distance for measuring information leakage.
Influence (Function) (Derivatives of) Mean Distance LOOD. The problem of influence estima- tion arises in the context of explaining machine learning predictions (Koh & Liang, 2017) and ana- lyzing the robustness of training algorithm against adversarial data (Steinhardt et | (017)) where the influence of an (adversarial) training data = E D on the trained model's 's loss at test data Stest is defined as looss (z, 2teat) = L(o=es Preat) - L(O; Ztest), where 0 = arg min L(;; D) on dataset D and 0_, = arg ming T(00: D \ {=}). This quantity is thus the change of trained model's loss on the test point at due to the removal of one point z from the training dataset (when 2 = Zteat, reflects self-influence of 2). This is exactly LOOD given by mean distance (Definition
between loss
distributions, for an (idealized) training algorithm that outputs the optimal model given the training dataset. Estimating influence incurs computational difficulties, as it requires leave-one-our retraining and exact minimizers. Koh & Liang (2017) approximately compute this influence without reirain- ing for small perturbation of training data, via first-order Taylor approximation. They compute loert, 2test) = (g
7test) where d., dz,-- arg ming L0; D) - L(; 2) + L(; 24) via non-exact estimations of the Hessian (as exact Hessian computation is computationally expensive for large model). For simple models such as linear model and shallow CNN, the influence function approximation correlate well with leave-one-out retraining. However, for deep neural networks, the influence function approximation worsens significantly ((Koh & Liang 2017 Figure 2) and Fig- ure Ic By contrast, propose an analytical method for computing mean distance LOOD that shows high correlation to leave-one-out retraining even for deep neural networks (Figure Ic).
Memorization as LOOD.
Memorization, as defined in (Feldman 2020 page 5), is given by mem(A, D) = Ar = Ms) - PrA>A(D)ss) (h(xs) - ys): this measures the change of prediction probability on training data S after removing it from the training dataset D. given training algorithm A. To avoid training many models on and D S for each pair of training data and dataset, one approach is to resort to boosting llik techniques (Feldman & Zhhngg 2020 Zhang et al. (02T) to reduce the number of retrained models (at drop of estimation quality). which still requires training hundreds or thousands of models. The memorization measure is exactly mean distance LOOD Definition C.T when query equals the differing data (i.e., self-inffuence) and output function f is the prediction accuracy. Thus, we can analytically compute and reason about memo- rization in ML using LOOD.
'DP guarantee certainly applies to black-hox setting too, hut its primary analysis is done for the stronger white-box threat model, where adversary observes the model parameters (or parameter updates during training). KI. divergence (and thus L.OD)) can also be used for semantically measuring the advantage of reconstruc- tion attacks and attribute inference attacks. See (Guo al. 2022 Section 5.2) for concrete example.
3

---

Published as a conference paper at ICLR 2024
NNN modeek)
Irt fun (Koh & Langi (r=0.53, p-7.7e-38)
tampled from uniform distibbtiion sampled from CIAALO
Teclh (Pruti (1=0.58, p=3.1e-09)
2.0
(Mean distance LOGDir- 2-2.5e220)
MB 12 prediction ditetiinniieetttte diferrnce
1.5
0.0
1.0
0.
0.5
0.3
0.6
0..
0.5
0.0
0.0
AUC soare of NIA cander NNGGP)
0.7
0.0
0.9
(b) MIA performance on NNGP and NN models matches, for 90 random training data records, each represented by dot. For MIA on NNs, we evaluate over 50 models on D and 50 models on D'. For MIA on NNGP, we compute the prediction variance of NNGP (on the differ- ing data) given D and D', and evaluate MIA over 5000 sam- ples from the two Gaussian dis- trihutions. Models are trained on "car" and 'plane" images from CI- FAR10.
MIA performance (AUC score)
022 0.5
0.5
Estimate various methods
(a) High correlation between the LOOD and MIA performance. The leave-one-out dataset is a class- balanced subset of CIFAR-10 with size 1000. and D' D U S for differing record S. We evaluate over 200 randomly chosen S (from CIFAR-I0 or uniform distribution over the pixel domain). This indi- cate that LOOD faithfully captures information leakage under MIA. See Appendix C.3 for more discus- sions on this correlation curve.
Mean distance LOOD matches the prediction difference after leave-one- out retraining of NN models, for 90 randomly chosen differing data (each represented by dot). Dataset and model setups are the same as Fig- ure [b] As comparison, we plot the influence function estimates (Koh and Liang [10]) and TracIn scores (Pruthi et al.(18]). We normalize all esti- mates to [o, 1]. report Pearson's correlation and p-value in the legends.
and
Figure 1: Validation of our analytical framework (based on LOOD and mean distance LOOD for NNGP), according to information leakage measure (the performance of membership inference at- tacks) and influence definition (prediction difference under leave-one-out retraining).
OUR ANALYTICAL FRAMEWORK FOR ESTIMATING LOOD
Prior approaches for estimating leakage and memorization are heavily based on experiments, thus suffering from (a) approximation error (e.g., first-order Taylor expansion and non-exact Hessian approximation); (b) modelling error (c.g., suboptimal MIAs for quantifying leakage); (c) high com- putation cost (e.g., training many models for accurate estimation of memorization); and (d) experi- mental instability. To address this, we propose an analytical method that estimates LOOD accurately and efficiently, by modelling the prediction distribution of neural networks as Gaussian process.
Gaussian process. To analytically capture the randomness of machine learning on model out- puts we use Gaussian Process (GP). A GP f ~~ GP(u, K) is specified by its mean function uxx) = [x][( and covariance kernel function K(x, x²)= E [[f(x) =(x))(f(xx) - =xx')) for any inputs x and x' (Williams & Rasmussen 2006). The mean function is typi- cally set to zero, under which the kernel function K completely captures the structure of a GP. Some commonly used kernel functions are isotropic kernels and correlation kernels (Appendix D.2). Given training data D) with noisy labels yn
S(D) + o2),, the posterior prediction distribution of GP on a query data Q follows multivariate Gaussian (Q) ~ N(HD " (Q), DD.." (Q)) with analytical mean pDD.a
KoD + o211)-10 and covariance matrix Ep., (QqQ))))----(---- KQq Kon KDD + - K (by GP regression). Here for brevity, we denote Kxz = (K(x,, 2,),,, as the kernel matrix between X and Z.
Neural Network Gaussian process and connections to NN training. Prior works (Lee et al. 2017 Schoenholz al., 2017 2016) have shown that the output of neural networks on a query converges to Gaussian process, at initialization as the width of the NN tends to infinity. The kernel function of the neural network Gaussian process (NNGP) is given by K Kxx,x)) = Ee-rrior [(fo (x), fx)))) where 0 refers to the neural network weights (sampled from the prior distribution), and fo refers to the prediction function of the model with weights A. In this case, instead of the usual gradient-basee training. we can analytically compute the Gaussian Process regression posterior distribution of the network prediction given the training data and query inputs. See Appendix B for more discussions on the connections between NNGP and the training of its corresponding neural networks.
4

---

Published as a conference paper at ICLR 2024
LOOD for NNGP effectively captures leakage in practical NN training. Under Gaussian process (GP) modelling, we can analytically compute and analyze LOOD on the GP models associated with a given neural network (i.e.. NNGP). For the underlying NNGP models, we first investigate whether the LOOD accurately reflects the level of information leakage in the idealized leave-one- membership inference attack (ee et al. 2022] Section 4.5). Specifically, the leave-one-out MIA distinguish between two prediction distributions and of models trained on leave-one-out datasets that only differs in one record S. In Figure Ia we indeed observe strong correlation between LOOD and MIA performance. This concludes that LOOD effectively reflects information leakage in NNGP. We also empirically validate whether the leakage under NNGP models accurately reflects the leakage in neural networks trained using SGD. In Figure lb we observe strong correlation between the MIA performance on NNGP models and the MIA performance on neural network models. This suggests that NNGP captures the leakage in the corresponding neural networks.
Mean distance LOOD for NNGP agrees with memorization in practical NN training. We further investigates whether our analytical framework allows accurate estimates for memorization (self- influence). In Figure Ic we show the estimates given by different methods (y-axis), and compare it with the actual prediction difference (ı-axis) between 100 NN models trained on each of the leave-one-out datasets, over 50 randomly chosen differing data. That is, the r-axis is exactly the memorization or self-influence as defined by (Feldman 2020 Page 5) and (Feldman & Zhang, 2020 Equation 1), and serves as a ground truth baseline that should be matched by good influence estimations. In the y-axis, we show the normalized mean distance LOOD (under NNGP), influence function (Koh & Liang (017)) and TracIn estimates (Pruthi et al.) 2020). We observe that mean distance LOOD has the highest correlation the actual prediction difference among all methods.
Analytical LOOD allows improved computational efficiency. One significant advantage of our method is the efficiency of computing analytic expressions. In Figure [c we observed over 140x speed up for estimating leakage and influence using our framework versus empirically measuring it over retrained models is performed in the literature YYe et al. 2022, Feldman & Zhang 2020 This allows us to efficiently answer many advanced questions (that are previously computationalyy expensive or infeasible to answer via experimental approaches), such as: Where, i.e., which query Q, does the model exhibit the most information leakage or influence of its training data? How do different model architectures impact influence? We discuss these questions in Section
OPTIMIZING LOOD TO IDENTIFY QUERY WITH MAXIMUM LEAKAGE
LOOD (6) measures the information leakage of a model about its training data S when queried on a query data point Q. For assessing the maximum possible leakage, we need to understand which query Q should the attacker use to extract the highest possible amount of information about S. This question is closely related to the open problem of constructing a worst-case query data for tight privacy auditing via MIAS (Steinke et al.) 2024 Jagielski et al. 2020, Nasr et al. 202T, Wen et al.) 2022 Nasr et al. 2023). However, evaluating leakage via MIAS can be computationally expensive due to the associated costs of retraining many models on a fixed pair of leave-one-our datasets. To address this, we will show that estimating LOOD under GPs offers an efficient alternative for answering this question. In this section, we consider single query (IQ) = 1) and single differing point ([S) = 1) setting and analyze the most influenced point by maximizing the LOOD:
arg max LOOD(LOOOOQ D. D') = dist(fp..=(Q)..f... Q
(2)
We theoretically analyze and empirical solve the LOOD optimization problem ( Our analysis also extends to the multi-query (IQ) 1) and group of differing data ([S) > 1) setting (Appendix
Analysis: querying the differing data itself incurs stationary LOOD (information leakage). We
first prove that for a wide class of kernel functions, the differing point is stationary solution for (2). i.e., satisfies the first-order optimality condition. In the following theorem, we prove that if the kernel function satisfies certain regularity conditions, the gradient of LOOD at the differint point zero, i.e., querying the differing incurs stationary LOOD over its local neighborhood.
'Evaluating the MIA and prediction difference under leave-one-out retraining in Figure 1b took 100 GPU hours for training 50 90 FC networks. For the same setting, estimating L.ODD only took 40 GPU minutes.
5

---

Published as a conference paper at ICLR 2024
Theorem 4.1. Let the denle be K. Assume that (i) (xx, (2,,2))) 2: for all training data i.e., all training data are normalized under kernel function K; (i) 0K(w,x') d7. -='=== 0 for all training data x. L.e, the kernel distance between two data records and x' is stable when they are equal. Then, the differing data is a stationary point for LOOD as a function of query data, as follows
VoLOOD(fD..< (Q) (Q)) lQ=s= 0,
where by definition we have that V (gL.00D/0..0(11lll....ooo = -(1 (1- =(12) I(Q)
-QQ
QQ (Ilx(Q) (Q)1 (') + -
The full proof is deferred in Appendix D.1 D.1 Theorem 4.1 proves that as long as the kernel function K satisfies weak regularity conditions (which hold for commonly used RBF kernel and NNGP kernel on sphere as shown in Appendix D.2). the first-order stationary condition of LOOD is satisfied when the query equals differing data. This stationarity is a necessary condition for maximal leakage, and suggests that the differing point itself could be the query that incurs maximal leakage.
differ data LOOD 0.19717
opt query
differ data
opt query
LOOD 0.19717
LOOD 0.13503
LOOD 0.00089
Frequency
200
100
0.000
0.025 0,050 0,075
ssim score: 0.37212
ssim score: -0.12516
(c) Prediction similarity in KL (Balle et al. 2022 Section V.B.) between difter data & opt query.
(a) Exact reconstruction
(b) Color-fliped reconstruction
Figure 2: Examples of optimized query after LOOD optimization, given all 'car' and 'plane' images from CIFARIO as training dataset, under NNGP for 10-layer FC network with ReLU activation. We show in (a) the optimized query gets close to the differing point in LOOD; and in (b) the optimized query converges to a query with significantly lower LOOD than the differing data, yet they visually resemble each other. In (c) further statistically check the prediction similarity between query and differing data in KL divergence [Balle et al. 2022 Section V.B.) across 500 runs with randomly chosen differing data (see Figures
18 and 19for all the images for these runs).
Experimentally Optimizing LOOD enables a Data Reconstruction Attack. To further understand what query Q induce maximal leakage about training data S. jective (2). Interestingly, we observe that the optimized query by (2 generally recovers the differing data with high visual precision across random experiments: see Figure 2a Appendíx DD3 and Fig- ure 1419 for examples under RBF and NNGP kernels. The optimized query also tends to have very similar LOOD to the differing data itself (see Figure 2c even when there are more than one differing data, i.e., scave-one-group-ee
Adam on the non-convex ob-
setting (see Figure 3 only in its right-most column does the optimized single query have slightly higher LOOD than the differing record that it recovers). We remark that, possibly due to the nonconvex structure of LOOD, cases exist where the optimized query has significantly lower LOOD than the differing data (Figure 2b). Noteworthy that even in this case the query still resembles the shape of the differing data. This shows the significant amount of information leaked through the model's prediction on the differing data itself. Such results show that LOOD can be leveraged for data reconstruction attack in setting where the adversary has access to the prediction distribution of models trained on leave-one-our datasets. This setting aligns with prior works (Balle al. 2022 Guo et al. 2022) that focus reconstruction attacks under the assumption that the adversary possesses knowledge of all data points except for one.
Our reconstruction via optimizing LOOD methodology extends to the group setting. Figure demonstrates that even when the attacker does not know a group of data points, they can successfully
We tried several image similarity metrics in the reconstruction literature, e.g., per-pixel distance (Carlini 2023 Section 5.1), SSIM (Haim et al. 2022 Figure 1) and LPIPS (Balle et al. 2022 Section V.B)
et al.
but they do not reflect the visual similarity in our setting. This is possibly because LOOD optimization only
recovers the highest amount of information about the differing data which need not he the exact RGB values.
It is an open prohlem to design better image similarity metrics for such reconstructions beyond RGB values.

---

Published as a conference paper at ICLR 2024
Muiti-quary 0/3 LOOD: 0.002
Multi-query 1/3 LOOD: 0.056
Multiquaryy 2/3 LOOD: 0.029
Single query LOOD: 0.004
Differing data 0/3
Differing data 1/3
Differing data 2/3
LOOD: 0 0.029
LOOD: 0,034
LOCD: 0,056
Correct
Trup
Multi-query D/3
Multi-query 1/3 0.029
Multi-qurry 2/3
Single query
0.034
0.056
Figure 3: Query optimization (data reconstruction) by Seave-one-group-00 distinguishability. when there are three differing records (the three leftmost images). Above each differing record, we show whether is predicted correctly by the NNGP model trained without it. The three middle images are the optimal queries in the multi-query setting, and the rightmost image is the optimal single query; each row presents the results for a different random query initialization. Observe that differing images with highest LOOD (and for certain initialization even all differing images) can be recovered.
reconstruct the whole group with high visual similarity. Additionally, by examining the frequency of L.ODD optimization converging to a query close to each data point in the differing group (across optimized queries in multiple runs), we can assess which S's are most vulnerable to reconstruction (Appendix[E.2) Reconstruction under the save--ne-group-ouu setting mimics a practical real world adversary that aims to reconstruct more than one training data. In the limit of large group size that equals the size the training dataset, such leave-one-group-one setting recovers one of the difficult reconstruction attack scenario studied in (Haim et al. 2022), where the adversary does not have exact knowledge about any training data. Therefore, extending our results within the leave-one- seave-one-groupoow setting to practical reconstruction attacks presents a promising open problem for future exploration.
Experimental maximality of leakage when query equals differing data for NNGPs and NNs. To complement Theorem and explain the success of our reconstruction attack, we experimentally investigate whether querying on differing point itself incurs maximal LOOD. We observe many em- pirical evidences that support this hypothesis, e.g.. for one mie-dmmeniionl toy dataset in Figure 4a In Figure 4b and its caption, we further observe that querying on differing data S itself consistently incurs larger LOOD than querying
random perturbations of S for CIFAR1O 10 under NNGP. Ap- pendix [ also support this empirical maximality of LOOD under multiple queries and a group of differing data (Appendix E.. Finally, Figure 44c and its caption show that MIA performance on NN models is generally the highest when queries are near the differing data. On the other hand, we note that in practice deeendeenn normalization is often used prior to training. In such a setting there exists optimized query with higher LOOD than the differing data itself- see Appendix D.4
5 OPTIMIZING LOOD TO IDENTIFY THE MOST INFLUENCED POINT
Mean distance LOOD (Definition C. measures the influence of training data S on the prediction at query Q. Thus we can use mean distance LOOD to efficiently analyze where (at which Q) the model is most influenced by the change of one train data S. This question shed light on the fundamental question of which prediction influenced the most by (adversarial) training data (Koh & Liang 2017) (and from there connections to robustness against data poisoning Steinhardt et al. (017)).
We now use mean distance LOOD for a more fine-grained analysis of influence and memorization within our framework. In Proposition 5.1 we analyze the different behaviors of mean distance LOOD, based on where the differing point is located with respect to the rest of the training set.
Proposition 5.1 (First-order optimality condition for influence). Consider dataset D, differing point S, and a kernel function K satisfying the same conditions of Theorem 4.7 Then, we have that
V (Q) lQ=s= -a 21 KSD KDS)(US KSDM yD) KSDM KDs;
7

---

Published as a conference paper at ICLR 2024
0.6
Training dataset
1.3
differ S -2.5
MIA performancc
differ $-0
3.3
1033
0.4-
differ $ 2.5
2.5
0.8-
0.3
0.2
8.
0.0
Ceury
perturbation scale x
perturbation scale
(a) LOOD under RBF
(b) LOOD under NNGP
(c) MIA performance on NNs
Figure 4: Empirical maximality of information leakage when query equals differing data. Figure (a) uses a toy dataset D = (ar, y) with x E [-5, 5). sin( and computes LOOD for all queries the data domain. Figure (b), (c) use "car' and 'plane' images in CIFAR-I0. 10, and evaluate leakage on perturbation queries {Q S + R E [-50, 50] of the differing data S along random direction R E Loc unit ball. For five randomly chosen differing data, we show in (b) and (c) that LOOD and MIA performance the experiment for 70 randomly chosen differing data, and observe the scale perturbation with maximal information leakage is x 0.000 0.000 for NNGPs, and x 2.375 + 5.978 for NNs.
visually maximal around
repeat O. For statistical pattern,
where MQ) the mean distance LOOD C.1 KSD, KDS, KDD are kernel matrices as defined KsDM Ds o².
Section KSD = 8Q KQD lQ=s, MD KDD + o²T, and a
proves that when S is far from the dataset D (i.e., KSD se 0) or close to the dataset D (i.e., ys M yn. that S is perfectly predicted by models trained on D), the mean distance LOOD gradient VQ) lQ=s is close to 0 (i.e., the differing data S has stationary influence on itself). When S is far from the remaining training dataset D and the data labels are bounded (which is the case in a classification task), we further analyze the Hessian of mean distance LOOD and prove that differing data S has locally maximal influence on itself (Lemma H:4). Between the two cases, when the differing data is neither too close nor too far from the remaining dataset, Proposition 5.1 proves that (Q) le=s O, i.e. the differing data S does not incur maximal influence on itself. To give intuition, can think of the Mean LOOD as the result of an interaction of two forces, a force due to the training points, and another force due to the differing point. When the differing point is neither too far too close to the training data, the synergy of the two forces creates region that does not include the differing point where the LOOD is maximal. See Figures 13 for experimental illustrations of the behaviors of mean distance LOOD.
Proposition 5.1 p.oves
and
EXPLAINING THE EFFECT OF ACTIVATION FUNCTIONS ON LOOD
LOOD under NNGP enjoys analytical dependence on model architectures, thus enabling us to effi ciently investigate questions such as: how does the choice of activation function affect the magnitute of information leakage? Answers to such questions would provide guidance for how to train models in more privacy-preserving preserving manner. To theoretically analyze how the choice of activation function affect the magnitude of LOOD, we will leverage Proposition G.T in the Appendix; shows that for the same depth, smooth activations such that GeLU are associated with kernels that are farther away from a low rank all-constant matrix (more expressive) than kernel obtained with non-smooth acti- vations, e.g. ReL.U. Therefore to understand the effect of activation, suffices to analyze how the low-rank property of kernel matrix affects the magnitude of LOOD. Thus, we establish the following proposition, which shows that LOOD small if the NNGP kernel matrix has low rank property.
Proposition 6.1 (Informal: Low rank kernel matrix implies low LOOD). Let D and D' SUS be an arbitrarily fixed pair of leave-one-out datasets, where D contains n records. Define the function a). Let amin aegmingeahh(o and thus h(Omiin) quantifies how close the kernel matrix is to a low rank all-constant matrix. Assume that amin > O. Then, there exist constants An, B such that
h((a)) supa,''ED KK (x, x') (x,z²)
max
(2) (Q)) S (a + -nn"
Thus the smaller h((mmin) is, the smaller LOOD could be. See Gor formal statement and proof.
8

---

Published as a conference paper at ICLR 2024
Proposition 6. proves that the LOOD (leakage) worst-case query is upper bounded by term h(amin) that is small if the kernel matrix is close to a low-rank all-constant matrix. Intuitively, this happens because the network output forgets most of the information about the imput dataset under low rank kernel matrix. By combining Proposition and Proposition we conclude that smooth activations induce higher leakage than non-smooth activations. This is consistent with the intuition that smooth activations allow deeper information propagation, thus inducing more expressive kernels and more memorization of the training data. Interestingly, recent work (Dosovitskiy al. 202T) also showed that GeLU enables better model performance than ReLU even in modern architectures such as ViT. Our results thus suggest there is crivacy-aceuraye tradeoffs by activation choices.
Experimental results on NNGP validates that GeLU activa- tion induces higher LOOD than ReLU activation. We now empirically investigate how the activation function affects the LOOD under NNGP. We evaluate NNGP for fully con- nected neural networks with varying depths {2, 4, , ,10,, trained on 'plane' and 'car' images from CIFARIO. com-
Her-record leakage under GELU
pute LOOD over 500 randomly chosen differing data, and observe under fixed depth, the LOOD under ReLU is more
y-x
NN mocals
NNGP
than 1.1x x higher than LOOD under GeLU in 95% cases. We
visualize randomly chosen 20 differing data in Figure [S] For
Per-record leakage under ReLU
0.3
each fixed differing data, also observe that the gap be-
Figure 5: Empirical validation of our analytical results in of Section 6 that per-record information leakage is higher under GELU activation than under ReLU, for both NNGPs and NN models. We evaluate per- record leakage with membership in- ference attack performance on mod- els trained on leave-one-out datasets. Dataset contains 'car' and 'plane' images from CIFAR-10. The NN model is fully connected network with depth 10 and width 1024.
tween LOOD under ReLU and LOOD under GeLU grows with the depth. This is consistent with the pattern predicted by our analysis in Proposition
G.I. i.e., GeLU gives rise to higher leakage in LOOD than ReLU activation, and their gap increases with depth. It is worth noting that our experiments for the LOOD optimization are done without constraining the query data norm. Therefore, our experiments shows that Proposition G. could be valid in more general settings.
Experimental results on NNs validate the effect of activa- tion choice on leakage. We further investigate whether leakage (measured by MIA performance on cave-one-our retrained models) for deep neural networks are similarly af- fected by activation choice. Figure 5 shows that DNNs with GeI.U activation incurs higher per- record leakage (MIA performance in AUC score) compared to DNNs with ReI.U activation, for 20 randomly chosen differing data. This is consistent with analytical results (Proposition G. [ and 6.I), and aligns with prior empirical observations Wei et al, (2020); Haim et al. (2022); Shamsabadi al. (2023) that data reconstruction is harder under ReLU activation (than its smooth counterparts).
7 CONCLUSIONS
We show that the estimations of information leakage, influence, and memorization are intrinsically the same problem of estimating leave-one-out distinguishability (LOOD). We propose one analyt- ical framework for accurately and efficiently computing LOOD in machine learning via Gaussian process modeling. This framework facilitates more efficient and interpretable answers to existing questions and enables us explore new questions about how a data point influences model predic- tions. Notably, the analytical nature of LOOD enables performing optimization to identify predic- tions that leak the most information about each training data record (Section which enables exact reconstruction of each training data.
Future Works An interesting direction is to LOOD to analyze how other factors affect infor- mation leakage, such as the training data (distribution) and architecture choices. It is also interesting extend our framework for estimating other concepts closely related leave-one-out distinguisha- bility, such as Shapley value (Shapley et al. 1953 in the literature of data valuation. X
SShapley value is linear combination of LOOD all subsets of the training dataset. Thus, in certain scenarios e.g., submodular utility, L.OD regarding the complete dataset is lower hound for Shapley value.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The authors would like to thank Martin Strobel for the help in reproducing influence function, and thank Yao Tong. Tuan-Duy H. Nguyen and Yaxi Hu for helpful feedback on earlier drafts of the paper. The work of Reza Shokri was supported by the Asian Young Scientist Fellowship 2023, and the Ministry of Education, Singapore, Academic Research Fund (AcRF) Tier 1.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in neural information processing systems, 31, 2018.
Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1138-1156. IEEE, 2022.
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. arXiv preprini arXiv:2301. 13/88, 2023.
Jinshuo Dong, Aaron Roth, and Weijie Su. 1rXiv::905.02383. 2019.
Gaussian differential privacy. arXiv preprint
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265-284. Springer, 2006.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211 407, 2014.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
Chuan Guo, Brian Karrer, Kamalika Chaudhuri, and Laurens van der Maaten. Bounding training data reconstruction in private (deep) learning. In International Conference on Machine Learning, pp. 8056-8071. PMIR, 2022.
Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. arXiv preprint 2XXx:2206.007588 2022.
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In International Conference on Machine Learning, 2019.
Soufiane Hayou, Eugenio Clerico, Bobby He, George Deligiannidis, Arnaud Doucet, and Judith Rousseau. Stable resnet. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statisties, volume 130 of Proceedings of Machine Learning Research, pp. 1324-1332. PMLR, 13-15 Apr 2021.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nagp and ntk for deep attention networks. In International Conference Machine Learning, pp. 4376- 4386. PMLR, 2020.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems, 33: 22205-22216, 2020.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference machine learning, pp. 1885-1894, PMLR, 2017.
Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the accuracy of influence functions for measuring group effects. Advances in neural information processing systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2024
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-886 86, 1951.
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 1:X.iv1111.001660 2017.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. arXiv preprint (tti:22210110066 2022.
Yizhang Lou, Chris Mingard, and Soufiane Hayou. Feature learning and signal propagation in deep neural networks. In International Conference Machine Learning, pp- PMLR, 2022.
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer foundatiin sympo- sium (CSF), pp. 263-275. IEEE, 2017.
Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 866-882. IEEE, 2021.
Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tramèr, Matthew Jagielski, Nicholas Carlini, and Andreas Terzis. Tight auditing of differentially private machine learning. arXiv preprint arXiv:2302.079555 2023.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Jerzy Neyman and Egon Sharpe Pearson. Ix. on the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing 1933.
Papers of Mathematical or Physical Character, 23 (314677776)
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. arXiv preprint arXiv: arXiv:1810.051488 2018.
Daniel S. Park, Jaehoon Lee, Daiyi Peng, Yuan Cao, and Jascha Sohl- Sohl-Dickstein. Towards nngp- guided neural architecture search, 2020.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 11920-19930, 2020.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information 2016.
propagation. arXiv preprint
S.S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. Deep information propagation. In International Conference Learning Representations, 2017.
Ali Shahin Shamsabadi, Jamie Hayes, Borja Balle, and Adrian Weller. Mnemonist: locating model parameters that memorize training examples. In Uncertainty in Artificial Intelligence, pp. 1879- 1888. PMLR, 2023.
Lloyd S Shapley et al. A value for n-person games. 1953.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Jacob Steinhardt, Pang Wei W Koh, and Percy $ Liang. Certified defenses for data poisoning attacks. Advances in neural information processing systems, 30, 2017.
Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2024
Wenqi Wei, Ling Liu, Margaret Loper, Ka-Ho Chow, Mehmet Emre Gursoy, Stacey Truex, and Yanzhao Wu. A framework for evaluating gradient leakage attacks in federated learning. arXiv preprint arXhv:2004.10397 2020.
Yuxin Wen, Arpit Bansal, Hamid Kazemi, Eitan Borgnia, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Canary in a coalmine: Better membership inference with ensembled adversarial queries. XrXiv preprint arXiv:2210.10750, 2022.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning- MIT press Cambridge, MA, 2006.
Greg Yang. Wide feedforward recurrent neural networks of any architecture are gaussian pro- cesses. Advances in Neural Information Processing Systems, 32, 2019.
Greg Yang and Etai Littwin. Tensor programs ivb: Adaptive optimization in the infinite-width limit. arXiv preprint Ar3iv:2308. 008144 2023.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. En- hanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pp. 3093-3106, 2022.
Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks. arXiv e-prints, pp. arXiv-2312, 2023.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint 1221:2212.112388 2021.
13

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
2 LOOD Definition and its Connections to Related Concepts
3 Our Analytical Framework for Estimating LOOD
Optimizing LOOD Identify Query with maximum leakage
Optimizing LOOD to Identify the Most Influenced Point
Explaining the Effect of Activation F Functions on LOOD
7
Conclusions
Table of Notations
15
Additional discussion on the connections between NNGP and NN training
15
Deferred details for Section
15
Mean Distance LOOD
15
KL Divergence LOOD
16
Comparison between Mean Distance LOOD and KL Divergence LOOD
17
Deferred proofs and experiments for Section 4
19
D.I
Deferred proof for Theorem 4.1
19
D.2 Proofs for regularity conditions of commonly used kernel functions
21
More examples for query optimization under NNGP kernel functions
D.4 When the differing point not the optimal query
23
E
Analyzing LOOD for Multiple Queries and Leave-one-grouppout Setting
24
E. LOOD under multiple queries
24
Leave-one-group out distinguishability
25
Optimal query under mean distance and deferred proofs for Section 5
25
Differing point is in general not optimal for Mean Distance LOOD)
25
Differing point is optimal in Mean Distance LOOD when it is far from the dataset
27
The optimal query for Mean Distance LOOD on image datasets
29
Deferred proofs for Section 6
29
14

---

Published as a conference paper at ICLR 2024
A
TABLE OF NOTATIONS
Table Symbol reference
Symbol
Meaning
D
Leave-one-out dataset
D'
Leave-one-out dataset combined with the differing data
Differing data record(s)
8
number of differing data records
Queried data record (s)
number of queries
d
dimension of the data
K(x.x')
Kernel function
K(X, E Rox for two finite
collections of vectors X (z1,...,
(K (Iz, ISiSP 1SiSk
and Z = (Zi...., 7x)
Kxz E Roxk
Abbreviation for K Z) Abbreviation for KDD o²1 Abbreviation for KD' + o²T
MD
M
Abbreviation for mean distance LOOD under query Q and leave-one-out datasets D) and D' Abbreviation for membership inference attack
(())
MIA
B ADDITIONAL DISCUSSION ON THE CONNECTIONS BETWEEN NNGP AND NN TRAINING
Note that in general, there is performance gap between trained NN and its equivalent NNGP model. However, they are fundamentally connected even when there is a performance gap: the NNGP describes the (geometric) information flow randomly initialized NN and therefore cap- tures the early training stages of SGD-trained NNs. In highly non-convex problem such as NN training. the initialization is crucial and its impact on different quantities (including performance) is well-documented [Scheenholz et al. (2016). Moreover, it is also well-known that the bulk of feature learning occurs during the first few epochs training (see (Lou et al. 2022, Figure 2)), which makes the initial training stages even more crucial. Hence, it should be expected that properties of NNGP transfer (at least partially) to trained NNs. More advanced tools such as Tensor Programs Yang & Littwin 2023) try to capture the covariance kernel during training, but such dynamics generally intractable in closed-form.
C DEFERRED DETAILS FOR SECTION 2
C.1 MEAN DISTANCE LOOD
The simplest way of quantifying the LOOD is by using the distance between the means of the two distributions formed by coupled Gaussian processes:
Definition C.1 (Mean distance LOOD). By computing the distance between the mean of coupled Gaussian processes on query set Q, we obtain the following mean distance LOOD.
D.D'
2 (VD.0((Q) o2 (Q)1l
(3)
[KQD + -2 yn - KQD -ocKKKDDDD>+---- +
(4)
where Q is query data record, and # (Q) and prr..a (Q) are the mean of the prediction function When the context is clear, we
for GP trained on D and D' respectively, as defined in Section 2 abbreviate D and D' in the notation and use M(Q) under differing data S to denote M(Q; D') where D' = U
15

---

Published as a conference paper at ICLR 2024
18-
S: sampled
sampled uniform strioution over pixel domain [255]
0.012
distribution over pixel domain [255]
1.-
sampled CIFNR10
sampled from CIFAR10
0.008
1.0
0.8
0.04
0.6
0.4
0.000
02-
0.70
0.75
3.83
0.85
0.70
0.75
0.80
0.85
AuC
0.023
NNGP, semplec from uniform
25-
NNGP, samplled from unifarm strioution over domain [255)
distribution over domain [255)
NNGP, samplec from CIFAR10
20-
NNGP, sampled from CIFAR1O
0.055
1.5
0.000
1.0
0.005
0.-
0.0
3.55
0.60
0.55
0.75
0.65
0.90
4sc
Figure 6: Correlation between the LOOD and MIA success. The leave-one-out dataset D is class- balanced subset CIFAR-10 with size 1000, and D' DUS for randomly chosen differing record S. We evaluate over 200 random choices of the differing data record (sampled from the CIFAR-10 dataset or uniform distribution over pixel domain [255].) .) We also evaluate under different LOOD metrics (mean distance and KI. divergence) and different kernel functions (RBF kernel NNGP kernel for fully connected network with depth 1). We observe that the correlation between KL. and AUC (right figures) is consistently stronger and more consistent than the correlation between mean distance LOOD and auc (left figures).
Example C.2 (Mean distance LOOD is not well-correlatd with MIA success). We show the corre- lations between Mean distance LOOD and the success of leave-one-out membership inference attack (MIA) in Figure (left). We observe that low mean distance and high AUC score occur at the same time, Le., low Mean distance LOOD does not imply low allack success. This occurs especially when the differing point is sampled from uniform distribution over the pixel domain [0, 255).
Another downside of the mean distance LOOD metrie is that solely first-order information is in- corporated; this results in it being additive in the queries: M(Q) = Σm M(2).. That is, Mean distance LOOD does not allow to incorporate the impact of correlations between multiple queries. Moreover, the mean distance LOOD has a less clear interpretation under certain kernel functions. For example, under the NNGP kernel, query with large f norm will also have high Mean Distance LOOD (despite normalized query being the same), due to the homogeneity of NNGP kernel. By contrast, we show below, KL divergence (relative entropy) LOOD mitigates these limitations.
C.2 KL DIVERGENCE LOOD
To mitigate the downsides of Mean distance LOOD and incorporate higher-order information, we use KI. divergence (relative entropy) LOOD.
Definition C.3 (KL Divergence LOOD). Let Q E _gx be the set of queried data points, where d the data feature dimension and q is the number of queries. Let D, D' be pair of leave-one-our datasets such that D' D U S where S is the differing data point. Let fD.0² and fo as coupled Gaussian processes given D and D'. Then the Leave-one-out KL distinguishability between fD.0=
This measured by the AUC of the attacker's false positive rate (FPR) versus his true positive rate (TPR).
16

---

Published as a conference paper at ICLR 2024
and oo queries is as follows,
(IUDD,=(Q11//,,.00000 -2 log [2)
[{(Q] 9 TTT(
(5)
(u(Q) - ,2(1)) "11))))))( p²
(6)
where for brevity, we denote u(Q) = u (Q), (2) , E(Q) En and (Q) = En En (Q) as defined in Section
Example C.4 (KL divergence is better correlated with MIA success). We show correlations between KL Divergence LOOD and AUC score of the MIA in Figure(((iight) We observe that KL Divergence LOOD and AUC score are well-correlated with each other.
On the assymmetry of KI. divergence. In this paper we restrict our discussions to the case when the base distribution in KI. divergence is specified by the predictions on the larger dataset D', which a reasonable upper bound on the KL divergence with base distribution specified by predictions on the smaller dataset. This is due to the following observations:
1. In numerical experiments (e.g., the middle figure in Figure 7 that ((Q)/E((Q) > 1 across varying single query Q. This is aligned with the intuition that the added information the larger dataset D reduce its prediction uncertainty (variance).
2. If (QQ))((()) 2 1, then we have that
(Q)f 2
Proof. Denote E(Q)/E((Q) 2 1, then by Equation (6) for single query Q, we have that
1 <T((D=> = (- (-logr + (
where (Q) is the abbreviation for mean distance LOOD as defined Definition C.1 Similarly, when the base distribution of KL divergence changes, we have that
1 (logr- 1+ 2M(2))
KL
Therefore, we have that
KL(f ( ((0))))((((f.f...)))l(l//....
=
2log r 22
For r 2 1, we have that r r- 2oor 2 and (r 1) 2 1. Therefore, we have that KL(fD. (Q)llf r == ((Q)/E'(Q) 1.
x Z 0 as long as
Therefore, the KL divergence with base prediction distribution specified by the larger dataset is an upper bound for the worst-case KL divergence between prediction distribu- tions on leave-one-out datasets.
C.3 COMPARISON BETWEEN MEAN DISTANCE LOOD AND KI DIVERGENCE LOOD
KL Divergence LOOD is better correlated with attack performance. As observed from Figure 6 KL divergence LOOD exhibits strong correlation to attack performances in terms of AUC score of the corresponding leave-one-out MIA. By contrast, Mean distance LOOD is not well-correlatdd with MIA AUC score. For example, there exist points that incur small mean distance LOOD while ex- hibiting high privacy risk in terms of AUC score under leave-one-out MIA. Moreover, from Figure O (right), empirically there exists a general threshold of KL Divergence LOOD that implies small AUC (that persists under different choices of kernel function, and dataset distribution). This is reasonable
17

---

Published as a conference paper at ICLR 2024
M(C) with 5
(
KLIG) with 5
0.5
argmass ( lo-sl
200)
40-
5 w3.9
0.4
Mrimax.MMi)
zio)
O"
0.3
20
0.1
0.0
0.
0
Figure 7: An example on a one-dimensional toy sine training dataset generated as Appendix F. where each training data is shown as red dot. We denote S as a crafted differing data record that incurs maximum gradient for the mean distance LOOD objective, which is constructed following the instructions in Algorithm []] In the left plot, we observe that the optimal query Q* for mean distance LOOD does not equal the differing point S On the contrary, the optimal query for KI. LOOD (which incurs maximal information leakage) the differing point S* (right plot). This shows the discrepancy between mean distance LOOD and KL LOOD, and shows the inadequacy of using mean distance LOOD to capture information leakage.
since KL divergence is the mean of negative log-likelihood ratio, and is thus closely related to the power of log-likelihood ratio membership inference attack.
Why the mean is (not) informative for leakage. To understand when and why there are discrep- ancies between the mean distance LOOD and KI. LOOD, we numerically investigate the optimal query Q" that incurs maximal mean distance LOOD. Figure 7 we observe that although Q* in- deed incurs the largest mean distance among all possible query data, the ratio log (og()(()/\"(Q)) between prediction variance of GP trained on D and that of GP trained on D', is also significantly smaller at query Q* when compared to the differing point S. Consequently, after incorporating the second-order variance information, the KL divergence at query Q" is significantly lower than the KL divergence at the differing point S despite of the fact that Q* incurs maximal mean distance LOOD. A closer look at Figure middle indicates that the variance ratio peaks at the differing point 8", and drops rapidly dataset. Intuitively, this is because (Q) i.c., the variance at the larger dataset, is extremely small the differing data point S. As the query gets further away from the larger training dataset, >' (Q) then increases rapidly. By contrast, the variance on the smaller dataset Σ(Q) is large at the differing data point S, but does not change a lot around its neighborhood, thus contributing to low variance ra- tio. This shows that there exist settings where the second order information contributes significantly to the information leakage, which is not captured by mean distance LOOD (as it only incorporates first-order information).
the query becomes farther away from the differing data and the training
As another example case of when the variance of prediction distribution is non-negligible for faith- fully capturing information leakage, we investigate under kernel functions that are homogeneous with regard to their input data, Specifically, for a homogeneous kernel function K that satisfifes K K(Ar, x²)= = K(x, Ax') K(( x²) for any real number À E IR we have that
Dp. (XQ)= E> (Q)
OD. (XQ) = X2 >D..,= (Q).
where ²² denotes the variance of the label noise. Therefore, for any À E R, we have that M D. D') X D,)
(9)
KL(fD. (x0))..== =
(10)
Consequently, the query that maximizes Mean distance LOOD explodes to infinity for data with unbounded domain. On the contrary, the KL Divergence LOOD is scale-invariant and remains stable as the query scales linearly (because it takes the variance into account, which also grows with the query scale). In this case, the mean distance is not indicative of the actual information leakage as the variance is not a constant and is nog-neglg
In summary, KL LOOD is a metric that is suiter for the purpose of measuring information leakage, compared to mean distance LOOD. This is because KL LOOD captures the correlation between multiple queries, and incorporates the second-order information of prediction distribution.
18

---

Published as a conference paper at ICLR 2024
D DEFERRED PROOFS AND EXPERIMENTS FOR SECTION 4
D.1 DEFERRED PROOF FOR THEOREM
In this section, we prove that the differing data record is the stationary point for optimizing the LOOD objective. We will need the following lemma.
Lemma D.I. For E IR b R and E R, we have that
A
6
A a A A-1 (
-a-1A-10
(1I)
bT A-1
a
where a Ab.
We proceed to prove Theorem 4.1
Theorem 4.7 Let the kernel function be K. Assume that (i) (xx, x)) for all training data x,
data æ, i.e. the kernel distance between two data records x and x' 2r -='=- 0 for all training stable when they are equal.
i.e, all training data are normalized under kernel function K; (ii) 0K(=x2)
Then, the differing data is stationary point for LOOD as function of query data, as follows
(QLLOD((D.><(Q)1fff))
le=s= 0.
Proof. In this statement, LOOD refers to KL. LOOD Definition entiation of Definition
C.3 therefore by analytical differ-
C.3
we have that
LOOD (Q))
K KL (Q)1l .0 -== (Q))
(12)
aQ
(2)
0
(QQ
E((Q)
iQ
('(Q)
0l( (Q)
('(
aQ
-1 - p2Q) (Q)
(13)
OQ
Note that here E(Q) and (Q) are both scalars, consider single query. We denote the three terms by (Q), F2 (Q), and Fs(Q). We will show that (S) = and F2(5) = (5( which is sufficient conclude. Let start with the first term.
1. (S) 0: Recall from the Gaussian process definition in Section that
Σ(Q) KQq KoD
where MD KDD d² Therefore.
0 Σ(Q) = -22 OKQD Mn¹ KpQ
aQ
aQ
where aKop _ddx= (d is the dimension of Q and n is the number of datapoints in the aQ
@ training set D). A similar formula holds for aq (Q).
From Lemma D.I and by using the kernel assumption that K(S,S) 1, we have that
Kps -1 ODD+
Mp
(14)
1+ MM + -' KDSKSDM -0-
KSD
-a-' MM KDS
(15)
a
where 1 1 2 Ks k 0² + E(S).
19

---

Published as a conference paper at ICLR 2024
Using the fact that M KD's ofen1 where en+1 (0....,0,.1)" E RR* simple calculations yield
E(()) = 1 KsDrMD KD'
= o - o²e M en+1)
0 - oaa
We now have all the ingredients for the first term. To alleviate the notation, we denote by KsD aq KoD lQ=s.
We obtain
aQ (2(0)) lQ=s=
-2KspMM'Kan M (E(S) Mf Kos
+
E'())
E(S)
A key observation here is the fact that the last entry of KSD is 0 by assumption. Using the formula above for M Mp. obtain
M Kps = o² KSD MD ent1
- -'sslsl-a' Mp Kns)
to KDs).
Now observe that a -0) a 0022 02)0 21 0²a -2a-1)-1 which concludes the proof for the first term.
-
2. F2(S) Fs(S) O:
le=s E'(S) The derivative here Let us start by simplifying E2(S) (llu(Q)- is that of (Q), and we have
lQ=s= 2({()) - ((D) yD KSD yDr). all (Q)
aQ
Using the formula of Mo! and observe that the last entry of KSD is zero, we obtain 0 (Q)) - (²Q) lQ=s= 2a (a()) - (ys - KsDMo'vD un) M K
aQ
Moreover, we have that
I(S) pS)) = =KsD - K yD
(16)
=Ksn yn e T (Mr o² o yD
(17)
-Ksn yn ys o² x M yn
(18)
=Ksn yn - ys o² MsnMpl yn a ys)
(19)
- Ksn
(20)
Therefore
F2(S) = 2a -1 (ys KSD M
(21)
Let deal with Fs(S) (u(S) (((S))20=
80 lQ=s- We have that
20

---

Published as a conference paper at ICLR 2024
(E)
lQ=s = (5) 2 KKDD Mp KD's
iQ
- cat1
" Mp KDs)
2ax 20'''0''(S)) 2 2Knn M Kns-
(22)
By plugging Equation 22 and 20 into Fz(S) = (u(S) - (S)) (Q) lQ=s, we prove
that
Fs(S) 2a 10000000---- (ys KsD yo) KsD KDs.
We conclude by observing that o' 1)E"(S)-+ = 0²(0²a- 1)o 1)
a²a aaa--1)1==-
D.2 PROOFS FOR REGULARITY CONDITIONS OF COMMONLY USED KERNEL FUNCTIONS
We first show that both the isotropic kernel and the correlation kernel satisfies the condition of Theorem 4.1 We then show that the RBF kernel, resp. the NNGP kernel on a sphere, an isotropic kernel, resp. a correlation kernel.
Proposition D.2 (Isotropic kernels satisfy conditions in Theorem 4.1g, p. Assume that the kernel func- tion K is isotoopc, i.e. there exists a continuously differentiable function g such that K (xx y) g[|[== - yll) for all x, Assume that satisfies = g g'((z) is bounded 10 and g(0) = 1. Then, K satisfies the conditions of Theorem 4.1
Proof. is straightforward that (1r, x) for all x. Simple calculations yield
0K(r,y) g'([1= vll) y). Fx | yl
0'( v/l is bounded for all and also in the limit + y. Therefore, it is easy to see that ---
By assumption, the term
0K(x, v)
lm=y- 0.
Ox
More rigorously, the partial derivative above exists by continuation of the function 2 - g'(x) 2 near 0+.
Proposition D.3 (Correlation kernels satisfy conditions Theorem Assume that the kernel (w.a)
function K is correlation kernel, i.e. there exists function g such that K K(x,y) v) 9
for
all x, y + 0. Assume that g(1) 1. Then, K satisfies the conditions of Theorem 4.1 |=
Proof. The first condition is satisfied by assumption on g. For the second condition, observe that
0K (x, v) dx
(v
(z. v) #) |
(xc, v)
g
|
| [||a]]
which yields @K(x.))
a=y 0.
We have the following result for the NNGP kernel.
AAttually, only the boundedness zero is needed.
21

---

Published as a conference paper at ICLR 2024
Proposition D.4 (NNGP kernel the sphere satisfies conditions in Theorem Theorem 4.1) The NNGP kernel function is similar to the correlation kernel. For ReLU activation function, has the form
K(x, = | ( (0.11)
This kernel function does not satisfy the conditions of Theorem (.. and therefore we cannol conclude that the differing point is a stationary point. Besides, in practice, NNGP used with normalized data in order to avoid any numerical issues; the datapoints are normalized to have || = r for some fixed r > With this formulation, we can actually show that the kernel K restricted to the sphere satisfies the conditions of Theorem 4.1) However, as one might guess, it is not straighffrward to compute a "derivative on the sphere" as this requires a generalized definition of the derivative on manifolds (the coordinates are not free). To avoid dealing with unnecessary complications, we can bypass this problem by considering the spherical coordinates instead. For x E IR!, there exists unique set of parameters (r, 101, , E Rd s.t.
cos((p1)
22 sin((1) cos(
23 = r sin (in) sin( cos((ps)
Rd-1 = r rin(((1) sin( ... sin((pa-2) cos((pa-)) Fd sin((p1) sin(p2) ... sin((p-2) sin((pd-1).
This is generalization of the polar coordinates in 2D.
Without loss of generality, let us assume that and let (411, (2,1, ...... Pdd-1). In this case, the NNGP kernel, evaluated at (sp.4') is given by
K(pp,00))= 9
Σ
sin( sin(pp)) cos((p:) cos(1p)) + II sin(p,) sin(p/))
with the convention I
Consider the spherical coordinates introduced above, and assume that 1. Then, the NNGP kernel given by K ((<p, sp') satisfies the conditions of Theorem
Computing the derivative with respect & is equivalent to computing the derivative on the Proof.
unit sphere. For / E (1, (1....,d 1}. we have
d-
OK (K(,, 4²) = J(42,
d-1
Σ
sin(),)) sin(pf)) cos((p)) cos(pp) + II
sin (in)) sin({p})
Oep
j=1
where
J((p,(0)= Σ
cos((pt) II sin((p5)
II sin((p))
cos(sp:) cos(yp))
i=|+1
1=1 399
d-1
d-1
[-1
L-1
+ cos((p))
sin
sin();s)
II sin sin()p3})
sin
II sin((v)) sin(pi) cos(pp)
When s0 = we obtain
22

---

Published as a conference paper at ICLR 2024
Differing Data LOOD: 0.02183
Optimized Query
Differing Data LODD: 0.12876
Optimized Query LOOD: 0.12876 AUC: 0.61394
0.02183 AUC: 0.53676
AUC: 0.54633
AUC: 0.62932
(a) A run LOOD optimization under NNGP kernel for fully connected network with depth 5
(b) rum of LOOD optimization under NNGP kernel for CNN with depth 5
Figure 8: Additional LOOD optimization results under NNGP kernels with different architectures. We evaluate on class-balanced subset of CIFAR-10 dataset of size 1000. The optimized query consistently has similar LOOD and MIA AUC score to the differing data record.
cos((p)) d-1
cos(pp)
d-1 sin (s,,)
((,,40)
sin s)n(
cos(12,)²
(in(
((i()
cos(spt)
sin(
sin(pp)
By observing that
Σ
sin(ss
cos(sp;)² +
sin({o,)²
= II 1=1
sin (sn))
1m141
we conclude that (K(p)pp))
1p=pp== 0 for all l.
0ip
D.3 MORE EXAMPLES FOR QUERY OPTIMIZATION UNDER NNGP KERNEL FUNCTIONS
We show the optimized query for NNGP kernel under different model architectures, including fully connected network and convolutional neural network with various depth, in Figure g
To further understand the query optimization process in details, visualize example LOOD
optimization process below in Figure 9
D.4 WHEN THE DIFFERING POINT IS NOT THE OPTIMAL QUERY
In practice data-dependent normalization is used prior to training, such as batch and layer normal- ization. Note that this normalization the pair of leave-one-out datasets results them differing in more than record Then the optimized query data record obtains higher LOOD than the LOOD of the differing points; we show this in Figure 10 thus in such a setting there is value for the attacker optimise the query; this is consistent with results from (Wen et al. 2022) that exploit the training data augmentation.
During inference, the same test record will again go through normalization step that depends on the training dataset before being passed as query the corresponding Gaussian process.
23

---

Published as a conference paper at ICLR 2024
0.020
dureggee
0.015
0.010
0.005
0.000
100
200
300
Per
(a) Query after 0 itera- tions of optimization
(b) Query after 10 itera-
(c) Query after 20 itera-
(d) LOOD of query versus number of iterations
tions of optimization
tions of optimization
Figure 9: Illustration of an example LOOD optimization process at different number of optimization class-balanced subset of CIFAR-10 dataset of size 1000 and NNGP
iterations. We evaluate kernel for fully connected network with depth 5. The query is initialized by uniform sampling from the pixel domain, and starts recovering the differing data record as the optimization proceeds. We plot the optimized queries after 0, 10 and 20 iterations and show the convergence of L.ODD for query with regard to the increasing number of iterations for LOOD optimization.
Differing Data LOOD: 0.08735
Optimized Query
Differing Data LOOD: 0.45951
Optimized Query LOOD: 0.45953
LOOD: 0.08738
(a) Differing data and optimized query run one
(b) Differing data and optimized query run two
Figure 10: Example of LOOD optimization under data-dependent training data normalization
(that uses per-channel mean and standard deviation of training dataset). The optimized query has slightly higher LOOD than differing point, and visually has darker color. Our setup uses 500 training data points of the CIFARIO 10 dataset (50 per class) and NNGP specified by CNN with depth 5.
E ANALYZING LOOD FOR MULTIPLE QUERIES AND LEAVE-ONE-GROUP-OrT SETTING
E.1 LOOD UNDER MULTIPLE QUERIES
We now turn to a more general question of quantifying the LOOD when having a model answer multiple queries {Q1, ..., Q₂) rather than just one query Q (this corresponds the practical setting of e.g. having API access to a model that one can query multiple times). In this setting, a natu- ral question is: the number of queries grows, does the distinguishability between leave-one-out models' predictions get stronger (when compared to the single query setting)? Our empirical obser- vations indicate that the significant information regarding the differing data point resides within its prediction on that data record. Notably, the optimal single query Q = S exhibits a higher or equal degree of leakage compared to any other query set.
Specifically, to show that the optimal single query Q S exhibits a higher equal de- gree of leakage compared to any other query set, we performed multi-query optimization D,,,")) which uses gradient descent optimize the LOOD
maxq..... ....Q........ Q. LOOD(Q,..... across queries, for varying numbers of queries q = 2, 4, We evaluated this setting under the following conditions: NNGP using fully connected network with depths of 2, 4. 6, 8, 10; using 500 random selections lcave-one-out datasets (where the larger dataset is size 10000) sam- pled from the 'car' and "plane" classes of the CIFARIO dataset. Our observations indicate that, in all
24

---

Published as a conference paper at ICLR 2024
evaluation trials, the LOOD values obtained from the optimized m queries are lower than the LOOD value from the single differing data record.
E.2 LEAVE-ONE GROUP OUT DISTINGUISHABILITY
Here we extend our analysis to the leave-one group-out setting, which involves two datasets. D and D', that differ in a group of data points (S,..., S. There are no specific restrictions on how these points are selected. In this setup, we evaluate the model using queries (Q.,... Qq where 2 1. The leave-one group-out approach can provide insights into privacy concerns when adding a group to the training set, such in datasets containing records from members of the same organization or family Dwork et al. (2014). Additionally, this setting can shed light on the success of data poisoning attacks, where an adversary adds set of points to the training set to manipulate the model or induce wrong predictions. Here, we are interested in constructing and studying queries that result in the highest information leakage about the group, i.e., where the model's predictions are most affected under the leave-one group-out setting.
Similar to previous approach, we run maximization of LOOD identify the queries that opti- mally distinguishes the leave-one group-out models. Figure 3 presents the results for single-query (g = 1) setting and multi-query (g 3) setting, with differing group size |S| = 3. We have conducted in total 10 rounds of experiments, where in each round we randomly select a fixed group of differing datapoints and then perform 10 independent optimization runs (starting from different initializationss. Among all the runs, 80% of them converged, in which we observe that the optimal queries are visually similar to the members of the differing group, but the chance of recovering them varies based on how much each individual data point is exposed through a single query (by measur- ing LOOD when querying the point). We observe that among the points in set S, the differing data point with the highest single-query LOOD was recovered 61% of the times, whereas the one with the lowest single-query LOOD was recovered only 8% of the times. This can be leveraged for data reconstruction attacks where the adversary has access part of the train dataset of the model.
F OPTIMAL QUERY UNDER MEAN DISTANCE AND DEFERRED PROOFS FOR SECTION 5
We discuss here the computation of the optimal query under mean distance. This equivalent the question of where the function is most influenced by change in the dataset.
F.1 DIFFERING POINT IN GENERAL NOT OPTIMAL FOR MEAN DISTANCE LOOD
Numerically identifying cases where the differing point is not optimal. We define an algorithm
that finds non-optimal differing points S such that when optimizing the query Q, the resulting query
is different from S. When the differing point S is not the optimal query, the gradient of M (Q) evaluated at S is non-zero. Hence, intuitively, finding points such that //V (Q) lo-s is maximal should yield points S for which the optimal query is not the differing point (as there is room for optimising the query due to its positive gradient). We describe this algorithm in Algorithm I
Algorithm 1 Identifying non-optimal differing points S
Given D and ø. we solve the optimization problem S* argmax V (Q) lQ=s || to identify a non-optimal differing point. (We use gradient descent to solve this problem.)
A toy example of where the differing point is not optimal. To assess the efficacy of Algorithm I
we consider a simple example in the une-dimensional case. The setup is as follows: (1) Data D
(ax, V 1,..., 10) is generated using the following distribution: I; ~ (O 1), y/1 = sin(x::) (2) Noise parameter is given by x² = 0.01. (3) Kerne is given by K (x,x²)
exp( - 22)
(RBF kernel). (4) In Algorithm only Sz is optimized, the label S,, is fixed to S,, = sin S.).
We show in Figure la the gradient of (Q) evaluated at S for varying S, and highlight the point S* (the result of Algorithm| I that has the largest con-stationarity for mean distance LOOD objective. Additionally, if we look at the query with maximal LOOD under differing data S S*, the optimal query Q" that we found is indeed different from the differing point S as shown in Figure TTb
25

---

Published as a conference paper at ICLR 2024
5,4001 lewd
M10) wi 3.889
argmaeal/R6M0) 3.09
argnoox/Mi01
0.4
64-
5.-
01-
-12
00
36
(a) The gradient (Q) evaluated at S.
(b) The mean distance (Q) under S*
Figure 11: We plot the stationary condition indicator function (QM(() lQ=s of LOOD when query Q equals the differing data S, over different values of differing data S with S E [-5, 5]. This allows to Algorithm| T to identify differing point S maxş [Q) lq=s || that has the highest objective (maximal non-stationarity), as shown in Algorithm (where S = 3.899).
In Figure lib validate that under the optimized differing data S the query Q' that maximizes LOOD (over leave-one-out datasets that differ in 5*) is indeed not the differing point S
Let us now look what happens when we choose other differing points S * S*. Figure 12 shows
16
M(Q)
1.50
0.8
0.8
0.6
Training data
argmaxoM(())
1.00
0.6
0.75
0.25
The plot of M(Q) for varying Q and S. We also show the optimal query Q' and the differing point S.
Figure 12:
the optimal query for three different choices of S. When S is far from the training data D (S,, e (-5,5), 5) the optimal query approximately coincides with the differing point When close to the training data (S, 0), the behaviour of the optimal query is non-trivial, but it should be noted that the value of the MSE is very small this case.
Explaining when the differing point is suboptimal for Mean Distance LOOD. We now give theoretical explanation regarding why the differing point is suboptimal for mean distance LOOD when it is close (but not close) to the training dataset (as observed in Figure 12 middle plot). We first give proposition that shows that the gradient of M (Q) large when the differing point close (but not so close) to the training dataset D)
Proposition [.7] Consider dataset D, differing point S, and kernel function K satisfying the same conditions of Theorem 4.1 Then, we have that
VQM lQ=s= a 21 KsDM KDS) K3D " KDS, (23)
where MQ) denotes the mean distance LOOD as defined Definition C.I
KSD E RIx[D is 8
the kernel matrix between data point S and dataset D, KDS = KJ KSD = कe Mp = KDD + 021, and a = KSDM5 KDS + o?.
KΩD la=s.
Proof. The left-hand-side of Equation 23 is ((Q) = aq (1) by definition the
mean distance LOOD M (Q) in Definition C.I This term occurs in the term (2(Q) in Equation [13
in the proof for Theorem 4.1
Specifically, we have (Q) = F2(S) E' (S). The detailed
expression for (S) is computed in line Equation (21) during the proof for Theorem 4.1 Therefore,
by multiplying (S) in Equation (21) with M' (S). we obtain the expression for V VoM(Q. in the
statement Equation
26

---

Published as a conference paper at ICLR 2024
As result, it should be generally expected that (Q) la=s 0. We can further that when far from the data in the sense that KSD ze 0. the gradient is also close to 0. In the opposite case where S is close to D, e.g. satisfying Us = KynMp yn. then the gradient is also close to zero. Between the two cases, there exists a region where the gradient norm is maximal.
F.2 DIFFERING POINT IS OPTIMAL IN MEAN DISTANCE LOOD WHEN IT IS FAR FROM THE DATASET
Explaining when the differing point is optimal for Mean Distance LOOD. We now prove that the differing point is (nearly) optimal when it is far from remaining dataset. We assume the inputs belong some compact C' IR
Definition F1 (Isotropic kernels). We say that kernel function K C C + is isotropic is there exists function g R I such that K (x,,²)) g(1 ='Il) for all x, x'' C.
Examples includes RBF kernel, Laplace kernel, etc. For this type of kernel, we have the following result.
Lemma F.2. Let K be an isotropic kernel, and assume that its corresponding function g is differen- tiable on C (or just the interior of C). Then, for all Q C, we have that ( S (d) d' (Q,
max(d(Q, (), D)),
where
D= DD(D Dah. D' = {
d'(Q. D) = [ax g Dill) and (Q. 0(Q -\))
d( max 1a(1QQ - Dill) and d(Q, s(lQQ 51))
<<> 0 a constant that depends only on D.
Proof. Let Q E C. For the sake of simplification, use the following notation: = pn + o21)~tyn, yn, and u' (K + -²11)-tyn
Simple calculations yield
vQQ) = (KQn'n KoDu) Inu (ul u4) Vo)o + wh Voko.
The first term is upperbounded by |1//n] uQI D) + |n/ IdQ.) where "/m] is the vector containing the first n coordinates of u', and u, refers to the tt coordinate of the vector u.
Now observe that ||u']] is bounded for all S, provided the data labels y are bounded. For this, it suffices see that [' a² T2 - -- because of the positive semi-definiteness of K KD
Moreover, have VoKØn, g (1QQ-1QQ1)) D. 1 B- which implies that VoKen, Ilg'(IQ D=ll). Therefore, using the triangular inequality, we obtain
Σ (1l - KonD, + th VoKo. 1=1
< [m[m] - =ulldd(QQ, + 1 -S1))
The existence of Ç is trivial.
From Lemma F2 we have the following corollary.
Corollary F.3. Assume that sup g(z) oc, limys o gz(z) (2)= and (0) 0. Then, provided that C is large enough. for any D = {Di. (D...., Dn) and 0. there exists point S, such that if
we define D' D U (S}, then (S)
That if we choose S far enough from the data D, then the point S is close stationary point.
27

---

Published as a conference paper at ICLR 2024
Although this shows that gradient is small at S (for S far enough from the data D). it does not say whether this is close to a local minimum or maximum. determine the nature of the stationary points in this region, an analysis of the second-order geometry is needed. More precisely, like to understand the behaviour of the Hessian matrix in a neighhourhood of S. In the next result, show that the Hessian matrix is strictly negative around 5 (for S sufficiently far from the data D), which confirms that any stationary point in the neighbourhood of S should be a local maximum.
would
Lemma E.4 (Second-order analysis). Assume that g(0) O, lim1-~ 9'(=) L_, where L_ is a negative constant that depends on the kerne function, and that litm--000 2 'g((z)) = O. Then, provided that C' is large enough, for any D = (Di..... Dn) and e 0, there exists point S. such that we define D' = DU(S), then there exists >> satisfying: for all Q E B(S, n), (Q) < 0.
negative constant that depends on the kernel function, and that limz-to 0=)) = 0. Then, provided that C' is large enough, for any {Di....; Da} and 0, there exists point S, such that if we define D' = (U{S), then there exists satisfying: for all Q E B(S, MQ) X 0 is negatively definite matrix.
LEmma F.4 Assume that g(0) > O, lim (2) L_, where 1_
Proof. Let Q E C. vet) = KQD - Kq The Hessian is given by
VQr(Q)Vqv(()) (Q) v(Q) T
where Vze(Q) Σ (ul u.) VoDDD + thu+1 Vo and VQu(Q) Ex VoVoD + 2/ VoKg.s. By Lemma F.2, we have that for far enough from the dataset D, we have that \Vu((S) s. Therefore, the Hessian at S simplilfies to the following.
((04 u, VoKoo + +ua V2Ke.) + +e.
Vole=s S v(Q)
For i E [n]), simple calculations yield
g'(1Q - D.ll), V2KeD, = D.) (Q - (.)(Q - D.)
IQ D.Il 1,
I|Q - Dill2
where ( = Dill) -g'' x((1Q-D.II1. 1Q-Dil
Notice that u = Us - KSDu. Without loss of generality, assume that ys > 0. For S far enough from the dataset D. we then have u/ > 0 and v(S) ys > 0. Let now what happens to the Hessian. For such S, and for any e > 0, there exists a neighbourhood B(S, 11). > O, such that for all Q E B(S, 17), we have D((1Q-D:11) |1Q-D.11
e, and Ja(Q, D.)1 < This ensures that KoD. S 2c where |-|| denotes the Frobenius norm of a matrix. On the other hand, we have v3 Kq.s lim g'(=) = I. It remains to deal with uQ)). Observe that for S far enough from the data D. and Q close enough to S, (QQ) is close to v(S) R ys > 0. Thus, taking small enough, there exists 7) satisfying the requirements, which concludes the proof.
Lemma F.4 shows that whenever the point S is isolated from the dataset D. the Hessian is negative in a neighbourhood of S, suggesting that any local stationary point can only be local maximum. This is true for isotropic kernels such as the Radial Basis Function kernel (RBF) and the Laplace kernel. As future work, Lemma F2 and Lemma F44 have important computational implications for the search for the 'worst" poisonous point S. Finding the worst poisonous point S is computationally expensive for two reasons: we cannot use gradient descent to optimize over S since the target function is implicit in S, and for each point S, matrix inversion is required to find maxQ (Q). Our analysis suggests that if S is far enough from the data D, we can use (S) as a lower-bound estimate of maxq MQ).
28

---

Published as a conference paper at ICLR 2024
F.3 THE OPTIMAL QUERY FOR MEAN DISTANCE LOOD ON IMAGE DATASETS
As we concluded in the analysis in Section only when the differing datapoint is far away (in some notion distance) from the rest of the dataset the optimal query (in mean distance LOOD) the differing datapoint. In Figure l1b we successfully identified settings on toy dataset where the differing point not optimal for mean distance LOOD. We now investigate whether this also happens image dataset.
Differing 0.03565
Optimizee Query Mean Distance: 0,00702 0.75635
Optimized 0.00211
Diszo 00087
AUC: 0.7E373
AJC. 0.64253
0.59861
(a) Differing data and optimized query in run one
(b) Differing data and optimized query run two
(c) AUC gap on Q, 50 trials
Figure 13: Results for running single query optimization for Mean distance LOOD on Mlas-balanced subset of CIFAR-10 10 dataset of size 1000 and RBF kemnl. The optimized query has higher mean distance LOOD than the differing data record, despite having lower MIA AUC score. Across the 50 trials that we evaluated, more than 50% of trials result in an optimized query that has lower AUC score (by more than 0.025) than the differing data.
In Figure 13 we observe that the optimized record is indeed similar (but not identical) to the differing record for majority fraction of cases. Therefore, also in image data the removal of certain datapoint may influence the learned function most in a location that is not the removed datapoint itself. However, the optimized query for Mean distance LOOD still has much smaller MIA AUC score than the differing data. This again indicates that MSE optimization finds suboptimal query for information leakage, despite achieving higher influence (mean distance).
G DEFERRED PROOFS FOR SECTION 6
Our starting point is the following observation in prior work (Schoenholz et al., 2017; Hayou al., 2019 2021), which states that the NNGP kernel is sensitive to the choice the activation function.
Proposition G.1. Assume that the inputs are normalized, ||=||| = and let K KNNGr denote the NNGP kernel function of an MLP of depth I, 2 1. Then, the for any two inputs, we have the following
[Non-smooth activations] With ReLU: there exists constant such that 1k 1K
((L-2)).
[Smooth activations] With GeLU, ELU, Tanh: there exists constant al OL-
such that
k NGGCC x²)
As result, for the same depth L, one should expect that NNGP kernel matrix with ReLU activation is closer to a low rank constant matrix than with smooth activations such as GeLU, ELU, Tanh 12
For proving Proposition 6.1 we will need the following lemma.
Lemma G.2. Let A and B be two positive definite symmetric matrices. Then, the following holds
||- - B S max( Amir (A) >miin (mii(B)->)lle BlI2,
where Amin(A) (A) is the smallest eigenvalue of A, and ||-112 is the usual 2-norm for matrices.
Proof. Define the function f(t) = -ttAA + (1)+)))-1 for t E [o, 1]. Standard results from linear algebra yield
(t) = (((t)(AB))ft)
1he different activations are given by GeI.U(z)) = ( where & is the c.d.f of standard Gaussian variable, FLU(z) (121az++ (ecr))(" 1), and Tanh(z) (e" ") " + =-")-
29

---

Published as a conference paper at ICLR 2024
Therefore, have
/ (
(I( -
(
I Bl12dt
(Amin (tA + (1 (1-))))) - - Bll2dt.
Using Weyl's inequality on the eigenvalues of the sum of two matrices, we have the following
Amin (tA + (1 B) 2 Amin (t.A) + Amin(( - B)
iAmin (A) (1 1))min (B)
2 min Amin ( (A), Amin (B)).
Combining the two inequalities concludes the proof.
We are now ready to prove Proposition 6.1 as follows.
Proposition 6.1 Let D and D' DUS be an arbitrary pair of leave-one-ous datasets, where D con- tains n records and D' contains + records. Define the function h(a) = supz,=' (z, a/)-
Let Ormin = argmin h((a), and thus h(amin) quantifies how close the kernel matrix is to a low rank all-constant matrix. Assume that Ominn > 0. Then, it holds that
max M(Q) S Cn.amann 22..0mm.c2772/ 125/2/(amm + Ca.n Q
where
(Ormin + [u(armin))
(1+ (1+n -1)5//2 (Amin + h((amin))
Cn,aminn = max
min ( |amin nh(amin))) miin(amiatt |amin 0 - (n + 1)m((mmin))
and Caminil ((1 mmmn).
As a result, if 1/5/2 maximum influence) satisfies maxQ (Q) < 1.
and
then the maximum mean distance LOOD (i.e..
Similarly, there exist constants A,, B > 0 such that
max (On
S Anm((amin) + B n
Proof. Let Q be a query point and a E R. We have the following
M(Q) } KonM yD KQD Mp
where MD (KDD + d²1) and the same holds for MD Let U aUn, + o21 and U" aUn+1 + =02 where U, is the n x n matrix having ones everywhere. With this notation, we have the following bound
2 M(Q) < ( -
-U-------pa U->)ynll + | |oo (M
+ K (U") " yD' KQDU ypl..
Let us deal with first term. We have that
[[Qo U-l)ool < [Koo|lly
M|| U--1 - n 1/2 (oootM))llllllMMM--00000 -U-1I.
30

---

Published as a conference paper at ICLR 2024
Using Lemma G.2
we have the following
|M - U- S
Amin (B - Ull
which implies that
n (a - h(a))llD x h((x),
[M -U wDDl=
min(ao?, 10002 -nh(a)])
where we have used the inequality (A) Amin (B)l - 1|A4 B for any two symmetric positive matrices A,
Similarly, we have the following
() h(a)) x h(a).s
|KQD (M - (0'))'))0) "Dol v
min(ao2, |ao2 - (n 1)h(a)))
For the last term, we have
[|KgD= (U") yD KQDU vDl s |[K@D ac, "0"))
+ [(KoD - ae, ybll
+ (1') yD ae, U ypl-
Moreover, we have
[I(Kqn -u- ydl So So-llullllihho
and,
[KKoD ae, (+)))) yo S B|yd Vn 1h(a).
For the last term, we have
a ||ae u- S aan ozIla(n + 1) 02
Σ YD.
a
+
Us
Ja(n 1) + -²
< (1 + a an
Combining all these inequalities yield the desired bound.
For the second upperbound, observe that (n(Q)=((Q))² 1 by the definition of KL LOOD in Definition
=log x' MQQ)
T(Q))
+
22"
C.3 Meanwhile, for sufficiently
TQQ)
small h((mmin) we have that
Σ(Q) = Qmin 2 12 + m0 + ((((amin)) = Omain 0² 222 -nami)) + ((h(amin))
and a similar formula holds for (Q) with (n + 1) instead of n. Therefore, we obtain
E(Q) +
Omin
(Q)
o2 + Ominn
Combining this result with the upperbound on the mean distance, the existence of A,, and B is straightforward.
31

---

Published as a conference paper at ICLR 2024
J 1rl
JS0:20h
107221
.aian
$222111
Figure 14: Visualization of 500 randomly chosen differing data and query optimized by LOOD (Part 1). We show differing data above the optimized query, and use their LOOD gap to sort the images.
32

---

Published as a conference paper at ICLR 2024
J til
LEPS
J 5%%
10/105
5022%
107215
.00.2%
.2:08
Figure 15: Visualization of 500 randomly chosen differing data and query optimized by LOOD (Part 2). We show differing data above the optimized query, and use their LOOD gap to sort the images.
33

---

Published as a conference paper at ICLR 2024
201
J 1r0
Jt M0
.52/115
JS0:00
107285
JPian
Figure 16: Visualization of 500 randomly chosen differing data and query optimized by LOOD (Part 3). We show differing data above the optimized query, and use their LOOD gap to sort the images.
34

---

Published as a conference paper at ICLR 2024
J tit
LEPS
2000
J NO.
107221
Figure 17: Visualization of 500 randomly chosen differing data and query optimized by LOOD (Part 4). We show differing data above the optimized query, and use their LOOD gap to sort the images.
35

---

Published as a conference paper at ICLR 2024
Pian
Figure 18: Visualization of 500 randomly chosen differing data and query optimized by LOOD (Part 5). We show differing data above the optimized query, and use their LOOD gap to sort the images.
36

---

Published as a conference paper at ICLR 2024
Figure 19; Visualization of 500 randomly chosen differing data and query optimized by LOOD
(Part 6). We show differing data above the optimized query, and use their LOOD gap to sort the images.
37


Reviews:
[{"review_id": "o97XD9hAwX", "reviewer": "Reviewer_mmKM", "summary": "The paper proposes a framework called LOOD that measures how a model's output on some samples Q changes when a set of samples S are added to the training set. It uses GP to model the model randomness and looks at the KL divergence or mean predictions between the distribution of the output on Q given datasets D and D' that differ in S. The Q that is affected by S the most is mostly S itself. \nThe paper then shows that LOOD can be applied to measure information leakage, data reconstruction, and influence. It can also be used to explain the effect of different activation functions.", "questions": "Maybe I'm missing some important point but why is some experiment done on training data that consists of two classes only (car & airplane)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed framework is good at capturing many different aspects of model memorization and is computationally efficient. \nThe empirical evaluations are pretty throughout and interesting.", "weaknesses": "The presentation might be improved to make the contributions easier to see and the analyses easier to follow. For example, I was a bit confused about the purpose when I first saw Section 3 on \"OPTIMIZING LOOD TO IDENTIFY THE MOST INFLUENCED POINT\". Maybe a more detailed explanation on why we want to do so would help (e.g. how would knowing the most influenced point benefit us in measuring memorization / leakage etc).\nIt is also a bit unclear what LOOD would enable us to do that cannot be done with previous method. It seems to me that one big advantage of LOOD is the computation efficiency. If so, I think the authors can consider adding more detail comparison to highlight that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aKR6cSu1Yj", "reviewer": "Reviewer_rJTi", "summary": "This paper introduces a new way to measure or quantify the output distributions of a machine learning model on changing a few points in the input dataset, which they call, \"leave-one-out distinguishability\" or \"LOOD\". This is defined as the statistical distance (in this case, the KL-divergence) between the output distributions of the model on changing a few data points in the dataset. The main applications or advantages of introducing this notion of LOOD is that LOOD could be used to quantify (1) the memorization of data, (2) the leakage of information (via membership inference attacks), and (3) the influence of certain training data points on the model predictions. In this work, the applications of LOOD are illustrated via Gaussian processes, which they use to model the randomness of the machine learning models. They also show the effect of activation functions on LOOD. From their empirical results, they show LOOD as a good measure for all the above phenomena.", "questions": "1. Have you thought about the connections between LOOD and differential privacy (DP)? As in, DP algorithms guarantee that a few data points cannot influence the output of the algorithm a lot, so will LOOD give any useful information about DP ML algorithms?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Their definition of LOOD captures the influence of data points in the trained machine learning models quite well, which they show for different phenomena, such as information leakage via membership inference attacks, and memorization.\n2. Their experiments cover enough breadth, for example, by considering different kernels (like RBF and NNGP). So, the set of results seems comprehensive enough.", "weaknesses": "From my limited understanding of the subject, I can't find any significant weaknesses in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4qFlhKE020", "reviewer": "Reviewer_Azxn", "summary": "This paper introduces a notion of algorithmic stability called *leave-one-out distinguishability*. For a learning algorithm that outputs a predictor, a pair of data sets $D$ and $D'$ that differ in a single record, and a specific query point $Q$, LOOD captures: how much does the algorithm's prediction on $Q$ change between $D$ and $D'$? This is a notion of local stability. The paper formalizes \"change\" through either KL divergence or average prediction, depending on the application. We also sometimes allow the data sets to differ in multiple points, or allow $Q$ to be a set of queries.\n\nThe paper aims to both unify existing work on memorization and influence and to answer questions about specific learning procedures. It contains a large number of both theoretical and experimental results. The theory results focus on Gaussian processes, while the experiments supplement these theorems and also show how the results can extend to neural networks.\n\nAfter introducing the definition, we move to Section 3, which addresses the question of what query point maximizes LOOD. We see strong theoretical and experimental evidence pointing to the conclusion that, for GPs, the optimal query is the point that differs between $D$ and $D'$. (Nonconvexity makes establishing global optimality difficult.) We see an experiment with some evidence that the story is similar for neural networks.\n\nSection 4 relates LOOD to existing notions of memorization and privacy. Most interesting to me is Figure 3a, which shows that data points tend to have similar susceptibility to membership inference attacks under NNGPs and NNs. We also see that GPs are vulnerable to data reconstruction attacks.\n\nFinally, Section 5 explores how the activation function affects NNGPs and NNs. We get a theorem about the rank of NNGPs and complementary experiments on both classes of models.", "questions": "When having coffee with other researchers, which results from this paper are you most excited to discuss?\n\nWhich results in this paper will have the biggest impact on future research?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This submission is full of ideas and deserves a treatment longer than nine pages. Despite that, the paper is relatively clear. The definition of LOOD closely builds on existing ideas in the literature, but uses them in new ways. The fact that we get clear results on GPs is nice, and I appreciate how we saw the results extended to NNs.\n\nWith so much content, I think it's a paper that will attract a large audience. I hope it gives them food for thought. I vote for acceptance.", "weaknesses": "The paper's density caused rushed discussions. In particular, I would have appreciated more \"hand-holding\" alongside the theorems and proofs. Similarly, I might prefer fewer experimental results with clearer explanations.\n\nMy largest critique of LOOD is that I feel it lacks a \"killer application\" which cleanly shows off its value. To elevate the paper, I would hope for a stronger answer to \"what can we do now that we could not do before?\" The paper gives answers about GPs, but I find these a bit underwhelming: I do not expect kernel methods to protect privacy. (Of course, people who do not work on privacy may have different expectations.) The results on NNs are interesting, but none of them are explored in enough detail to serve as a headline result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "py8I08xBTo", "reviewer": "Reviewer_s1Kq", "summary": "This paper introduces a comprehensive framework using Gaussian processes to analyze the influence of individual and groups of training data points on machine learning model predictions, with a focus on privacy implications. The authors propose Leave-One-Out Distinguishability (LOOD) as a metric to quantify information leakage and demonstrate its applicability in various settings, including Gaussian processes and deep neural networks (NNGPs). They also provide theoretical insights, especially on the impact of activation functions on information leakage, and validate their findings with extensive experiments.", "questions": "1.Does the approach of modeling the randomness of models using Gaussian processes have general applicability beyond neural networks?\n\n2.Given the definition of LOOD, using it to analyze and measure MIAs seems like a natural fit. However, for other potential attack methods, is there a way for LOOD to measure their privacy capabilities?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel theoretical framework using Gaussian processes to analyze the influence of individual and groups of data points on machine learning model predictions. The authors creatively combine ideas from influence analysis, memorization, and information leakage, providing a comprehensive perspective on how training data affects model predictions. The application of LOOD (Leave-One-Out Distinguishability) as a measure of information leakage, and the exploration of its relationship with activation functions, add a unique angle to the existing body of work.", "weaknesses": "1.The authors have considered the fact that LOOD is a non-convex objective when analyzing global optimality using first-order information, acknowledging that analyzing global optima is quite challenging. However, if one solely utilizes the RBF kernel, as a nonlinear equation, many techniques from geometric analysis could be applied to examine the connections between global and local optima (such as considering the local properties of solutions to nonlinear equations using Sard's Lemma, etc.). Relying on experiments to complement the theoretical proof significantly undermines the credibility of the theory.\n\nSimilar issues are present in other sections of the paper as well. As an article that introduces a theoretical framework, the feasibility of this framework is supported by experimental evidence in many places. This approach raises doubts about the theoretical correctness of the framework, as it heavily relies on empirical validation rather than providing rigorous theoretical proofs throughout the paper.\n\n2.The paper extensively analyzes the LOOD under RBF and NNGP kernels. While these are commonly used kernels, the generalizability of the results to other types of kernels or models is not clear. The paper could be strengthened by either extending the analysis to other kernels or by providing a clear justification for focusing on these specific kernels.\n\n3.Given the definition of LOOD, using it to analyze and measure MIAs seems like a natural fit. However, the paper does not discuss whether LOOD can be used to measure the privacy capabilities against other potential types of attacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)