Parameters or Privacy: A Provable Tradeoff Between Overparameterization and Membership Inference
Authors: Jasper Tan
Blake Mason
Hamid Javadi
Richard Baraniuk
Venue: NeurIPS 2022 Accept (2022)
Keywords: Membership inference
Privacy
Linear Regression
Overparameterization

Abstract: A surprising phenomenon in modern machine learning is the ability of a highly overparameterized model to generalize well (small error on the test data) even when it is trained to memorize the training data (zero error on the training data). This has led to an arms race towards increasingly overparameterized models (c.f., deep learning). In this paper, we study an underexplored hidden cost of overparameterization: the fact that overparameterized models may be more vulnerable to privacy attacks, in particular the membership inference attack that predicts the (potentially sensitive) examples used to train a model. We significantly extend the relatively few empirical results on this problem by theoretically proving for an overparameterized linear regression model in the Gaussian data setting that membership inference vulnerability increases with the number of parameters. Moreover, a range of empirical studies indicates that more complex, nonlinear models exhibit the same behavior. Finally, we extend our analysis towards ridge-regularized linear regression and show in the Gaussian data setting that increased regularization also increases membership inference vulnerability in the overparameterized regime.

Parameters or Privacy: A Provable Tradeoff Between Overparameterization and Membership Inference
Jasper Tan Rice University
Blake Mason Rice University
Hamid Javadi Rice University
Richard G. Baraniuk Rice University
Abstract
A surprising phenomenon modern machine learning is the ability of a highly overparameterized model to generalize well (small error on the test data) even when it trained to memorize the training data (zero error on the training data). This has led to an arms race towards increasingly overparameterized models (c.f., deep learning). this paper, we study an underexplored hidden cost of overpa- rameterization: the fact that overparameterized models may be more vulnerable to privacy attacks, in particular the membership inference attack that predicts the (potentially sensitive) examples used to train model. We significantly extend the relatively few empirical results on this problem by theoretically proving for an overparameterized linear regression model in the Gaussian data setting that mem- bership inference vulnerability increases with the number parameters, Moreover, range of empirical studies indicates that more complex, nonlinear models exhibit the same behavior. Finally, we extend our analysis towards udge-regularized linear regression and show in the Gaussian data setting that increased regularization also increases membership inference vulnerability in the overparameterized regime.
Introduction
As more machine learning models are being trained on sensitive user data (e.g., customer behavior, medical records, personal media), there is growing concern that these models may gateway for malicious adversaries to access the models' private training data [1 For example, the possibility of performing membership inference (MI), the task of identifying whether a specific data point was included a model's training set or not, can be greatly detrimental to user privacy. In settings like healthcare, knowledge of mere inclusion in a training dataset (e.g.. hospital visitation records) already reveals sensitive information. Moreover, MI can enable the extraction of verbatim training data from released model II]
The general motivation for this paper is the recent realization that privacy issues around machine learning might be exacerbated by today's trend towards increasingly overparameterized models that have more parameters than training data points and so can be trained to memorize (attain zero eгтоr on) the training data. Surprisingly, some overparameterized models (e.g., large regression models (3). massive deep networks 4 (5)) generalize extremely well 7. Limited empirical evidence suggests that overparameterization may lead greater privacy vulnerabilities been little to no analytical work on this important problem.
However, there has
In this paper, we take the first steps towards analytical understanding of how the number of parameters in a machine learning model relates to MI vulnerability. In theoretical direction, we prove for linear regression on Gaussian data in the overparameterized regime that increasing the number parameters of the model increases its vulnerability to MI (Theorem 3.2). In a supporting empirical direction, demonstrate that the same behavior holds for range of more complex models: a latent space model, a time-series model, and a nonlinear random ReLU features model (Section 5)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We also extend our analysis towards the techniques of ridge regularization and noise addition in Section We first prove that MI vulnerability similarly increases with the number of parameters for ridge-regularized models. Surprisingly, we also demonstrate analytically that additional regularization increases this vulnerability in overparameterized linear models. Hence, ridge regularization is not always an effective defense against MI and can even be harmful. Afterwards, we show that the privacy-utility trade-off induced by reducing the number of parameters of linear regression model is equivalent to that obtained by adding independent Gaussian noise to the output of the model when using all available parameters.
Overall, our analyses show that there are settings where reducing the number of parameters of a highly overparameterized model is a simple strategy to protect a model against MI. As the trend towards increasingly overparameterized machine learning models accelerates, our results make the case for less overparameterization when privacy is concern.
Related work. The problem of membership inference has received considerable attention, and we refer the reader to [I2] for survey of the prior art. Many works have demonstrated how deep neural networks can be highly vulnerable to MI attacks (cf., 13 for example). Various types of attacks, such as shadow models [ confidence-based attacks
and label-only attacks [19] have risen to further expose privacy vulnerability of machine learning models. Our analysis in this work is based on optimal MI attacks, a framework also employed by other studies of MI M [CC
While we study the effect parameterization has on MI vulnerability, several studies examine how other aspects of the machine learning pipeline, such as data augmentation (20), dropout [15). sparsity (21). and ensembles (22) affect MI vulnerability. [16] show that pruning a neural network can defend against MI, lending further experimental support to the link between overparameterization
and membership inference. g analyzes the relationship between overfitting (as measured by generalization error) and membership inference but does not connect this to the number of parameters. Much recent work has shown though that overparameterization does not always lead to increased generalization error and indeed sometimes can even decrease it [X] 3 K 77 suggesting that more work is needed linking overparameterizatioo and membership inference beyond mere generalization error. Despite some empirical evidence that overparameterized models are vulnerable to MI attacks [ЛЛ. to the best of our knowledge, there has been no theoretical study of the effect of this connection.
Differential privacy (DP) [44) is another popular framework frequently used to study the privacy properties of machine learning algorithms, and models that have DP also enjoy MI guarantees [14). 25 bounds the minimum dataset size function of dimension to ensure privacy, however its results do not extend to data of continuous values (e.g., regression data) as is done in this work and does not account for the randomness in the data generating process, which as argued in [14. is necessary condition to bound the optimal MI risk.
While our theoretical analysis focuses linear regression, many works have studied overparame- terized linear models as an interpretable and tractable test case of more complex overparameterized suggesting links of our results to more complex
models such as deep networks [26 27 nonlinear models.
Contributions. In this paper, we establish the first theoretical connection between overparameteriza- tion and vulnerability to membership inference attacks. Our contributions are as follows:
1. We derive the optimal MI accuracy for linear regression on Gaussian data and show that increasing the number of parameters increases the vulnerability to MI.
2. We empirically show for more complex models that increasing the number of parameters also increases MI vulnerability.
3. We theoretically show for overparameterized linear models that increased ridge regulariza- tion increases MI vulnerability in the Gaussian data setting.
4. We use our analysis to show that reducing the number of parameters of linear regression model yields an equivalent privacy-utility trade-off as adding independent noise to the output of model using all available parameters.
Our code is in httppp//ghhhho.ooommmaassppppppp
2

---

2 The Membership Inference Problem
We now introduce our notation and formally state the membership inference (Ml) problem. Let D E Z+ denote the data dimension. Let 2 = (æ, y) E RDD x IR denote a data point, and let 'D be distribution over the data points. Consider set 8 sampled S D of size E Zt data points and denote it by S = {a1, 22, ..., Zn )= { (( y1), (In. Jn . Let fs represent a machine learning model obtained deterministic function in RDD + IR. Typically, it is a function that minimizes _ ((f())) y1) over family of functions f for some loss function f R R R20. We denote yi [s(x;)
applying a training algorithm T on the dataset S. In particular, is is
Given a data point 2g and a trained model fs. membership inference is the task identifying if 20 was included in the set S used to train fs. Different MI attacks are characterized by the types of information accorded to the adversary. In this work, we focus on blind black-box setting where the adversary has access to 2p. fs(aa), n, the training algorithm T, and the data distribution D, but not to the ground truth yo. the loss value {(fs( yo). or any learned parameters of fs. Thus, adversary is a function whether Do is member of S based on fz())))
Rd x + {0, }} such that given (æo, fs()))) outputs 0 or 1,
representing its prediction
There are two main interests in studying blind MI wherein the adversary does have access to the loss value. First, there are many realistic scenarios where the ground truth variable of interest unknown for the general population, and hence the need for model fs to predict it. Second, since we study the interpolation regime where the training loss can be driven down to exactly O, MI becomes trivial given the loss value: an adversary that predicts any data point that achieves loss member will have perfect accuracy. Multiple works in the literature also study this blind setting [8 28) We emphasize that our results on the blind adversary lower bound the performance of adversaries that additionally have access to more information.
MI often defined as an experiment to facilitate precise analysis, and we follow the setup of summarized below.
Experiment 1. Membership inference experiment. Given a data distribution D, an integer machine learning training algorithm T' and adversary A, the MI experiment is as follows:
1. Sample S D
2. Apply the training algorithm T' S to obtain fs.
3. Sample m {0,1} uniformly at random.
4. If m 0, sample data point (æo, D. Else, sample data point (æo, yo) S
uniformly at random.
5. Observe the adversary's prediction A(zo, fs (xo)) E {0, 1}.
We wish to quantify the optimal performance of the adversary A and turn the popular membership advantage metric defined in 9 'he membership advantage of A is the difference between the true positive rate and false positive rate of its predictions.
(1) The membership advantage of an adversary A is: Adv(A) == ((A(zo, fs(())) Definition 2.1
m = 1) ([
m = 0),
where IP()) is taken jointly over all randomness in Exp.
Note that membership advantage is an average-case metric, opposed to worst-case metrics con- sidered by other privacy frameworks such as differential privacy. Our analysis is thus focused on average-case privacy leakage, and we do not provide worst-case guarantees.
3 Theoretical Results
3.1 Optimal Membership Inference Via Hypothesis Testing
In this paper, rather than studying current MI attacks, we study the optimal MI adversary: the adversary that maximizes membership advantage. such, our analysis and results are not restricted to the current known attack strategies, which are constantly evolving, and instead serve upper bounds for the performance of any MI attack, now the future. The following proposition supplies the theoretically optimal MI adversary.
3

---

Proposition 3.1. The adversary that maximizes membership advantage is:
{
if P(ño I m = 1, > P(io m = 0, mo),
A" A (wo, fss())))
otherwise,
where 90- = fs and P denotes the distribution function for To over the randomness in the membership inference experiment conditioned on Zo-
As observed by 129 TOl, the optimal adversary performs a hypothesis test with respect to the posterior probabilities, with the two hypotheses being:
Ho S ~ D", (2o. 3/0 ~ DD and H1 : ~ D", (2o, yo) ~ S,
so that maximizing membership advantage is achieved by performing likelihood ratio test.
3.2 Linear Regression with Gaussian Data
We begin our study of overparameteriiation''s effect on MI with linear regression with Gaussian data. We find that in the sufficiently overparameterizee regime, increasing the number of parameters increases vulnerability to MI. We denote by n, the number of data points, number of model parameters, and data dimensionality, respectively. Let n. p. D E Z be given such that y S D, and let o > 0 be given. Consider a D-dimensional random vector
N (0, } [D) representing the true coefficients. We consider data points (æ;, 3:) where 2; ~ (O, La) and yi x 3 + es, where & ~ (,,N(0,,)) Sampling n data points, we denote by X the n x D matrix whose 7t row is x and by y the ne-dimensional vector of elements ye- Further, let X, p be the p matrix containing the first p columns of X. Least squares linear regression finds the minimum-norm p-dimensional vector that minimizes |yy X 1. which is 3 = X|y. where X denotes the Moore-Penrose inverse of Xp. Then, for a given feature vector Fg, the model predicts yo TT where To.p is the vector containing the first p elements of Zn- In this setting, we can derive the membership advantage of the optimal adversary as a function of the number of parameters p used by the model.
D. Let xo be given D- dimensional vector. Let 3 ~ NO, p)) xi ~ NO, ID), G ~ NN0,00), and y; = x. 3 + G, for i {1, ...,, (1.,...., n. Let be variable whose value is either 0 or such that, if 2o for k chosen uniformly at random from 1, 1,2,.., n. Let X be the n x D matrix whose rows are a, 2 ...., xT Finally, let yo 20.p" X1y where y is the n-dimensional vector with elements Yi-
Theorem 3.2. Let n.p, D Zt be given such that + < p
x& is set to
Then, as n. D + 0o such that P. - y E (1,00), we have:
(Jo m = 2o) ~ NN .b) and (Jo m = 100 ~ (( m11100))
(1)
where
1+ 1+ - p n
|
of
| and of 0²
(2)
D
D
Consider the case when 01 > ag- The resulting optimal membership inference algorithm A is
ob log
A' (2o, 06) =- 50 a²]
where a a== \
(3)
o
with membership inference advantage Adv:
Adv(A*) = Euro
+ a
(4)
where (b) denotes the CDF of the standard normal, and a, To. d1 are conditioned Zn- Remark 3.3. The case where < To occurs when is small (p Re n)). The same membership advantage result holds in this, reversing the roles of on and o in Adv(A") in Eqs. and and reversing the inequality in Eq. S
Remark 3.4. The above result holds using the asymptotic distributions nt, D - c0. In Lemma in the Appendix, we derive the non-asymptotic distributions for the predictions of the minimum-norm least squares interpolator, though they cannot be written in closed form.

---

Standard Deviation
8
0.30
m=0
0.30
m=0
0.25
m-1
m-1
0.15
0 15
0.10
0.10
0.05
0.00
0.00
-15
5
-15
-10
-5
10
15
y= pin
96
(a) Standard Deviations
(b) 1.1
(c) 7 = 1.2
0.30
2.00
0.25
m-1
m=0
0.6
0.7
m=0
1.50-
175
m=0
1 25
0.15
0.4
0.10
0.3
0.75
0.50-
0.05
0.1
0.25
0.00
0.0
0.00
15
-10
55
0
5
15
15
-0
-5
0
10
15
15
-10
5
10
15
yo
96
96
(d)
Y
(c) 10
50
Figure I: In the highly overparameterized regime, increasing the number of parameters p yields more distinguishable posterior distributions. (a) Standard deviations of model outputs yo for minimum- norm least squares as function of parameterization for y > 1. (b-f) The Gaussian distributions in Eq. ① (broken lines), as well as empirical histograms of 20,000 yo samples for different y values with 400, D = 20, 000, a = 1. The prediction variance for m = 1 stays constant, while the variance for m 0 decreases with increased parameterization, making the distributions easier distinguish.
In Theorem Eq. shows that the posterior distributions of the outputs for test points (when m 0) and training points (when m = 1) are both O-mean Gaussians but with variances of and of respectively given in Eq. 2 Recall from Prop. 3.1 that the optimal MI adversary is likelihood ratio test (LRT) between the distribution of the model's output for new test points (when m = 0) and that for reused training points (when standard sufficient statistic for an LRT between two O-mean Gaussian distributions with deviations ao and 01 Finally, compute the membership advantage of this adversary in Eq. 4 by taking an expectation over the random draw æg. as defined Experiment| ! noting that do. 01 and a are all functions of æg. Membership advantage is defined to be the difference between the true and false positive probabilities in Defn. 2.1 This difference given by ((%/oo) (( for each side the Gaussian distributions, leading to the expression in Eq.
1). This is reflected in Eq. 4 where we note that is the
To understand the implication of the result for MI, first observe that when m 1 and Zg a training point that is memorized by the model, the variance of the model's output is equal to the variance of measurement yo itself independent of p. On the contrary, when ra and 3g is test point, the variance of the model's output is decreasing with p (Figure Ta Hence, though the output distribution means stay the same, as the variance for the m case decreases far below that of the m = case, an LRT can easily distinguish these two distributions. In the extreme case, suppose D > then of + 0, while of remains a nonzero value. Hence Adv(A") +
We confirm our derivations of the output distributions numerically in Figure where we plot empirical histograms of the predicted distributions from Eq. | by repeatedly computing the minimum norm least squares solution over 20,000 independent trials] We also plot the standard deviations from Eq. 2 in Fig. Visually, from the distributions in Figure given f(z)), MI reduces identifying whether it is more likely that sample came from the blue distribution or from the orange with the adversary simply predicting the more likely outcome. These distributions intersect Laa, so sufficient to compare Jo to a² perform the LRT.
In Figure 2a we plot the membership advantage for > using Eq. Since the differennc between variances of the model's output when m 0 and increases with 7 for 7
We detail additional experimental details and the computational hardware in Appendix E
5

---

Membership Advantage
1.0
Wembership Advantage
0.7
0.6
0.8
0.5
0.6
0.4
0.3
0.4
0.2
0.2
Experimental
0.1
Asymptotic
0.900
0.0
10a
102
10a
10
10¹
y=
y= p/n
(a) Asymptotic membership advantage
(b) Empirical verification for non-asymptotic
Figure 2: Increasing the number of parameters increases membership advantage. (a) Theoretical membership advantage for linear regression with Gaussian data (Eq, + as function of the number of parameters for 100 sampled Zo 8, membership advantage averaged over 20 sampled zu for n 100, D 000, and estimating the two posterior distributions for each using empirical histograms with 100,000 samples. We plot it alongside theoretical asymptotic membership advantage, showing a close agreement between the two.
10 D 10 (b) We empirically approximate the
by
(cf., Fig. la it becomes easier to distinguish the two distributions. This results in an increase the membership advantage. The initial decrease in membership advantage when is consequence of 0 S in this regime as shown in Fig. la Since a is decreasing in initially, this decrease makes the train and test output distributions harder distinguish, leading lower membership advantage and 0 advantage for 7 2. However, for 7 > 2. Tp decreases past σ and membership advantage approaches as - 30. practice, the 7 regime (only slightly overparameterizzed is less interesting since models suffer larger generalization error in this setting (cf.. Theorem I of (3) and are rarely (if ever) used in practice. A key takeaway is that for linear regression in the Gaussian data setting, extreme overparameterization increases the vulnerability of a machine learning model MI.
While Theorem 5.2 operates the asymptotic regime, we empirically approximate membership advantage for 100 and D 3, 000 by approximating the posterior distributions with histograms of samples. As we see in Figure 2b the asymptotic formula agrees very closely with the non- asymptotic experiment. Experimental details are provided in Appendix E
Mitigating Membership Inference Attacks
Next, extend our analysis from the previous section towards two methods commonly used for preserving privacy: regularization and noise addition. We present two key results. First, we show that for the overparameterized Gaussian data setting, ridge regularization actually increases membership advantage and thus detrimental to privacy. Second, we show that the privacy-utiliis trade-off induced by reducing the number of parameters of a linear regression model with Gaussian data is equivalent to that of adding independent Gaussian noise to the output of a model that uses all available features.
4.1 Kidge-Regularizer Linear Regression
We analyze membership inference in ridge-regularized linear regression in the same setting as y. where is regularization parameter. Larger values of yield greater regularization, and X 0 reduces to the case of Section
Section 3.2 except that, now, the estimate is 3^ = xx
3.2 Regularization a common method to reduce overfitting and has thus been proposed previous works defense mechanism to protect models from MI attacks [K] 30)
surprising observation resulting from our analysis that, in the highly overparameterized regime (y > 1), ridge regularization actually increases the model's vulnerability to MI attacks for linear

---

Wembership advantage ge
0.9
Membership advantage
Membershit advantage
0.7
0.8
0.6
Feature reduction
0.7-
0.8
Noise addition
0.5
0.6
0.5
0.4
0.4
MM 10000
0.4
0.3
0.3
DA 1000
0.2
0.2
0.1
nA 100
0.1
nA
0.00
10p
10a
10a
2000
4000
6000
0.0
= p/n
Regularization amount nd
Generalization Error
(a) Ridge (vs. parameters)
(b) Ridge (vs. regularization)
(c) Noise Addition
Figure 3: We analyze ridge regression and noise addition, two different methods that aim reduce membership advantage (MA). (a) MA vs. overparamterizztionn 7 for few ridge regularization strengths Stronger ridge regularization harms privacy (increased MA). (b) MA vs. for a few y's. explicitly showing how increasing À increases membership advantage. (c) We plot the privacy- utility trade-off obtained when tuning the number parameters (feature reduction) and when adding independent noise to the output of a model that uses all available parameters (noise addition) and demonstrate that the two trade-offs are essentially equivalent. For (a) and (b), n 10a, 107,
and 1. For (c), 100, D 3000, and a 1.
regression with Gaussian data. In Theorem D. presented in the appendix, we derive an analogous result for the distributions of the ridge-regularizer predictions as Theorem 3.2 demonstrate for the unregularized case. In particular, if A is the adversary for the X-regularized problem, its membership advantage Adv(A)) is an increasing function in À when y > 1. To visualize this, we plot the theoretical membership advantages for 100 sampled æt 's (each with iid standard normal elements) for differing regularization strengths Figure with the same setting as in Figure 2a Figure Sa we plot membership advantage as a function of the overparameterization ratio for few n. values. Conversely, Figure 3b we plot membership advantage versus regularization nλ for few different values of y. In particular, note how in both subplots, increasing never decreases the membership advantage. This observation has been made empirically (but not analytically) for techniques that have similar regularizing effects, such dropout [15], ensembling [22) and weight decay [30)
Intuitively, the reason ridge regularization increases MI vulnerability is because while it decreases the variance of the model's 's output on training points, it also significantly decreases the variance for test points such that the two output distributions become more distinguishable. Figure the appendix, we plot the gap between the variances for the m 0 and m = cases for different regularization amounts to visualize this effect.
4.2 Noise Addition Feature Reduction
consequence of Thm. 3.2 is that, in the Gaussian data setting, one can reduce vulnerability to MI attacks by simply decreasing the number of parameters/feature.. However, this comes at the cost of decreased generalization performance (utility) due to the "double descent" effect K 23 Z]], wherein generalization
decreases with increased overparameterization.. An alternative and popular method to increase a model's privacy adding independent noise to the model output [24] 31, but this also decreases generalization performance. Interestingly, show in this subsection that for Gaussian data, the privacy-utility trade-off's induced by both feature reduction and noise addition are actually equivalent.
Feature reduction: To characterize the privacy-utility trade-off feature reduction, we first need the generalization eтror for given number of parameters, provided in Corollary 2.2 of 23
Proposition 4.1. (Adapted from Corollary 2.2 of (231). the same setting as Theorem for Zp ~ (O, ID), the generalization error is given by
1 11 1 p p- n-
E 3 xT ) = 0²
+
Alternatively, this expression be written 1 + 2² + Ex, j - with of in Theorem 3.2
7

---

Noise addition: Next, we consider noise addition performed by perturbing the m = 0 model output
with independent noise before releasing it: 50 T + where ~ (O, a²). Recall that MI is possible for overparameterized linear regression models because the variance of the output of test points (m 0) is lower than the variance of the output for training points (m = 1) (Fig. D Thus, adding random noise to the model output when test point 2o is queried can make the output distributions harder to distinguish. In the following lemma, compute the membership advantage and generalization error of noise addition using the full set of features p= DD
Lemma 4.2. Consider the setup of Theorem 3.2 restricted p = D, σ1 dv. and 7 1, and suppose independent O-mean, variance ō² Gaussian noise is added to the outputs when m = 0 (Io is not a member of the training set). Then, (Yo m 0, zo) ~ NO, + σ²). The optimal adversary
is
(oī + 0 oo log
A (oo (o) == 11 1000 > obr
where as ==:=
of (o + x2)
Its membership inference advantage is
Adv(A))= - Exs 2 T
0a
aa +
(5)
Vog
Ø2
σ1
Additionally, for xo ~ (O, p)), the generalization error incurred is
o²
-
11 D
[ - 3T xx))) x0 = + +n
+02.
(6)
D D-n-t
The terms in Lemma 4.2 similar to those in Theorem 3.2 except for the added dependence on the
added noise's variance 22 the case that m = O (æg is not a member). Increasing noise variance
_ x2 decreases membership advantage (Eq. 5 the cost of increased generalization error (Eq.
Indeed, it is possible to add sufficient noise such that of = og + σ², rendering the membership advantage O, though possibly at the cost of impermissible generalization performance.
we plot the membership advantage vs. generalization error trade-off' for both feature reduction (blue) and noise addition (orange). The plots follow the setting of Fig. 2b For the blue curve, we employ the expressions in Thm 3.2 and Prop 4.] while varying the overparamettrizztioo ratio y. For the orange curve, the noise variance o2. Fig. 3e shows that the trade-offs induced by both feature reduction and noise addition are essentially equivalent.
In Figure 3C
Lemma 4.2 using all available features (p D) while varying
5 More Complex Models
In this section, we present three more complex data models wherein we empirically observe increased overparameteriization leading to increased MI vulnerability.
For each data model, we perform the following experiment. We first sample a 2o vector, which is the data point we wish to perform MI on. Then we sample training dataset X, measurements y. as well as other random elements according to the data model. To obtain an m prediction, we learn model on X that we then apply Eg. To obtain m prediction, with æy and the corresponding element of with yo before learning the model and applying it on Tn. Keeping 2n fixed throughout the experiment and resampling all other random data (such as X) many times, we collect a large set of m = and m = prediction samples. We build a histogram of these samples by assigning them into fine discrete bins obtain approximations of the conditional densities P( m 1,, 20) and ( m 0, w0) needed for the optimal adversary (cf., Prop 3.1 To approximate membership advantage, we sum up the differences P(jo m1; mo) - P( mp, #n) over all the histogram bins where P( mm, 2n) > P( my, zu). We repeat this experiment 20 times, each with newly sampled Zu, and plot the means (as points) and the estimated standard errors (as shaded error regions) of the membership advantage values across the 20 experiments in Figure 4 We next discuss the data models in detail.
first replace
of X
Linear Regression on Latent Space Model: We first consider the latent space model from 3
where the output variable ys is noisy linear function of a data point's d latent features z;, but one
8

---

Membership Advantage
Membership Advantage
1.0
Membership Advantage
0.6
0.18
0.8
0.5
0.16
0.14
0.4
0.12
0.3
0.4
0.10
0.2
0.08
0.2
0.1
0.06
0.0
0.0
10a
10
10"
10a
10P
10'
y= p/n
y= pin
V = p/n
(a) Latent space model
(b) Time-series data
(c) Random ReLU features
Figure 4: Empirically measured membership advantage vs. parameterization for various data models
The dashed line at divides the underparameterized and overparameterized regions. For all settings, when sufficiently overparameterized, increasing the number of parameters increases vulnerability to membership inference attacks. (a) Latent space model with
detailed in Section 5
200 where latent features are observed. (b) Regression over n 128 time samples
p covariates of d
of linear combination of D 1024 Fourier features. (c) Nonlinear random ReLU features with n 100, = 5000, and σ = 1.
only observes vector æ; containing p 2 d covariates rather than the direct features 2μ. Let Z be an x d matrix where each row is the vector of latent features for an observation. We then have:
ZB +
Xij = wj z; + Wijs
To
2g X p
where w; is a d-dimensional vector, and Wi is a noise term. In this experiment, we set = 200, 20, and vary p. For each experiment, we sample single æa ~ NN Ia) and single set of w vectors, each from (O, Ia), and keep them fixed. We leave the other variables random with the following distributions: z; ~ (0, Ia). E ~ o") 3 ~ N (o, aa and Uzy N( 1). In Fig. 4a we plot the empirical membership advantage values, which increase with the number p of features in the overparameterized regime.
Noise-Free Time-Series Regression: In this experiment, consider model that aims to interpolate a time-series signal using frequency components as the features. For example, consider a patient who visits hospital at irregular times to get their blood glucose level measured. After obtaining a number of measurements taken over time, the hospital fits time-series signal representing the patient's blood glucose level any time. Using this learned model, an adversary wishes identify whether the patient visited the hospital at particular time lg. that is, if to is one such time point included the hospital's 's training dataset for learning the patient's 's blood glucose level.
To formalize this, we fix D 1024 as the number of frequency components each signal contains D matrix whose elements W At /W4z cos (2a4) That
and let M 2D Let W be the is, each column of the matrix half a period of an M-dimensional discrete cosine with frequency 2ar Sample a 3 ~ (o, D) and let z = W3 denote the true length- signal. Thus, the signal random linear combination of cosines. We randomly select 128 indices from 2, D uniformly without replacement, and let X be the n D matrix whose rows are the rows of W at these selected indices. Then, y X is the signal observed the randomly selected n indices. The regressor learns 3 X! and then predicts the signal any other time point as To _T. where Zo is row to of W. Thus, identifying whether to was a time point in the training dataset is equivalent identifying if 2o
M
in X. The membership advantage values for this task, plotted in Figure Hb increases with the number of frequency components included in the model.
Random ReLU Features: We next consider nonlinear data model based on Random ReLU feature networks B3 | Let Z be a random n x matrix whose elements are iid standard normal. Let V be random D matrix whose
sampled iid from the surface of the unit sphere in IRP Let = max ZV, 0), where the max taken elementwise. The target variables are given by = Z + de, where B
(0, ID) and e ~ (o, Finally, for the data point æp. let its prediction be a X y. We plot the membership advantage in Figure 4 with n 100, D 5000, and a We again observe that in the highly overparametrized regime, membership advantage increases with parameters.

---

Discussion and Conclusions
We have shown theoretically for (regularized) linear regression with Gaussian data and empirically for more complex models (latent space regression, time-series regression using Fourier components, and random ReLU features) that increasing the number of model parameters renders them more vulnerable to membership inference attacks. Thus, while overparameterization may be attractive for its robust generalization performance, one must proceed with caution to ensure the learned model does not lead to unintended privacy leakages.
More speculatively, we hypothesize that the same overppaaneteiizatoovvvvlenniiiii tradeoff should exist in many machine learning models (c.g.. deep networks) beyond those we have studied. Intuitively, the output of model that achieves zero training eπтоr but generalizes well must i) fit to any noise (e.g. additive Gaussian noise) present in the training data to get a perfect fit to the noisy training data but also ii) eliminate the effect of noise in the training data predicting for unseen data achieve good generalization. This disparate behavior towards training and non- con-training data points leads to different output distributions when the input is not among the training data and is universal for overparameterized models. Ultimately, this causes difference in the distributions of training and test predictions that can be leveraged to perform a membership attack.
There are still many open questions this line of research. While we have shown multiple settings where reducing the number of parameters can increase privacy, remains to be verified that the phenomenon holds widely for other types of machine learning settings such as language tasks or large-scale image recognition. Another interesting next step would be to investigate how increased overparameterization affects privacy for models trained with crivacy-preserving techniques or mem- bership inference defense schemes other than ridge regularization. We believe the findings of our work can provide insights towards developing the next generation of -reeeeeree techniques. is our hope that the observations and analyses in this paper take step towards keeping sensitive training data safer in world increasingly intertwined with machine learning.
Acknowledgements
This work was supported by NSF grants CCF-1911094, I18-1838177, and IIS-1730574: ONR grants N00014-18-12577 -00014-20-1-2534, and MURI N00014-20-140 1-2-777----7 AFOSR grant FA9550-22-1- 0060; and Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047.
10

---

References
[1] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagiclski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ülfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Sympo- sium, pages 2633-2650, August 2021.
[2] Bo Liu, Ming Ding, Sina Shaham, Wenny Rahayu, Farhad Farokhi, and Zihuai Lin. When machine learning meets privacy: A survey and outlook. ACM Computing Surveys, 54(2), Mar. 2021.
[3] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903.08566 2019.
[4] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03966 2021.
[5] Junyang Lin, An Yang, Jinze Bai, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Yong Li, Wei Lin, Jingren Zhou, and Hongxia Yang. M6-10t: A sharing-delinking paradigm for efficient multi-trillion parameter pretraining. arXiv preprint arXiv: 10.03888, 2021.
[6] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116632):15849-15533
2019.
[7] Yehuda Dar, Vidya Muthukumar, and Richard G Baraniuk. A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning. arXiv preprint MXXi:22009.02355. 2021.
[8] Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. In 29th USENIX Security Symposium, pages 1602-1622, August 2020.
[9] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE 31st Computer Security Foundations Symposium, pages 268-282, 2018.
[10] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. arXiv preprint arXiv:21/2.03570, 2021.
[11] Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg- Kirk-Kikka, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint arXiv:2203.03929 2022.
[12] Hongsheng Hu, Zoran Salcie, Gillian Dobbie, and Xuyun Zhang. Membership inference attacks on machine learning: A survey. arXiv preprint arXiv:2 103. 100.0023300...853.. 2021.
[13] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, pages 3-18, 2017.
[14] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, pages
2019.
[15] Erick Galinkin. The influence of dropout on membership inference in differentially private models. arXiv preprint wxXiv:2103.09000 2021.
[16] Yijue Wang, Chenghong Wang. Zigeng Wang, Shanglin Zhou, Hang Liu, Jinbo Bi, Caiwen Ding, and Sanguthevar Rajasckaran. Against membership inference attack: Pruning is all you need, arXiv preprint arXiv:2008. 13578, 2020.
[17) Yunhui Long, Vincent Bindschaedler, and Carl A Gunter. Towards measuring membership privacy. arXiv preprint arXivv1712.0091333 2017.

---

[18] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint (rXivv:800.002200 2018.
[19] Christopher A. Choquette-Choo Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label- only membership inference attacks. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 1964-1974, July 2021.
[20] Yigitcan Kaya and Tudor Dumitras. When does data augmentation help with membership inference attacks? In International Conference on Machine Learning, pages 5345-5355, 2021.
[21] Aadesh Bagmar, Shishira R Maiya, Shruti Bidwalka, and Amol Deshpande. Membership inference attacks on lottery ticket networks. arXiv preprint wrXiv:2108.03506, 2021.
[22] Shahbaz Rezaci, Zubair Shafiq, and Xin Liu. Accuracy-privary trade-off in deep ensemble. arXiv preprint arXiv:2105.05388 2021.
[23] Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 2(4): 2020.
[24] Cynthia Dwork. Differential privacy: A survey of results. In International Conference on Theory and Applications of Models of Computation, pages 1-19, 2008
[25] Mark Bun, Jonathan Ullman, and Salil Vadhan. Fingerprinting codes and the price of approxi- mate differential privacy. SIAM Journal on Computing,
2018.
[26] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30033-3007 2020.
[27] Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. arXiv preprint arXiv:2009.14286 2020.
[28] Bo Hui, Yuchen Yang, Haolin Yuan, Philippe Burlina, Neil Zhenqiang Gong, and Yinzhi Cao. Practical blind membership inference attack via differential comparisons. arXiv preprint arXiv:2101.0134/ 2021.
[29] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, and Reza Shokri. Enhanced membership inference attacks against machine learning models. arXiv preprint arXiv:2111.09677 2021.
[30] Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. On the effectiveness of regularization against membership inference attacks. arXiv preprint arXiv:2006.05333 2020.
[31] Rathindra Sarathy and Krishnamurty Muralidhar. Evaluating Laplace noise addition to satisfy differential privacy for numeric data. Transactions on Data Privacy, 4(1):1-17, 2011.
[32] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Proceed- ings of the 20th International Conference on Neural Information Processing Systems, pages 1177-1184 1184, 2007.
[33] Youngmin Cho. Kernel Methods for Deep Learning. PhD thesis, University of California, San Diego 2012.
[34] Francisco Rubio and Xavier Mestre. Spectral convergence for general class of random matrices. Statistics & Probability Letters, 6555)5922600, 201
[35] Edgar Dobriban and Yue Sheng. Wonder: Weighted one-shot distributed ridge regression in high dimensions. Journal of Machine Learning Research, 21:66 1, 2020.
[36] Edgar Dobriban and Yue Sheng. Distributed linear regression by averaging. The Annals of Statistics, 49(2)9918-943, 2021.
[37] R Dennis Cook and Liliana Forzani. On the mean and variance of the generalized inverse of a singular Wishart matrix. Electronic Journal of Statistics, 5:146-158, 2011.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes], please see Section A.
(c) Did you discuss any potential negative societal impacts of your work? (es), please see Section A.2
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "rsogH4NSt-r", "reviewer": "Reviewer_YQ6y", "summary": "This paper considers the problem of membership inference, i.e. guessing whether a given sample was in a model's training set. The authors propose to use the tools of random matrix theory in the asymptotic regime to analyze membership inference in the simple case of a linear model on Gaussian data. The authors start by deriving the explicit advantage of the attack in the asymptotic regime. Further derivations allow the author to analyze several interesting machine learning ingredients, starting from L2 regularization (ridge regression). There, the authors show that regularization has the counter intuitive effect of increasing the performance of membership inference attacks. The authors go on to show that feature reduction is more effective than noise addition in the generalization error - membership advantage tradeoff. Finally, the authors consider more complex models such as linear regression on latent space models or noise free time series regression.", "questions": "- Can you clarify if ||x_0, p||^2  ~ p in Theorem 3.2?", "limitations": "The authors have addressed limitations and potential negative societal impact (Appendix A.1 and A.2)", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper tackles an important problem\n- The paper is well-written and easy to follow\n- The setup of using random matrix theory to analyze membership inference on linear models is interesting and to the best of my knowledge new \n- The results predicted by the theory seem to match closely the empirical results on Gaussian data\n\nWeaknesses\n- I am not very familiar with random matrix theory but some notations seem surprising to me, e.g. in Theorem 3.2 ||x_0, p||^2 should be p in the asymptotic regime if I understand correctly. \n- I find the term \"generalization error\" to be ambiguous, in this case it refers really to the \"test\" performance rather than the train-test gap. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ivjj5UGVvw", "reviewer": "Reviewer_uY2r", "summary": "This paper first mathematically formalizes the existing empirical observation that larger models (linear regression) memorize more data (through the membership inference looking glass), and then provides some experiments on slightly more complicated models to further support the conclusion that having smaller models is better for privacy, even better than adding noise/ regularizing during training. ", "questions": "1. As mentioned in the weaknesses, I am curious to see the distribution of MIA success on individual samples, when comparing noise addition to making model size smaller. ", "limitations": "Apart from the questions above, the limitations are addressed. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The observation that increasing the ridge regularization coefficient increases memorization is new and interesting.\n\n\nWeaknesses:\n\n1. The fact that larger models memorize more has been shown abundantly in the past (not theoretically, but this paper only looks at linear regression theoretically), for instance in [1] and [2] this is shown for GPT-2 and BERT families, respectively. \n\n2. The conclusion that downsizing models is better than adding noise for privacy is too broad and even detrimental. I think it is important to clarify that smaller models only memorize less, 'on average'.  I wonder if the observation of figure 3c holds only when we look at average memorization. I would guess if we look at individual sample memorization, the worst case  would be better on the noise addition method (which is similar to DP).  I think calling smaller models a privacy mitigation/defense could have negative consequences. \n\n3. This final point is not completely relevant, but as far as the broad conclusion of smaller models are better for privacy goes, there is another way in which it is not accurate. Recent work has shown that  when DP-SGD is used for training models, it is in fact better to have larger, pre-trained models that you fine-tune on [3,4]. I understand that this is outside the scope of this paper. \n\nSummary: The paper formalizes empirical observations and existing heuristics, but other than that I do not see any added insights that would change the way we deploy or think about models. Yes smaller models are better in terms of memorization, when you have smaller data, but as datasets get larger we need to scale up. I will increase my score if my understanding of parts of the paper is lacking and I have misunderstood something.\n\n[1] Carlini N, Tramer F, Wallace E, Jagielski M, Herbert-Voss A, Lee K, Roberts A, Brown T, Song D, Erlingsson U, Oprea A. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21) 2021 (pp. 2633-2650).\n\n[2] Mireshghallah F, Goyal K, Uniyal A, Berg-Kirkpatrick T, Shokri R. Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks. arXiv e-prints. 2022 Mar:arXiv-2203.\n\n[3] Li X, Tramer F, Liang P, Hashimoto T. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679. 2021 Oct 12. (ICLR)\n\n[4] Yu D, Naik S, Backurs A, Gopi S, Inan HA, Kamath G, Kulkarni J, Lee YT, Manoel A, Wutschitz L, Yekhanin S. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500. 2021 Oct 13.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7Fk2erIytID", "reviewer": "Reviewer_ggEs", "summary": "Membership inference attacks are one way to break (differential) privacy by identifying whether a given instance belongs to the dataset of an accessible model trained on that data set or not. Neural nets (and sometimes other models) are shown to be susceptible to such attacks, due to their memorization. \n\nThis paper aims to show the intuitive fact that a learning algorithm with more parameters can provably and empirically be more susceptible to membership inference attacks.\n\nThe focus of the paper is on linear regression and it shows that learning such functions with more parameters (in a formal sense) not only leads to more accuracy but also more vulnerability to membership inference.  The exact formalization is by picking a challenge instance with probability ½ from the training set and ½ as a fresh instance and computing “advantage” eps such that adversary's chance of winning is $(1+\\epsilon)/2$ (achieving advantage $\\epsilon=0$ is trivial). To control the parameters the learner is limited to picking the model to be of the form $(x_1,...x_p,0,0,...)$ with total dimension D (and effective dimension p) when the data examples are of dimension D. The paper then analytically computes the exact optimal strategies for guessing the correct answer (with maximum probability) by using the Bayes optimal rule for hypothesis testing.\n\nThe paper then studies a few other scenarios:\n\n-specific (ridge) regularization turns out to be not helpful for preventing MI.\n\n-it is shown that for the same generalization error, using fewer parameters is more effective (against MI attacks) than adding *independent* noise to the answers. This should not be confused with adding noise in a central way to get DP schemes. \n\n-The paper then studies the same phenomenon in other contexts such as time series and latent spaces. I particularly liked the time series setting, because it models an interesting scenario in which a person's dates of visiting the hospital might be deduced from the (approximate) Fourier representation of the sequence of visits.\n", "questions": "I don't have a questions that fundamentally changes my mind about the paper. Some minor questions are listed above.", "limitations": "I see no serious societal down side with this paper. Quite the contrary, it studies an important question about privacy and it does it formally. However I think the paper should address its limitations discussed above (as weaknesses) in more detail.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength is that the paper formally studies a well motivated question and is among the first to do it *formally*. This is a big plus Understanding what causes membership inference is an important question and overparameterization is a well-known suspect. It is helpful to have a deeper understanding of this phenomenon. \n\nOn the down side:\n\n-It is not clear from this work whether a method *designed* for resisting MI would be hurt in the overparameterized regime. In particular, …\n\n-No *comparison* (between this paper’s methods for positive solutions resisting attacks) with differentially private methods is made. Ideally, I would have loved to see a comparison between the “pruning” method and DP-linear regression. \n\n-The study is only for regression.\n\n-One major limitation is that the membership inference’s adversary is very limited. The typical (I’d say standard) adversary shall be able to at least make more than 1 query to the model (or even have the model given in a full whilte box setting) while the attacker of this paper gets a *single* query. The paper seems to address this when saying “We emphasize that our results on the blind adversary lower bound the performance of adversaries…” which is true, but it is also true that by limiting the adversary the comparison baseline changes and you *compare* limited adversaries for different levels of parameterization.\n\n-Overparameterization is a major issue for neural nets and NNs also suffer from MI, but the formal study here is for the clean setting of basically everything being Gaussian (and the tasks being linear regression and variants). \n\n-The paper over-interprets its findings and ends with, in my opinion, dangerous recommendations such as “simply reducing the 15 parameters of an overparameterized model is an effective strategy”. What this paper does is to show that in a *very specific* setting this is the case, and making a general blanket statement like this is not correct.\n\nEditorial comments:\n\nL 39: “prove that that”\n\nL 64: “MI can be seen as a special case of differential privacy (DP)” \nThis sentence is not precise. MI is an attack and DP is a defense.\n\nL 90: f_S in the loss should be just f, and the minimization is done with respect to a class of functions (as otherwise there is usually a trivial function to reach loss zero).\n\nL 110: “training algorithm f_S,” -> training algorithm T\n\nProposition 3.1, just note that the probabilities here are not real probabilities but rather density functions (in case of continuous distributions).\n\nL143: I_d -> L_D\n\nTheorem 3.2 please don’t be too concise: explain the range of i, explain what happens to m=0, and please use parentheses as in (y|m=0,x_0)~N… \nright now it took me a while to even get to read the math here due to lack of parentheses.\nAlso, my calculation gives a different alpha (e.g., the fraction in the logarithm does not have the squared exponents). Please either give a computation or a citation.\n\nLine 160: you cannot simply reverse the variances, you need to flip the decision of the adversary too.\n\nProp 4.1: why does the last expectation (about the length of x_0,p) exist in this statement? Why not substitute it with the amount of this expectation which can be computed since you pick things from Gaussian.\nAlso, why does not this proposition appear in Section 4.1? It seems not relevant to Section 4.2 and more relevant to 4.1 .\nAlso, how is it the case that this proposition “follows immediately as a consequence of Theorem 3.2.”\n\nL 233: “However, as has been shown in a variety of works” I don’t get the relevance of this sentence.\n\nL 252: y should be \\hat{y}_0\n\nMany of the graphs have insufficient explanation. For example, for Figure 4, how are n,p,D exactly chosen? Only gamma is given .\n\nPage 8 gets too dense. I find many of the sentences in the last paragraph before Section 5 imprecise and unhelpful. For example “The bottom left in the plot represents the nexus of perfect generalization and no membership advantage”. Where is the bottom left of the plot exactly? Similarly “lower and further to the left in the figure”\n\nPage 9 is also too dense. What is the exact setting of the Time-Series Regression? In particular, is it still the case that an adversary who guesses at random has advantage 0?\n\nL 328: what does “max is taken elementwise” mean?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept