Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Membership inference attacks are one way to break (differential) privacy by identifying whether a given instance belongs to the dataset of an accessible model trained on that data set or not. Neural nets (and sometimes other models) are shown to be susceptible to such attacks, due to their memorization. 

This paper aims to show the intuitive fact that a learning algorithm with more parameters can provably and empirically be more susceptible to membership inference attacks.

The focus of the paper is on linear regression and it shows that learning such functions with more parameters (in a formal sense) not only leads to more accuracy but also more vulnerability to membership inference.  The exact formalization is by picking a challenge instance with probability ½ from the training set and ½ as a fresh instance and computing “advantage” eps such that adversary's chance of winning is $(1+\epsilon)/2$ (achieving advantage $\epsilon=0$ is trivial). To control the parameters the learner is limited to picking the model to be of the form $(x_1,...x_p,0,0,...)$ with total dimension D (and effective dimension p) when the data examples are of dimension D. The paper then analytically computes the exact optimal strategies for guessing the correct answer (with maximum probability) by using the Bayes optimal rule for hypothesis testing.

The paper then studies a few other scenarios:

-specific (ridge) regularization turns out to be not helpful for preventing MI.

-it is shown that for the same generalization error, using fewer parameters is more effective (against MI attacks) than adding *independent* noise to the answers. This should not be confused with adding noise in a central way to get DP schemes. 

-The paper then studies the same phenomenon in other contexts such as time series and latent spaces. I particularly liked the time series setting, because it models an interesting scenario in which a person's dates of visiting the hospital might be deduced from the (approximate) Fourier representation of the sequence of visits.

strengths_and_weaknesses: The strength is that the paper formally studies a well motivated question and is among the first to do it *formally*. This is a big plus Understanding what causes membership inference is an important question and overparameterization is a well-known suspect. It is helpful to have a deeper understanding of this phenomenon. 

On the down side:

-It is not clear from this work whether a method *designed* for resisting MI would be hurt in the overparameterized regime. In particular, …

-No *comparison* (between this paper’s methods for positive solutions resisting attacks) with differentially private methods is made. Ideally, I would have loved to see a comparison between the “pruning” method and DP-linear regression. 

-The study is only for regression.

-One major limitation is that the membership inference’s adversary is very limited. The typical (I’d say standard) adversary shall be able to at least make more than 1 query to the model (or even have the model given in a full whilte box setting) while the attacker of this paper gets a *single* query. The paper seems to address this when saying “We emphasize that our results on the blind adversary lower bound the performance of adversaries…” which is true, but it is also true that by limiting the adversary the comparison baseline changes and you *compare* limited adversaries for different levels of parameterization.

-Overparameterization is a major issue for neural nets and NNs also suffer from MI, but the formal study here is for the clean setting of basically everything being Gaussian (and the tasks being linear regression and variants). 

-The paper over-interprets its findings and ends with, in my opinion, dangerous recommendations such as “simply reducing the 15 parameters of an overparameterized model is an effective strategy”. What this paper does is to show that in a *very specific* setting this is the case, and making a general blanket statement like this is not correct.

Editorial comments:

L 39: “prove that that”

L 64: “MI can be seen as a special case of differential privacy (DP)” 
This sentence is not precise. MI is an attack and DP is a defense.

L 90: f_S in the loss should be just f, and the minimization is done with respect to a class of functions (as otherwise there is usually a trivial function to reach loss zero).

L 110: “training algorithm f_S,” -> training algorithm T

Proposition 3.1, just note that the probabilities here are not real probabilities but rather density functions (in case of continuous distributions).

L143: I_d -> L_D

Theorem 3.2 please don’t be too concise: explain the range of i, explain what happens to m=0, and please use parentheses as in (y|m=0,x_0)~N… 
right now it took me a while to even get to read the math here due to lack of parentheses.
Also, my calculation gives a different alpha (e.g., the fraction in the logarithm does not have the squared exponents). Please either give a computation or a citation.

Line 160: you cannot simply reverse the variances, you need to flip the decision of the adversary too.

Prop 4.1: why does the last expectation (about the length of x_0,p) exist in this statement? Why not substitute it with the amount of this expectation which can be computed since you pick things from Gaussian.
Also, why does not this proposition appear in Section 4.1? It seems not relevant to Section 4.2 and more relevant to 4.1 .
Also, how is it the case that this proposition “follows immediately as a consequence of Theorem 3.2.”

L 233: “However, as has been shown in a variety of works” I don’t get the relevance of this sentence.

L 252: y should be \hat{y}_0

Many of the graphs have insufficient explanation. For example, for Figure 4, how are n,p,D exactly chosen? Only gamma is given .

Page 8 gets too dense. I find many of the sentences in the last paragraph before Section 5 imprecise and unhelpful. For example “The bottom left in the plot represents the nexus of perfect generalization and no membership advantage”. Where is the bottom left of the plot exactly? Similarly “lower and further to the left in the figure”

Page 9 is also too dense. What is the exact setting of the Time-Series Regression? In particular, is it still the case that an adversary who guesses at random has advantage 0?

L 328: what does “max is taken elementwise” mean?

questions: I don't have a questions that fundamentally changes my mind about the paper. Some minor questions are listed above.
limitations: I see no serious societal down side with this paper. Quite the contrary, it studies an important question about privacy and it does it formally. However I think the paper should address its limitations discussed above (as weaknesses) in more detail.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considers the problem of membership inference, i.e. guessing whether a given sample was in a model's training set. The authors propose to use the tools of random matrix theory in the asymptotic regime to analyze membership inference in the simple case of a linear model on Gaussian data. The authors start by deriving the explicit advantage of the attack in the asymptotic regime. Further derivations allow the author to analyze several interesting machine learning ingredients, starting from L2 regularization (ridge regression). There, the authors show that regularization has the counter intuitive effect of increasing the performance of membership inference attacks. The authors go on to show that feature reduction is more effective than noise addition in the generalization error - membership advantage tradeoff. Finally, the authors consider more complex models such as linear regression on latent space models or noise free time series regression.
strengths_and_weaknesses: Strengths
- The paper tackles an important problem
- The paper is well-written and easy to follow
- The setup of using random matrix theory to analyze membership inference on linear models is interesting and to the best of my knowledge new 
- The results predicted by the theory seem to match closely the empirical results on Gaussian data

Weaknesses
- I am not very familiar with random matrix theory but some notations seem surprising to me, e.g. in Theorem 3.2 ||x_0, p||^2 should be p in the asymptotic regime if I understand correctly. 
- I find the term "generalization error" to be ambiguous, in this case it refers really to the "test" performance rather than the train-test gap. 
questions: - Can you clarify if ||x_0, p||^2  ~ p in Theorem 3.2?
limitations: The authors have addressed limitations and potential negative societal impact (Appendix A.1 and A.2)
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper first mathematically formalizes the existing empirical observation that larger models (linear regression) memorize more data (through the membership inference looking glass), and then provides some experiments on slightly more complicated models to further support the conclusion that having smaller models is better for privacy, even better than adding noise/ regularizing during training. 
strengths_and_weaknesses: Strengths:

1. The observation that increasing the ridge regularization coefficient increases memorization is new and interesting.


Weaknesses:

1. The fact that larger models memorize more has been shown abundantly in the past (not theoretically, but this paper only looks at linear regression theoretically), for instance in [1] and [2] this is shown for GPT-2 and BERT families, respectively. 

2. The conclusion that downsizing models is better than adding noise for privacy is too broad and even detrimental. I think it is important to clarify that smaller models only memorize less, 'on average'.  I wonder if the observation of figure 3c holds only when we look at average memorization. I would guess if we look at individual sample memorization, the worst case  would be better on the noise addition method (which is similar to DP).  I think calling smaller models a privacy mitigation/defense could have negative consequences. 

3. This final point is not completely relevant, but as far as the broad conclusion of smaller models are better for privacy goes, there is another way in which it is not accurate. Recent work has shown that  when DP-SGD is used for training models, it is in fact better to have larger, pre-trained models that you fine-tune on [3,4]. I understand that this is outside the scope of this paper. 

Summary: The paper formalizes empirical observations and existing heuristics, but other than that I do not see any added insights that would change the way we deploy or think about models. Yes smaller models are better in terms of memorization, when you have smaller data, but as datasets get larger we need to scale up. I will increase my score if my understanding of parts of the paper is lacking and I have misunderstood something.

[1] Carlini N, Tramer F, Wallace E, Jagielski M, Herbert-Voss A, Lee K, Roberts A, Brown T, Song D, Erlingsson U, Oprea A. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21) 2021 (pp. 2633-2650).

[2] Mireshghallah F, Goyal K, Uniyal A, Berg-Kirkpatrick T, Shokri R. Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks. arXiv e-prints. 2022 Mar:arXiv-2203.

[3] Li X, Tramer F, Liang P, Hashimoto T. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679. 2021 Oct 12. (ICLR)

[4] Yu D, Naik S, Backurs A, Gopi S, Inan HA, Kamath G, Kulkarni J, Lee YT, Manoel A, Wutschitz L, Yekhanin S. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500. 2021 Oct 13.
questions: 1. As mentioned in the weaknesses, I am curious to see the distribution of MIA success on individual samples, when comparing noise addition to making model size smaller. 
limitations: Apart from the questions above, the limitations are addressed.
review_title: null

Meta-review: metareview: This paper considers the problem of membership inference. The authors propose to use the tools of random matrix theory in the asymptotic regime to analyze membership inference in the simple case of a linear model on Gaussian data. They start by deriving the explicit advantage of the attack in the asymptotic regime. Further derivations allow the author to analyze several interesting machine learning ingredients, starting from L2 regularization (ridge regression). There, the authors show that regularization has the counterintuitive effect of increasing the performance of membership inference attacks. The referees are leaning toward acceptance and I concur.
confidence: Less certain
Decision: Accept