Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work analyzes the accuracy-privacy trade-off in ensemble learning by performing model inference attacks. The key finding of the paper is that the presence of an ensemble (that averages the predictions of individual learners) exacerbates the disparity between the confidence distribution of samples that were seen during training v/s those that weren't. They highlight how the main reason for this observation is the reduced agreement between base models for data points that were not seen during training. There is some evaluation of prior membership inference defenses in the ensemble setting.
main_review: ## Strengths
1. The division of confidence values by the number of ensemble learners who correctly classify a sample is a very useful way of distinguishing the reason for the distribution change -- it is not because these models have inherently lower confidence on test examples, but because they incorrectly answer them more often. These observations can be very interesting to the whole of MI literature -- because they suggest that all MI attacks in practice do as well as "Gap Attack" (when not considering an ensemble). However, this also makes me wonder if the authors used the strongest attack setting for attacking the ensemble models.
2. The experiments are comprehensively evaluated across a wide variety of datasets, training settings, defense settings -- which helps to be confident about the generality of the results.
3. The ensemble models appear to perform significantly worse in terms of privacy than their base model counterparts. The authors could emphasize the percentage increase in privacy risk.

## Suggestions
1. It would be nice to have some background about previously attributed causes of the success of ensembling techniques. Because from the language in the paper, it appears that the authors are the first to attribute that ensembling increases confidence on train samples, and not on the test set (on average)

2. Since this paper is about "deep ensemble models", it appears incomplete without attention to ensemble-based defenses such as those highlighted by the authors -- like training the ensembles on different subsets of datasets ( which are not necessarily disjoint) -Salem et. al. Or in related work-- Huang et al. (2020); Li et al. (2021); Rahimian et al. (2020); Yang et al. (2020). I would appreciate if this paper was able to categorize scenarios in which ensembles are privacy-preserving, and when they are not.


## Questions
1. Do you have access to all n base models in the ensemble when this is black-box? Based on the graphs, there does not appear to be much difference in the white-box and black-box performance.
2. "We consider a worst-case scenario where 80% of the training dataset is given to the attacker and the goal is to infer the membership of the remaining samples, similar to Rezaei & Liu (2021)" -- Can you explain what this setting is in more detail? 
3. Are all models trained on complete training sets? Or on smaller splits of the training dataset (as is done in many MI papers to get the attack performance high)


## Writing
1. Please use \citep for parenthesized citations.
2. For graphs in the appendix, what is the y-axis? It appears to be the reverse of what was followed in the main paper.
4. Sometimes the language of the paper is hand-wavy and unsubstantiated. Such as:
“As they move from under-fitted region to overfitted region, they start to memorize features more specific to train samples Feldman (2020). Consequently, the average gap of correct agreement between train and test set widens. Hence, the wider *the* generalization gap of base learners is, the more effective the membership inference attack would be on *the* ensemble. “
There is no experiment that talks about memorization per se. And membership inference must not be taken as a proxy for memorization to describe itself.
5. “In the black-box setting, the attacker only access the output of the ensemble “ --> has access to

## Post Rebuttal
I am keeping my score unchanged. The following is my response to authors:

---
Thank you for your detailed comments! The paper started with an interesting premise -- some prior work says ensemble methods help reduce MI threat, while there is evidence for ensemble hurting MI. Let us understand what's going on and set things straight. Unfortunately, at the end of the paper, the discussion and experimentation do not live up to this initial expectation. Apart from a small curve for partitioning, bagging in Figure 5, there is hardly any discussion on when and what types of ensemble methods help against MI.

The goal of requesting additional experiments was not to make you do additional experiments and fill up the Appendix, but to highlight how this discussion is very useful for the reader. I thank you for the efforts you put in with the experiments in the Appendix. Unfortunately, they do not see any discussion in the paper. For example, while the authors show how confidence varies with ensemble size -- what happens when this ensemble is not just model randomization but partitioning -- what if this was 10% sets, or 50% sets. What is the tradeoff now? The size of these experiments may seem large, but I am trying to point out from a reader's view -- what types of discussions in the paper (and conclusions) will help resolve the initial dilemma -- should I do ensemble or not? If i want to protect myself from MI.

I am sorry that I am delayed in responding to you. But I believe that while this work is beneficial to the community, it only partially caters to the expectation it built. Re-prioritization and minor additional experiments can make this paper a delight for the reader!

summary_of_the_review: While the paper does provide some interesting results about the privacy threat of deep ensemble models, it lacks in the discussion of "deep ensemble models" as a whole -- which has been discussed as a defense in prior work. Further, some of the claims can be further substantiated as highlighted in my review. Finally, it would be good to clarify if the finding of the change in confidence distribution (for deep ensembles) of train and test samples has been observed in prior work, or is a new contribution in this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provide a systemantic analysis on the accuracy-privacy trade off for deep ensmebles. They show that the effectiveness of
membership inference attacks is likely to increase when ensembling improves accuracy. The authors further study  the impact of various factors such as prediction confidence and agreement between models that constitute the ensemble.
main_review: Strength:
1. The idea of studying the accuracy-privacy trade off is interestrting and relatively new.
2. The conclusion on the effectivess of the level of correct agreement among models is important and likely to motivate more related future work towards this direction.
Weakness:
1. The expriments and conclusions are all based on a very simple ensemble setting: averaging the individual predictions.  However in practice, we usually use some more advanced aggregating stratgies such as weighted averaging. In this case, it is not clear whether the conclusions in the paper will still be applicable.
2. Instead of using underfitting ensembles, which usually yield less satisfactory accuracies,  to reduce the MI attack, the authors could also try some advanced ensemble methods. such as those that have enhanced diversities. Some good examples include:
The Diversified Ensemble Neural Network, Deep Negative Correlation Learning

summary_of_the_review: Generally speaking, this is an interesting paper because the ``accuraacy and privacy" trade off is under-resaearched. However, the experimental setting is relatively resrective. It is interesting to see whether the conclusion still holds for more advanced and generic ensemble settings.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes and analyzes an interesting and curious phenomenon: a standard ensemble of models is typically more vulnerable to membership inference attacks, despite being less overfit than any single model.
main_review: This paper's main insight---that ensembles can be more vulnerable to membership inference---is interesting and a priori somewhat surprising.
But this paper nicely explains how ensembling strengthens privacy leakage: for training examples, all models tend to be correct and highly confident so the ensemble's prediction is essentially the same as any individual model. But for test examples, there is a fraction of examples where the ensemble disagrees. And these become easy to recognize as they have lower confidence.

The most interesting result in my opinion is in Figure 4, where the authors show that as overfitting *decreases* (and thus the trivial gap-attack becomes weaker), membership leakage actually *increases*. I think this result would merit to be discussed more prominently much earlier in the paper, as it runs contrary to some of the beliefs in the literature on membership inference attacks.

While the observation in this paper is quite simple, it is analyzed in depth across a number of setups, datasets, architectures to form a convincing story.
A few suggestions here:
- Figure 1 (& Figure 5) uses a fairly "bad" model that achieves <80% accuracy on CIFAR-10. Do the results still hold if you switch to a better setup (e.g., a Wide-ResNet with standard data augmentation should easily reach >92% accuracy).
- Figure 2 shows that "something" changes qualitatively with the distributions, but it is hard to convincingly say that one should lead to a stronger attack just via visualization. It would be useful here to also provide some quantitative measure of the distinguishability of these distributions (e.g., a simple TV distance).

Other comments:
- The introduction mixes two different forms of ensembling. It first talks about "neural networks (NN) that are independently trained on the same dataset with different random initialization" and then about how "training each model on a different subset of data makes the ensemble less prone to overfitting". This is a bit confusing, since it isn't clear at this point what type of ensembling will be considered in this paper. The note on the use of subsampled ensembles to defend against MI could maybe be relegated to the related work as it is fairly orthogonal to this paper.
- The graphs in the paper are overall a little hard to read, as they tend to show too many things at once. The light green color used in some plots is also hard to see.
summary_of_the_review: A fairly simple observation but nicely analyzed
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors in the paper perform empirical studies to investigate the trade-off between accuracy and privacy (measured by membership inference attacks) in deep ensembles.
main_review: The authors in the paper perform empirical studies to investigate the trade-off between accuracy and privacy (measured by membership inference attacks) in deep ensembles. They find out that the level of correct agreement among models is the most dominant factor that improves the performance of MI attacks in deep ensembles. They support their claim by visualizing the distribution shifts of correct agreement in train/test examples. They further implement a variety of existing defenses, such as differential privacy and regularizations, etc., to investigate the ​​effects of existing defense mechanisms. Overall, the paper is well-written and the experiments are well conducted.

My biggest concerns on the paper are its limitations of novelties and contributions in the literature of membership inference attacks and privacy-preserving machine learning. The biggest finding of the paper is that the improved advantage of deep ensembles on MI attacks mainly comes from the level of correct agreement among models in the deep ensembles. The finding is purely empirical without theoretical justifications. Besides, it is not sure whether the similar conclusions extend to ensemble learning methods, which limits its contributions to the community. 

Besides, the no defense mechanism targeted for the finding is proposed and the defense methods evaluated in the experiments already existed. The privacy-utility trade-offs are expected and it seems the trade-offs are similar to the settings where no deep ensembles are adopted. 

summary_of_the_review: Although the paper is well-written and the experiments are well conducted, I think the novelties and contributions of the paper is not enough for acceptance: the authors just find out the level of correct agreement among models in the deep ensembles improves MI attacks in a particular deep ensemble setting, no algorithms nor theoretical insights targeted for the setting is provided.
review_title: null

Meta-review: The authors in the paper perform empirical studies to investigate the trade-off between accuracy and privacy (measured by membership inference attacks) in deep ensembles. They find out that the level of correct agreement among models is the most dominant factor that improves the performance of MI attacks in deep ensembles. They support their claim by visualizing the distribution shifts of correct agreement in train/test examples. They further implement a variety of existing defenses, such as differential privacy and regularizations, etc., to investigate the ​​effects of existing defense mechanisms. Overall, the paper is well-written and the experiments are well conducted.
While these findings are interesting, they do not reveal something useful or surprising about deep ensemble learning. It is not clear what the contribution is to the membership inference attack literature and private machine learning literature. they do not propose anything new to make the attacks stronger or defenses stronger.
Decision: Reject