Accuracy-Privacy Trade-off in Deep Ensemble: A Membership Inference Perspective
Authors: Shahbaz Rezaei
Zubair Shafiq
Xin Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Ensemble Learning
Deep Ensemble
Membership Inference

Abstract: Deep ensemble learning has been shown to improve accuracy by training multiple neural networks and fusing their outputs. Ensemble learning has also been used to defend against membership inference attacks that undermine privacy. In this paper, we empirically demonstrate a trade-off between these two goals, namely accuracy and privacy (in terms of membership inference attacks), in deep ensembles. Using a wide range of datasets and model architectures, we show that the effectiveness of membership inference attacks also increases when ensembling improves accuracy. To better understand this trade-off, we study the impact of various factors such as prediction confidence and agreement between models that constitute the ensemble. Finally, we evaluate defenses against membership inference attacks based on regularization and differential privacy. We show that while these defenses can mitigate the effectiveness of the membership inference attack, they simultaneously degrade ensemble accuracy. We illustrate similar trade-off in more advanced and state-of-the-art ensembling techniques, such as snapshot ensembles and diversified ensemble networks. The source code is available in supplementary materials.

Under review as a conference paper at ICLR 2022
ACCURACY-PRIVACI TRADE-OFF IN DEEP ENSEMBLE: A MEMBERSHIP INFERENCE PERSPECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
Deep ensemble learning has been shown to improve accuracy by training multiple neural networks and fusing their outputs. Ensemble learning has also been used defend against membership inference attacks that undermine privacy. In this paper, we empirically demonstrate a trade-off between these two goals, namely accuracy and privacy (in terms of membership inference attacks), in deep ensembles. Using a wide range of datasets and model architectures, we show that the effectiveness membership inference attacks also increases when ensembling improves accuracy. To better understand this trade-off, we study the impact various factors such as prediction confidence and agreement between models that constitute the ensemble. Finally, we evaluate defenses against membership inference attacks based on regularization and differential privacy. We show that while these defenses mitigate the effectiveness of the membership inference attack, they simultaneously degrade ensemble accuracy. We illustrate similar trade-off more advanced and state-of-the-ar ensembling techniques, such as snapshot ensembles and diversified ensemble networks. The source code available in supplementary materials.
INTRODUCTION
Ensemble learning has been shown improve classification accuracy of neural networks in particular, and machine learning classifiers in general (Kondratyuk et al., 2020; Kuncheva & Whitaker, 2003; Sagi & Rokach, 2018). The most commonly used approach for deep models involves averaging the output of multiple neural networks (NN) that are independently trained on the same dataset with different random initialization, called deep ensemble (Lobacheva et al., 2020). Such a simple approach has been extensively used in practice to improve accuracy (Lee et al., 2015; Wang et al., 2020). Notably, a majority of the top performers in machine learning benchmarks, such as the ImageNet Large Scale Visual Recognition Challenge (Russakovsky et al., 2015), have adopted some form of ensemble learning (Lee et al., 2015; Szegedy et al., 2015; He et al., 2016).
Other forms of ensemble learning (different from deep ensembles), such partitioning, has also been used to defend against privacy-harming membership inference (MI) attacks, where the goal of attacker is infer whether sample has been used train model-i.e., whether the sample belongs to the train set. Membership inference attacks generally use the prediction confidence of NN models to infer membership status of sample (Salem et al., 2018; Shokri al., 2017; Truex et al., 2019; Yeom et al., 2018) by leveraging the insight that trained models may output higher prediction confidence on train samples than non-train samples (Choo et al., 2020). The intuition behind using ensemble learning approaches, like partitioning, to defend against MI attacks is that training each model on a different subset of data makes the ensemble less prone to overfitting (Salem et al., 2018). This idea of using ensemble learning to defend against MI attacks has since been discussed in the literature (Huang et al., 2020; Li et al., 2021; Rahimian et al., 2020; Yang et al., 2020). However, none of these papers theoretically empirically demonstrate usefulness of ensemble learning defense mechanism.
In this paper, we show that these two goals of ensemble learning, namely improving accuracy and defending against MI attack, do not trivially sum up in a unified solution. Figure illustrates accuracy and privacy trade-off by plotting accuracy and membership inference attack effectiveness for ensembles comprising of varying number of base models (1, 2, 5, and 10) that are trained for different numbers of epochs (5, 45, and 90). We make two key observations here. First, there is increase in

---

Under review as a conference paper at ICLR 2022
60
Traning epoches Traning spoch+90
Better Prougyhoouuccy
55
Privacy (100%. Attee
Ensembiing to improve privcy
40
38
Ensemboing improve socusee
30
25
50
60
70
75
80
85
Test Accuracy 196)
Figure I: Trade-off between accuracy and privacy on an AlexNet model trained on CIFARIO. Each curve contains four points corresponding ensembles comprising of 2, 5, and 10 base models (from left to right). Using the single model trained for 90 epochs as baseline, there are two choices: (I) making an ensemble of these models to achieve the highest accuracy possible but worse privacy (purple arrow), or (2) making an ensemble of less overfitted models (epoch #5) to achieve slightly lower accuracy of a single model but better privacy (brown arrow).
both accuracy and MI attack effectiveness as we go from single model to ensembles comprising of an increasing number of base models. The trade-off is more noticeable for more accurate models trained for a larger number of epochs. Second, we can adapt the design of ensembles to suitably navigate the trade-off between accuracy and privacy. Specifically, the two extreme cases are to: (1) maximize accuracy by using an ensemble of highly accurate models but at the cost of worse privacy' (purple arrow); and (2) maximize privacy by intentionally using an ensemble of under-fitte fitted models instead of single model but the cost of accuracy (brown arrow).
To understand the root cause of this trade-off, we show that using deep ensembles to improve accuracy exacerbates its susceptibility to membership inference attacks by making train and non-train samples more distinguishable. By analyzing the confidence averaging mechanism of deep ensembles, we investigate potential factors that enable membership inference. We show that the most influential factor is the level of correct agreement among models. Simply put, the number of models that correctly classify train sample is often greater than the ones that correctly classify a test sample. This results in a wider confidence gap between train and non-train samples, when confidence values are averaged, enabling more effective membership inference attacks.
We further show that the difference in the level of correct agreement between train and non-train samples is correlated with models' generalization gap. Hence, a natural question to ask is "can deep ensembles that use less overfitted models mitigate privacy issues while achieving high accuracy?", To answer this question, we study several regularization techniques, common membership inference defenses, and few other ensembling approaches. We again observe privacy-accurgey trade-off pattern similar to that shown in Figure 1.
Summary of contributions: In this paper, perform systematic empirical study of MI attacks on deep ensemble models. First, we show that when deep ensembles improve accuracy, it also leads to a different distribution shift in the prediction confidence of train and test samples, which in turn enables more effective membership inference. Second, we analyze various factors that potentially cause the prediction confidence of train and non-train samples diverge. Among potential factors, we show that the most dominant factor is the level of correct agreement among models which indicates that more models an ensemble agree on their prediction when sample is a training sample. Hence, the aggregation of their prediction yields higher confidence output in comparison with non-train samples. We show that common defense mechanisms membership inference literature, including differential privacy, MMD+Mixup, L.1 and L2 regularization, as well as other ensemble training approaches, such as bagging, partitioning, and stacking (Salem et al., 2018), can be used to mitigate
Note that for complicated tasks, such as image classification, the common practice is to train deep models for a large number of epochs and avoid under-fitted models. That is because memorizing samples from long-tailed subpopulations shown be necessary to achieve close-to-optimal generalization error (Feldman, 2020).
2

---

Under review as a conference paper at ICLR 2022
effectiveness of MI attacks but at the cost of accuracy. Aithough the main focus of the paper is on deep ensembles, we also cover bagging, partitioning, stacking (Salem et al., 2018), logit averaging (Appendix A.3), weighted averaging (Appendix A.4), well more advanced and state-of-the the-art ensembling techniques, such as snapshot ensembles (Huang et al., 2017) and diversified ensemble networks (Zhang al., 2020) (Appendix A.5). We observe similar trade-off.
2
SYSTEM MODEL
2.1 ENSEMBLE LEARNING
Background. In literature, ensemble learning refers to various approaches that combine multiple models to make a prediction. Models used to construct ensemble are often called base learners. There are two main factors to construct an ensemble (Sagi & Rokach, 2018): 1) how base learners trained to ensure diversity, such as random initialization, bagging, partitioning, etc., and 2) how the output of base learners are fused to obtain the final output, including majority voting, confidence averaging, stacking, etc. Unlike ensemble of traditional machine learning algorithms, in deep ensemble, the main source of diversity often comes only from random initialization of base learners (Fort et al., 2019). In fact, other sources of diversity, such as bagging. have been shown considerably degrade the overall accuracy of a deep ensemble (Lee et al., 2015; Lakshminarayanan et al., 2017).
System Model. We mainly focus on the most widely used deep ensemble (Kondratyuk al., 2020) unless otherwise specified. In this model, 1) base models are trained with random initialization on the same training dataset, and 2) their prediction confidence are fused through averaging. A less common approach is to average model logits which has been used in few studies (Webb et al., 2020; Wang al., 2020). See Appendix A.3 for experimental evaluation of logit averaging and A.4 for weighted averaging ensembles. We also evaluate two state-of-the-e deep ensembling approaches, namely snapshot ensemble and diversified ensemble network, in Appendix A.5. Other general ensembling approaches, such as bagging, partitioning, and stacking (Salem et al., 2018), are studied as defense mechanisms because they degrade accuracy but improve protection against MI attacks.
2.2 MEMBERSHIP INFERENCE
Background. Membership inference is a form of privacy leakage where the goal is to determine if a sample was used during the training of a target model. Samples used during training are often referred to as member or train samples, and other samples are referred to as non-member, non-train, or test samples. The first MI attack on neural networks was proposed in Shokri et al. (2017) where the attacker trains an attack classifier to predict the membership status. The attack classifier takes the prediction confidence of target model as an input. Assuming that the attacker has access a dataset with similar distribution, she trains a set of shadow models to mimic the target model. Since the membership status of the data with which the shadow models are trained are known to the attacker, she can use the data to train the attack classifier. Many papers use the same idea with different variations or less restrictive assumptions (Salem et al., 2018; Liu et al., 2019; Song ct al., 2019; Long ct al., 2017: Truex et al., 2019; Long al., 2018; 8;; Yeom et al., 2018; Rezaei & Liu, 2021; Zou et al., 2020; Li & Zhang, 2020). Most previous work built upon the idea of using prediction confidence to infer the membership status, except for Rezaci & Liu (2021); Choo et al. (2020); Rahimian al. (2020). In Rezaci& & Liu (2021), the authors assumed white-box access to the target model and launched a series of MI attack based on confidence values, distance the decision boundary, gradient w.r.t model weight, and gradient w.r.t input. In Choo et al. (2020), the authors proposed two attacks based on input transformation and distance to the boundary in black-box setting. Similarly, in Rahimian et al. (2020), the attacker randomly perturbs an input to obtain a set of random transformations of the input and uses the predicted labels to infer membership status.
System Model. Since most existing attacks use confidence values, we first focus on changes of confidence values when using deep ensembles. We show that when using deep ensembles the distribution of confidence values becomes more distinguishable between train and non-train set in comparison with non-ensemble case. Consequently, any MI attack that relies on confidence values would be more effective on deep ensembles. Since goal is to show a tra between accuracy and privacy, not to show which confidence-basee attack can slightly outperforms another, we focus on confidence-basee based attack proposed in Rezaei & Liu (2021) in both white-box and black-box
3

---

Under review as a conference paper at ICLR 2022
settings. Here, white-box means the attacker has access to base-learners' output before aggregation, and black-box means the attacker has only access to the aggregated confidence output. Decision coundary-based attacks are extremely computational and query inefficient and it is not trivial how to adopt them for ensemble learning where essentially an input is copied n. times and then fed to n models. Gradient-based approach of Rezaei & Liu (2021) also needs full knowledge of the entire deep ensemble. We consider adaptation of non-confidence-bant based attacks for deep ensembles as future work.
2.3 MEMBERSHIP INFERENCE DEFENSES
Background. Defense mechanisms against membership inference attacks can be summarized into two categories Rahimian al. (2020):
Generaaization-baseed Shokri (Shokri et al., 2017) was first to correlate membership inference success with overfitting. Since then, many standard regularization techniques have been used to alleviate overfitting, such as L1 regularization (Choo et al., 2020), L2 regularization (Choo et al., 2020; Truex et al., 2019; Nasr et al., 2019; Jia al., 2019; Shokri et al., 2017), differential privacy (Choo et al., 2020; Rahimian et al., 2020), dropout (Jia et al., 2019), and adversarial training (Nasr et al., 2018). Interestingly, ensemble learning has also been proposed as a defense mechanism. In Salem et al. (2018), they proposed combination of partitioning and stacking as a defense mechanism. The intuition that training each model with different subset of data makes the entire ensemble model less prone to overfitting. Note that these defense mechanisms often degrade the accuracy of the target model (see Section 4.2) (Choo ct al., 2020).
These defense mechanisms aim to reduce the amount of information that can be obtained from the output of a target model by perturbing (Jia et al., 2019) or limiting the dimensionality of the output (Shokri et al., 2017; Truex et al., 2019; Choo et al., 2020). Majority of these approaches are post-training methods and only work in black-box setting.
Confidence-maskine
System Model. In this paper, we mainly focus on generalization-baase defense mechanisms with the exception of MMD+MMxup eL al., 2021). This is because most confidence-masking approaches manipulate confidence values post-training. As result, the output values of these models do not reliably represent the "confidence" of the model. These approaches are built under the assumption that accurate prediction of confidence not needed. However, many applications require accurate estimation of confidence. Moreover, if the accurate prediction of confidence is not required, then the trivial MI defense would be to only output the class label and avoid these confidence-maste masking defenses altogether.
How DOES ENSEMBLING INCREASE MEMBERSHIP INFERENCE EFFECTIVENESS?
3.1 CONFIDENCE DISTRIBUTION SHIFT
Ensemble learning is only helpful when base learners disagree on some samples (Kuncheva & Whitaker, 2003; Sagi & Rokach, 2018). Otherwise, ensembling does not improve accuracy. Further- more, when deep ensemble is used, the confidence values of multiple models are averaged obtain the final prediction. Consequently, when ensembling improves accuracy, it averages the prediction confidence of highly confident predictions (mostly from models which correctly classified the sample) and less confident predictions (mostly from models which misclassified the sample). As a result, confidence distribution shift is inevitable for both train and test samples. This phenomenon can be observed as the distribution of Figure 2(a) shifts to that of 2(d). This can be better observed by separating correctly classified samples which have significantly higher prediction confidence (Figure 2(e)) and misclassified samples which have lower prediction confidence (Figure 2(f)). One can clearly observe that both distributions shift more towards the center from Figure 2(b) and (c) to Figure 2(e) and (f). However, the change in the distribution of train and test samples does not necessarily cause more effective membership inference if the change has a similar effect on the confidence distribution of both train and test samples. In the next subsection, analyze the potential factors that affect the distribution change and how they can change confidence distribution member and non-menne member sets differently.
4

---

Under review as a conference paper at ICLR 2022
Triin sompless Tha camples
Train samplles
sarppriss
Tenrr saarpe en
10
Freastiyy
0.2
0.4
0.6
0.
1.3
0.0
0.2
0.4
0.5
0.0
15
Modal confiderce
Model confidence
Mtddc confidance
(a) All samples (Non-EL)
(b) Correctly classified samples (Non-EL)
(c) Incorrectly classified samples (Non-EL)
somples
4.5
sumples 22
3.0
7.5-
2.0
2.5
3.0
0.3
0.0
Model midence
Mede cant dence
confidece
(d) All samples (EL-10)
(e) Correctly classified samples (EL-10)
(1 Incorrectly classified samples (EL-10)
Figure 2: AlexNet on CIFARIO (Class #2): Confidence distribution of single model (top row) versus an ensemble of 10 models (bottom row). Jensen Shannon divergence of the two distributions are as follows: a) 2777, b) .155, c) ,2850, d) .3575, e) .2134, and f) .3792.
3.2 EFFECT OF ENSEMBLING ON INDIVIDUAL SAMPLES
We use to denote the confidence value of the ith model in an ensemble of n models. Hence, for a given sample æ. the output of the ensemble is:
Σ (x( Σ (x) + y (z)
yet (x)
(I)
Given single sample, can further divide models in the ensemble into two groups: 1) models that correctly classified the sample denoted by yf and 2) models that misclassified the sample by y." For given sample æ, models correctly classify it and m models misclassify, where that the value of c and m depends on the sample?.
Note
Based on the Eq. (1), three major factors affect the final confidence value (yaz) of sample: yf, y and c. As a result, if these values are different for train and test samples, the ensembling causes different shift in the distributions, and consequently, membership inference attack will be more effective. These factors are as follows:
1. Confidence value of correctly classifying models (y/): Since the majority of samples are supposed to be correctly classified by practical model, any distinguishable confidence difference between train and test samples can lead to very effective membership inference attack. However, as shown in Figure 2(b), we can observe that there is no significant difference between train and test samples.
2. Confidence value of misclassifying models (y""): Unlike correctly classified samples, there is a marginal difference between confidence distribution of train and test samples of misclassified samples (see Figure 2(c)). This may be exploited by membership inference attack to partially distinguish between train and test samples.
3. Level of correct agreement (c) among models: As shown in Figure 3(a), the number of models that correctly classify sample (c) is greater for train samples than test samples (see
By an abuse of notation, for the model output of (in)correctly classifying models, that is, of (v(").
e (m) to refer to (in correctly classifying models and also superscript
5

---

Under review as a conference paper at ICLR 2022
0.45
Take doncies
Average Cordence 7a somplee
044
0.35
0.
0.0
0.95
0.6
0.7
60
0.5
0.15
0.1
0/25
20
900
200
100 100
Corneityy bla tout models
Gorples connuc classiled by rodel (AA
samplest
Traning agore
(a) Correct agreement distribution
(b) Distribution difference at each agreement level
(c) The effect of three factors on MI attack
The distribution of the number of times a sample is correctly classified by 10 models used in the ensemble. The models often make less classification mistake train samples than test samples. Center: By separating samples based on how many times they have been correctly classified, we can observe that the confidence output of these samples are negligible between train and test sets. Only when all samples are compared the distribution difference is significant and that is the direct effect of the third factor, namely the level of correct agreement. Right: The effect of the three factors on the MI attack. As the gap between the level of correct agreement of train and test widens (the blue lines), the MI attack on ensembles becomes more effective than a single model (green lines).
Figure 3: AlexNet model trained on CIFARIO. Left:
Appendix A.2 for more results). Since prediction confidence of correctly classified samples are higher than misclassified samples on average, i.e., y > y and e smaller for test samples, the ensemble confidence of test samples (yer) becomes lower than train samples. As a result, this factor can largely contribute to the effectiveness of membership inference attacks deep ensembles.
We note that the first two factors are not unique to ensembles and can be exploited by an attacker in a single model scenario well. As a result, these two factors have been extensively studied in Rezaei & Liu (2021) in a single model scenario across various image datasets and well-trained models. They have shown that for deep models the first factor (y/) is almost indistinguishable between train and test set and only the second factor (y"") is marginally distinguishable. However, this marginal difference does not have considerable impact on the different distribution shift in train and test sets.
On the other hand, the level of agreement has big impact on different distribution shifts. To better demonstrate this, we can analyze the distribution difference in each level of agreement separately. As shown in Figure 3(b), the average confidence of train and test samples are very close and indistinguishable when each level of agreement drawn separately. If the effect the first two factors were considerable, the two confidence values for each level of agreement would have been more distinguishable. Note that the average confidence between train and test sets is more distinguishable for the first two points in x-axis (where majority of models misclassify sample). However, these distributions only constitute a tiny portion of the training dataset, as shown with the first two blue bins in Figure 3(a). However, when all samples are combined, we vividly observe that the average confidence gap between train and test sets considerably widens, as shown in the last point in x-axis in Figure (b). This clearly demonstrates that the major factor in different distribution shift between train and test sets is the level of agreement (c).
Note that, unlike the first two factors, the third factor (c) only improves the effectiveness of member- ship inference attacks in ensemble scenario because it does not exist in single model. In other words, if a defence strategy eliminates the effect of the average level of correct agreement (i.e., it ensures that e is close between train and test samples), the membership inference attack is still possible on the ensemble, but only to the degree that is possible on single model?. As shown in Figure 3(c), as the gap between yf of train and test sets (red lines) and the gap between y" of train and test sets (brown lines) increases, the attack on both single model (non-ensemble) and also the ensemble
3 AIthhouhh this can be understood by analyzing the Eq. (1), it is difficult to demonstrate empirically. The reason is that these three factors are not independent, and, hence, our attempts to significantly change the third factor without changing the other two factors have been unsuccessful, See Appendix A.2 to see the dependency hetween these two factors across all datasets and models.

---

Under review as a conference paper at ICLR 2022
(EL-10) increases. However, only when the average level of correct agreement gap between train and test (blue lines) widens, the membership inference attack on ensembles becomes more effective than on non-ensembles.
Another important observation from Figure 3(c) is that the minimum level of agreement gap between train and test occurs when models relatively underfitted (i.e., the blue lines in first few epochs). This phenomena has also been partially observed in Fort et al. (2019) (Figure 2(c)). The main reason is that underfitted models often only learn the most common and generalizable features and, thus, they often agree on the features and predictions. As they move from underfitted region to overfitted region, their generalization gaps widen (blue lines in Figure 3(c)). As a result, they tend to correctly classify train samples more often than test samples. Consequently, they agree on train samples more than test samples and, hence, average gap of correct agreement between train and test set widens. Hence, the wider generalization gap of base learners is, the more effective membership inference attack would be on deep ensembles.
3.3 WHY DOES IT OUTPERFORM GAP ATTACK SIGNIFICANTLY?
Recently, several studies report simple baseline attack called gap attack (Choo et al., 2020), also known as naive attack (Rezaci & Liu, 2021; Leino & Fredrikson, 2020) that achieves similar performance as the confidence-based attacks in most scenarios. The gap attack predicts a sample as member if it is correctly classified by the target model, and as non-member otherwise. In other words, gap attack essentially reflects the generalization gap (Rezaei & Liu, 2021). In Rezaei & Liu (2021), authors extensively analyzed this phenomena in deep models and argued current MI attacks that barely outperform gap attacks are ineffective in practice because they only reflect the generalization gap and cannot infer the membership status of each individual sample accurately.
Figure 4 shows that the effectiveness of membership inference attacks increases and outperforms the gap attack deep ensembles are deployed. This raises significant privacy concern since the gap attack is often suggested as a baseline that is also hard to outperform in non-ensemble setting (Rezaei & Liu, 2021; Leino & Fredrikson, 2020; Choo et al., 2020). Note that gap attack can be viewed as a metric directly reflecting the generalization gap rather than a reliable membership inference. As suggested in Rezaei & Liu (2021), we can separate correctly classified samples and misclassified samples to understand why membership inference attacks can barely outperform gap attack. As shown in Figure 2(b) and (c), the distributions of train and test samples are similar when separated into correctly classified and misclassified sets. The reason why the distribution of all samples (Figure 2(a)) looks more distinguishable when correctly classified samples and misclassified samples are combined is that there are often more misclassified samples in the test set than the train set. This is the information that gap attack exploits which essentially reflects the generalization gap. In order for membership inference attack to considerably outperform the gap attack, the distribution of correctly classified samples and misclassified samples should leak membership status information, which is not often the case as it is shown in Figure 2(b) and (c), and Rezaci & Liu (2021).
When ensembling is used, the distribution of confidence values changes dramatically, as explained in Section 3. By comparing the confidence distribution of correctly classified samples in an ensemble (Figure 2(e)) with a non-ensemble scenario (Figure 2(b)), the distribution is clearly more distinguishable in ensemble case. This is of significant privacy concern because, as discussed in Rezaci & Liu (2021), majority of samples in practice belong to the correctly classified set. Similar trend is also observable in misclassified samples (Figure 2())) Hence, the confidence values, that barely leak more information than generalization gap itself in a single model scenario, now considerably leak more membership information than just the generalization gap. That is the reason why membership inference attacks are significantly more effective in deep ensembles in comparison to the gap attack.
4 EXPERIMENTS RESULTS
4.1 EXPERIMENTAL SETUP
We explore a wide range of datasets that are often used in deep ensemble literature or membership inference literature: Adult', Texas, Purchase (Shokri et al., 2017), MNIST (LeCun et al., 1998),
7

---

Under review as a conference paper at ICLR 2022
100
1p
1o
10
28
10
10
80
16
30-
Mi attack (black box0
attikakkkk AUC
Test sccuraty
Mi sitecs AUG jarnee box)
Adul NN
Purchase
Texas MNN
WNIST- -aahh
FANIST--LAA ahlet
CIFAR 10 Resteee
CIFARI Densotos
CIFARIO WWRushent
CI Alashat
CIFARTIOO RasNAio
CIFAROO Dww.-ooooooottttt
CIFARI WWw.ooo--eNNoooeeettt
Images Nett Resterro
Figure 4: Membership inference attack results across all datasets/modeels Each curve indicates an ensemble of (non-EL),, 2, 5, and 10 models from left right.
FMNIST (Xiao al., 2017), SVHN (Netzer et al., 2011), CIFARIO (Krizhevsky et al., 2009), CIFARIO0 (Krizhevsky et al., 2009), and ImageNet (Russakovsky et al., 2015). For non-image datasets (Adult, Purchase, and Texas), we use a fully connected neural network consisting of hidden layer of size 128 and a Softmax layer. All other training parameters for these datasets are suggested in Shokri al. (2017). For image datasets, we use a wide range of convolutional neural networks depending on the input dimension and the difficulty of the task. We use the model implementations adopted in Nasr al. (2019); Rezaci & Liu (2021)². We train 10 models for each dataset with random initialization and construct an ensemble of 2, 5 and 10 models, respectively.
For membership inference attack models, we consider two different threat models: black-box and white-box. In the black-box setting, the attacker only has access to the output of the ensemble, which the average over softmax output of individual models. In the white-box setting, the attacker can see the output of each model in ensemble. In the former case, the attack model takes the output of ensemble as input, while the latter case, it takes all confidence outputs of all models as input. Attack models are NNs with three hidden layers of size 128, 128, and 64, respectively. We consider a worst case scenario where 80% of the training dataset is given to the attacker and the goal is to infer the membership of the remaining samples, similar to Rezaci & Liu (2021). All other training parameters are set as suggested in Rezaei & Liu (2021). 1). All experiments are implemented in Python 3.5, PyTorch 1.6 and are conducted on server with Intel Xeon Processor 24-core 2.70GHz, 8 GeForce RTX 2080 GPUs and 384GB memory. See Appendix A.1 for full report. The results of the logit averaging, weighted averaging, and snapshot ensembles and diversified ensemble networks shown in Appendix A.3, Appendix A.4, and Appendix A.5, respectively.
Figure 4 shows the results on all datasets. For some datasets, such Adult, Texas, and MNIST, ensemble learning barely changes the accuracy or privacy. That is because the disagreement across models is insignificant in these datasets. For all other datasets, ensemble learning improves the accuracy as well as the effectiveness of membership inference attacks. As mentioned in Section 3.2, the most salient factor in membership inference effectiveness on deep ensembles is the accuracy gap between train and test set. Figure 4 clearly shows that whenever this generalization gap is large for non-ensemble case, the attack improvement is significant after ensembling. that the ensembling can often reduce the generalization gap and the effectiveness of the gap attack However, due to the
worth noting
(c.g., CIFARIO-DE Le-----eDeeet-lll 100, CIFAR AlcxNN, or SEFAR100-ResNae reasons explained in Section 3.3, the membership inference effectiveness still increases.
4.2 DEFENSE MECHANISMS
As discussed in previous sections, the main factor that causes more privacy leakage in ensemble learning the generalization gap of the base models. Therefore, any standard regularization technique can potentially work as a defense mechanism. In this paper, we study L.1 and I.2 regularization,
8

---

Under review as a conference paper at ICLR 2022
55
50 7-770
Privany Altack AUC)
45
35
None (epoch=|5.
30
Bagging
90)
(epoch-[15.60)
L2 11
25
DP SGD [t=115,2] 25
L1 (2-10.01, 0.04
MMD-Mixup
Stacking (Salem et al., 2018)
20
50
60
Test Accuracy (%)
Figure 5: Effect of defense mechanisms on an AlexNet model trained on CIFARIO. The size of each point indicates the relative value of its parameter, i.e., epoch, À and e.
DG-SGD and MMD+Mixup (Li et al., 2021). Furthermore, one can simply terminate training early to keep the model's 's weights in less overfitted region. Moreover, some ensembling techniques, such as bagging. partitioning, and stacking (Salem et al., 2018), limits the access of models all training samples, which can potentially reduce membership inference effectiveness on deep ensembles.
We train 10 AlexNet models with different initialization on CIFAR10 for each defense mechanism. Here, we consider black-box membership inference attack to measure privacy. Figure 5 shows the effect of defense mechanisms on ensemble learning. We can observe consistent trade-off between ensemble accuracy and privacy that resembles Pareto optimal points. Note that privacy degradation rate is clearly not constant. An ensemble of heavily regularized models or under-fitted models barely causes more privacy leakage (e.g., L2 regularization with X = 1). On the other hand, ensemble of overfitted models (e.g.. non-regularized models trained for 90 epochs) results large privacy leakage. It is worth mentioning that for a given accuracy, some approaches provide more privacy. For instance, the regular deep ensemble and the bagging of models trained for 30 epochs achieve similar accuracy. However, bagging is in fact more prone membership inference because each model in bagging is exposed to fewer training data samples and consequently is more overfitted.
5 CONCLUSION & DISCUSSION
In this paper, we investigate membership inference attacks in deep ensemble learning and demonstrate that there is trade-off between accuracy and privacy. We note some limitations our empirical analysis and opportunities for future work. First, privacy is multi-faccted concept and can be defined or quantified several ways. this work, we quantified privacy leakage in terms the effectiveness of membership inference attacks. Future work can quantify privacy in terms of other relevant attacks such as model inversion (Fredrikson et al., 2015; He et al., 2019), property inference (Ateniese et al., 2015; Ganju et al., 2018), and model stealing attack (Tramèr et al., 2016) as well formally-provable measures such as differential privacy (Dwork, 2008). Second, ensemble learning is an umbrella term covering variety of methods to combine multiple base learners. An arbitrary method of training and combining base learners can be construed as ensemble learning. We mainly focused to ensemble averaging because of its prevalent use for ensembling deep models. Given the privacy leakage of deep ensembles demonstrated here, future work can look into designing other fusion approaches to better navigate the trade-off between privacy and accuracy. One solution can focus on training models sequentially and applying some non-trivial criteria during training of each model to force the distribution correct agreement to be close for train and non-tran tna-rran samples. How to achieve this is not trivial and needs further research.
WW use Opacus implementation in PyTorch from http::/ggthhh.ccm/ppprrrooppppn

---

Under review as a conference paper at ICLR 2022
REFERENCES
Giuseppe Ateniese, Luigi Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, and Giovanni Felici. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers, International Journal of Security and Networks, 10(3):137-150, 2015.
Christopher Choquette Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. arXiv preprint PrXiiv2007.1432/1 2020.
Cynthia Dwork. Differential privacy: A survey of results. In International conference on theory and applications of models of computation, pp. 1-19. Springer. 2008.
Vitaly Feldman. Does learning require memorization? a short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 959, 2020.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint arXiv: 1et.::912200555 2019.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1322-1333, 2015.
Karan Ganju, Qi Wang, Wei Yang, Carl A Gunter, and Nikita Borisov. Property inference attacks on fully connected neural networks using permutation invariant representations. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communicutions Security, pp. 619-633, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Zecheng He, Tianwei Zhang, and Ruby Lee. Model inversion attacks against collaborative inference. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 148-162, 2019.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train get for free. arXiv preprint arXiv:1704.00100 2017.
Hongwei Huang, Weiqi Luo, Guoqiang Zeng, Jian Weng, Yue Zhang, and Anjia Yang. Damia: Leveraging domain adaptation as a defense against membership inference attacks. urXiv preprint arXiv:2005.08016, 2020.
Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong, Memguard: De- fending against black-box membership inference attacks via adversarial examples. Proceedings of the 2019 ACM SIGSAC Conference Computer and Communications Security, pp. 259-274, 2019.
Dan Kondratyuk, Mingxing Tan, Matthew Brown, and Boqing Gong. When ensembling smaller models is more efficient than single large models. arXiv preprint arXiv:2005.00570, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Ludmila I Kuncheva and Christopher J Whitaker. Measures of diversity in classifier ensembles and 2003.
their relationship with the ensemble accuracy. Machine learning, 51(2):
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 3 International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6405-6416, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-burt learning applied to document recognition. Proceedings of the IEEE, 86(18):3))) (1221))222222333 1998.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv: 15/1.663 14, 2015.
10

---

Under review as a conference paper at ICLR 2022
Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. In 29th {USENIX) Security Symposium (!USENIX) Security 20), pp. 1605-1622, 2020.
Jiacheng Li, Ninghui Li, and Bruno Ribeiro. Membership inference attacks and defenses in classifi- cation models. In Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy, pp. 5-16, 2021.
Zheng Li and Yang Zhang. Label-leaks: Membership inference attack with label. arXiv preprint arXiv:2007. 15528.. 2020.
Gaoyang Liu, Chen Wang, Kai Peng, Haojun Huang, Yutong Li, and Wenqing Cheng. Socinf: Mem- bership inference attacks on social media health data with machine learning. IEEE Transactions on Computational Social Systems, 6(5):907-921, 2019.
Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry Vetrov. On power laws in deep ensembles. arXiv preprint arXiv:2007.0844 2020.
Yunhui Long, Vincent Bindschaedler, and Carl A Gunter. Towards measuring membership privacy. arXiv preprint arXiv: 2017.
Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang. Haixu Tang, Carl A Gunter, and Kai Chen. Understanding membership inferences on well-generalized learning models. arXiv preprint 18X.v 048899 2018.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 634.646, 2018.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning. In 2019 ieee symposium on security and privacy, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 2011.
Shadi Rahimian, Tribhuvanesh Orekondy, and Mario Fritz. Sampling attacks: Amplification of membership inference attacks by repeated queries. arXiv preprint arXiv:2009.0033 2020.
Shahbaz Rezaei and Xin Liu. On the difficulty of membership inference attacks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1249) 2018.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246 2018.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18. IEFF, 2017.
Liwei Song, Reza Shokri, and Prateek Mittal. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 241-257, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
11

---

Under review as a conference paper at ICLR 2022
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In 25th {USENIX) Security Symposium ((USENIX) Security 16), pp. 601-618, 2016.
Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. Demystifying membership inference attacks in machine learning as service. IEEE Transactions on Services Computing, 2019.
Xiaofang Wang, Dan Kondratyuk, Kris M Kitani, Yair Movshovitz-Attaas and Elad Eban. Multiple networks are more efficient than one: Fast and accurate models via ensembles and cascades. arXiv preprint arXiv:2012.01985 2020.
Andrew Webb, Charles Reynolds, Wenlin Chen, Henry Reeve, Dan Iliescu, Mikel Lujan, and Gavin Brown. To ensemble not ensemble: When does end-to-end training fail? stat, 1050:6, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
Ziqi Yang. Bin Shao, Bohan Xuan, Fe-Chien Chang, and Fan Zhang. Defending model inversion and membership inference attacks via prediction purification. arXiv preprint arXiv:2005.039/1 2020.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pp. 268-282. IEEE, 2018.
Shaofeng Zhang, Meng Liu, and Junchi Yan. The diversified ensemble neural network. Advances in Neural Information Processing Systems, 33, 2020.
Yang Zou, Zhikun Zhang. Michael Backes, and Yang Zhang. Privacy analysis of deep learning in the wild: Membership inference attacks against transfer learning, arXiv preprint arXiv:2009.04877 2020.
A APPENDIX
A.1 FULL TRAINING REPORT
Figure demonstrates the improvement accuracy and MI attack over various training epochs. For datasets that ensembling outperforms single model, using ensemble of underfitted models is less prone to MI attack. However, it corresponds to lower accuracy. This is consistent with all experiments across different datasets and models, shown in Figure 6.
A.2 LEVEL OF CORRECT AGREEMENT
As discussed in section 3.2, overfitted models tend to disagree more on test samples than train samples. In other word, the distribution agreement for train and test sets becomes more distinguishable as models overfit. This distinction is more clear for datasets, such as CIFAR10 and CIFAR100, which shows most improvement when ensembling is used, as shown in Figure 7, 8, and 9, Furthermore, the actually improve prediction. If all models correctly
level of agreement can reveal if an ensemble classify a sample or all models misclassify a sample, ensembling fails to outperform a single model. This is the case for Adult, Texas, and MNIST, shown in Figure
A.3 AVERAGING LOGIT
In this section, we average logits (the output of a model before Softmax) of NN models instead of the confidences. We can still observe that ensembling leaks more membership status than non-ensemble scenario. However, the MI attacks with average confidence (Figure 6), in general, are slightly more effective than MI attacks with average logits (Figure 10). The reason is that that confidence values normalized and, hence, when aggregated all models have the same contribution to the overall confidence output of the ensemble. However, when logit is used, the confidence output of the ensemble is more influenced by highly activated neurons. These highly activated neurons, which
12

---

Under review as a conference paper at ICLR 2022
(a) Adult (NN)
(b) Texas (NN)
(c) MNIST (LeNet)
110
(d) FMNIST (LeNet)
SVHN (AlexNet)
(f) SVHN (ResNet20)
A CC
Turlig
CIFARIO (AlexNet)
(h) CIFARIO (ResNe(20)
ii CIFARIO (DenseNet100)
Trih
Te0 02-807
Tes S2C001
60
13
12
60 12
( CIFARIO (WResNet16-2)
(k) CIFARIO0 (AlexNet)
(1 CIFARIO0 (ResNet20)
CC
CleaGee
(m) CIFAR100 (0))
(n) CIFAR100 (WResNet16-2)
(0) ImageNet (ResNet50)
Figure 6: Target models' accuracy and MI attacks' AUC across all datasets and models.
13

---

Under review as a conference paper at ICLR 2022
07
04
62
33
34
34
32
(a) SVHN-AlexNet (epoch=60)
(b) SVHN-AlexN (epoch=60)
(c) SVHN-AlexNet
64-
32
33
Concll mestee Mo Scce Hamooes
Singer met conce desenee by modees Our roler suppoet
CIFARIO-AlexNet (epoch=75)
(c) CIFARIO-AlexNet (cpoch=75)
(f) CIeARIO-AlexNet
(g) CIFARIO-ResNe120 (epoch=25)
(h) CIFARIO-ResNec20 (epoch=25)
ii) CIFARIO-ReSNe1220
ausige
37
cleated
Single ba condel ta Lampees
(k)
CIFARIO 10FAR1O-BE DenscNet100
()) SEFAARO-DenneNettt
(7pach=170)
(epach=170)
(m)
CIFARIO WResNet16-2
CIFARIO-WResNerlle (8poch=80)
(n)
(o) HHEARIO-WResNetll..
(8poch=80)
Figure 7: The first row contains correct agreement distribution for SVHN and CIFAR1O datasets. The second row shows the average and standard deviation of distribution of samples based on the level of correct agreement. The third row shows the effect of the three factors on the MI attack.
14

---

Under review as a conference paper at ICLR 2022
(a) CIFAR100-AleXNet (epoch=90)
(b) CIFAR NOt (epochhh
(c)
2
Cuaniiy
(d)
1EAR100-RRsNet20
(c)
SPPAR100-ResNeree
(f)
1IFAR100-Resha
(epoch=25)
(cpoch=25)
(cpoch=25)
(g)
(h)
CIFAR1000DeenseNee1.00
(i)
CIFARI00-DenseNeer
(epoch=140)
(epoch=140)
Carady Sardyy-usssaa coae modes
Stap wwc.caaaaaaa.... cay salles waroee CAN MEAMPINN
(
KIFAR10--WReNNetle.
CIFARI00-WReSNetIRO (epoch=110)
(I)
(epoch=1 10)
63
(m)
ResNe50
(n) ResNet00 (epoch=40)
(0) GringeNet-ResNerteee
(cpoch=40)
Figure 8: The first row contains correct agreement distribution for CIFARION 100 and ImageNet datasets. The second row shows the average and standard deviation of distribution of samples based the level of correct agreement. The third row shows the effect of the three factors on the MI attack.
15

---

Under review as a conference paper at ICLR 2022
32
60
Quaelyy slaaleed Nteoiaes
Liggl Longlnnd
Adult-NN (epoch=25)
(b) Adult-NN (epoch=25)
(c) Adult-NN
34
(d) Purchase-NN (epoch=100)
(c) Purchase-NN (epoch=100)
(l) Purchase-NN
and Wysige Car faacc
Cunntiy deaderity
Texas-NN (cpoch=100)
(h) Texas-NN (epoch=100)
(i) Texas-NN
0.-
(i) MNIST-LeNet (epoch=25)
(k) M.MIST-LNNet (epach=25)
MNIST-LeNet
38
34
34
32
(m) MNIST-LeNet (epoch=55)
(n) MNIST-LeNet (epoch=55)
(0) MNIST-LeNet
Figure 9: The first row contains correct agreement distribution for Adult, Purchase, Texas, MNIST, and FMNIST datasets. The second row shows the average and standard deviation of distribution of samples based on the level of correct agreement, The third row shows the effect of the three factors on the MI attack.
16

---

Under review as a conference paper at ICLR 2022
Je pard
(a) FMNIST (LeNet)
(b) SVHN (AlexNet)
(c) SVHN (ResNet20)
C/b
(d) CIFARIO (AlexNet)
(e) CIFARIO (ResNel20)
(l) CIFARIO (DenseNet100)
Tea
MACT.
CIFARIO (WResNet16-2)
(h) CIFARIO0 (AlexNet)
(i CIFAR100 (ResNet20)
CCT CEESEEC
CAFEE
(i) CIFAR100 (DenseNet100)
(k) CIFAR100 (WResNet16-2)
ImageNet (ResNet50)
Figure 10: Target models' accuracy and MI attacks' AUC across all datasets and models. Here, MI attack model uses aggregated logits instead of aggregated confidences.
often belong to the correctly classifying models, has significantly more influence on the confidence output of ensemble in comparison with lightly activated neurons of misclassifying models. Hence, the confidence output is heavily influenced by only a portion of models in ensemble that have high activation neurons. In other words, can be seen as an ensemble of only portion of models, not all models in the ensemble. Since ensembling with fewer models leaks less membership status, logit averaging of n models leak membership status than confidence averaging of the same number of models. Note that logit averaging is still prone with the same degree to membership leakage in a white-box attack since a white-box attacker has access to all confidence values. Note that the consequence of using logit in certain applications, such confidence estimation, that requires reliable confidence estimation is out of the scope this paper. Moreover, one major drawback of using logit is that it can be arbitrary scaled (Wang al., 2020). However, in scenarios where only accuracy is concerned and white- box access is unavailable to the attacker, averaging logits seems to have a better privacy protection of training data than averaging confidences.
17

---

Under review as a conference paper at ICLR 2022
Teclng cood
CIFARIO (AlexNet)
(b) CIFARIO (ResNet20)
CIFARTOO (AlexNet)
Trrn om Cag 200.
(d) CIFARIO0 (ResNel20)
(e) SVHN (AlexNet)
(1 SVHN (ResNe120)
Figure 11: Target models' accuracy and MI attacks' AUC across all datasets and models using
weighted averaging ensemble networks.
A.4 WEIGIITED AVERAGING
In this section, we evaluate weighted averaging of deep models. We focused on image CIFARIO, CIFARI00, and SVHN datasets. We trained each model with random initialization and all hyper- parameters are similar to Section 4.1. Here, we use stochastic gradient descent (SGD) using the entire training set to learn the weight associated with each model. As shown in Figure 11, we observed similar cccuracy-privacy trade-off.
A.5 MORE ADVANCED ENSEMBLING APPROACHES
In this section, we evaluate two state-of-the- ensembling approaches, namely snapshot ensembles (Huang et al., 2017) and diversified ensemble networks (Zhang et al., 2020). For snapshot ensemble, we train several models on several datasets for 500 epochs and restart the cycle every 50 epochs, similar to the original paper (Huang et al., 2017). Note that the goal of our evaluation is show the accuracy-priin privacy trade-off, not to achieve the highest accuracy possible. Due to this reason and limited time had, we did not perform an exhaustive hyper- parameter tuning. Nevertheless, similar trade-off can be observed in Figure 12.
We also conduct the same experiment with diversified ensemble networks (Zhang al., 2020). The original paper used pre-trained VGG and ResNet models. We did not use pre-trained models for two reasons: 1) It make comparison with other approaches unfair, and 2) It may interfere with the membership inference analysis. We find that by using randomly initialized models to start training, La varies significantly and prevents the optimization converge. Therefore, we add weight the La term to reduce its effect on the entire loss. We use 0.01 for CIFARIO and SVHN and 0.001 for CIFARI00. For the shared layer, we use fully-connected layer of size 128 followed by batch normalization and Relu activation. We use SGD to train models for 60 epochs while dropping the learning at each 20 epoch by O.1. We could not achieve the exact same results as reported in the paper for two main reasons: 1) we did not use pre-trained models in the ensemble, and 2) many pyrer-parameters and implementation details are missed from the original paper. We could not find set of paramterrrettess and conditions to consistently achieve higher accuracy when increasing the number models. This was also reported in the original paper where they found that more models in the diversified ensemble do not always improve accuracy. One penitential reason is that training base models in diversified neural networks are not independent. These models are trained during the training of the entire network. This means when the number of models in the diversified neural
18

---

Under review as a conference paper at ICLR 2022
The 1967
(a) FMNIST (LeNet)
(b) SVHN (AlexNet)
SVHN (ResNer20)
ERCE
1
20
(d) CIFARIO (AlexNet)
(e) CIFARIO (ResNe120)
1 CIFARIO (DenseNet100)
MAT
M000 26
20
20
(g) CIFARIO (WResNet16-2)
(h) CIFARI00 (AlexNet)
ii CIFARI00 (ResNet20)
(i) CIFAR100 (DenseNet100)
(k) CIFAR100 (WResNet16-2)
ImageNet (ResNet50)
Figure 12: Target models' accuracy and MI attacks' AUC across all datasets and models using snapshot ensemble (Huang et al., 2017).
19

---

Under review as a conference paper at ICLR 2022
Thinin 19ch
CIFARIO (AlexNet)
(b) CIFARIO (ResNet20)
CIFARTOO (AlexNet)
Gag 22c
(d) CIFARIO0 (ResNel20)
(e) SVHN (AlexNet)
( SVHN (ResNe120)
Figure 13: Target models' accuracy and MI attacks' AUC across all datasets and models using
diversified ensemble networks (Zhang et al., 2020).
network is increased, there are significantly more parameters to train, but the number epochs are constant. So, it is expected that diversified neural network with less models to sometimes outperform diversified neural network with more models the number of training epoch is fixed. Nevertheless, as shown in Figure 13, in cases where accuracy increases, the membership inference attack effectiveness also increases.
20


Reviews:
[{"review_id": "i-v4iTO-Piz", "reviewer": "Reviewer_G7t1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper's main insight---that ensembles can be more vulnerable to membership inference---is interesting and a priori somewhat surprising.\nBut this paper nicely explains how ensembling strengthens privacy leakage: for training examples, all models tend to be correct and highly confident so the ensemble's prediction is essentially the same as any individual model. But for test examples, there is a fraction of examples where the ensemble disagrees. And these become easy to recognize as they have lower confidence.\n\nThe most interesting result in my opinion is in Figure 4, where the authors show that as overfitting *decreases* (and thus the trivial gap-attack becomes weaker), membership leakage actually *increases*. I think this result would merit to be discussed more prominently much earlier in the paper, as it runs contrary to some of the beliefs in the literature on membership inference attacks.\n\nWhile the observation in this paper is quite simple, it is analyzed in depth across a number of setups, datasets, architectures to form a convincing story.\nA few suggestions here:\n- Figure 1 (& Figure 5) uses a fairly \"bad\" model that achieves <80% accuracy on CIFAR-10. Do the results still hold if you switch to a better setup (e.g., a Wide-ResNet with standard data augmentation should easily reach >92% accuracy).\n- Figure 2 shows that \"something\" changes qualitatively with the distributions, but it is hard to convincingly say that one should lead to a stronger attack just via visualization. It would be useful here to also provide some quantitative measure of the distinguishability of these distributions (e.g., a simple TV distance).\n\nOther comments:\n- The introduction mixes two different forms of ensembling. It first talks about \"neural networks (NN) that are independently trained on the same dataset with different random initialization\" and then about how \"training each model on a different subset of data makes the ensemble less prone to overfitting\". This is a bit confusing, since it isn't clear at this point what type of ensembling will be considered in this paper. The note on the use of subsampled ensembles to defend against MI could maybe be relegated to the related work as it is fairly orthogonal to this paper.\n- The graphs in the paper are overall a little hard to read, as they tend to show too many things at once. The light green color used in some plots is also hard to see.", "summary_of_the_paper": "This paper describes and analyzes an interesting and curious phenomenon: a standard ensemble of models is typically more vulnerable to membership inference attacks, despite being less overfit than any single model.", "summary_of_the_review": "A fairly simple observation but nicely analyzed", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g2xmzyyaqbw", "reviewer": "Reviewer_YtVz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The division of confidence values by the number of ensemble learners who correctly classify a sample is a very useful way of distinguishing the reason for the distribution change -- it is not because these models have inherently lower confidence on test examples, but because they incorrectly answer them more often. These observations can be very interesting to the whole of MI literature -- because they suggest that all MI attacks in practice do as well as \"Gap Attack\" (when not considering an ensemble). However, this also makes me wonder if the authors used the strongest attack setting for attacking the ensemble models.\n2. The experiments are comprehensively evaluated across a wide variety of datasets, training settings, defense settings -- which helps to be confident about the generality of the results.\n3. The ensemble models appear to perform significantly worse in terms of privacy than their base model counterparts. The authors could emphasize the percentage increase in privacy risk.\n\n## Suggestions\n1. It would be nice to have some background about previously attributed causes of the success of ensembling techniques. Because from the language in the paper, it appears that the authors are the first to attribute that ensembling increases confidence on train samples, and not on the test set (on average)\n\n2. Since this paper is about \"deep ensemble models\", it appears incomplete without attention to ensemble-based defenses such as those highlighted by the authors -- like training the ensembles on different subsets of datasets ( which are not necessarily disjoint) -Salem et. al. Or in related work-- Huang et al. (2020); Li et al. (2021); Rahimian et al. (2020); Yang et al. (2020). I would appreciate if this paper was able to categorize scenarios in which ensembles are privacy-preserving, and when they are not.\n\n\n## Questions\n1. Do you have access to all n base models in the ensemble when this is black-box? Based on the graphs, there does not appear to be much difference in the white-box and black-box performance.\n2. \"We consider a worst-case scenario where 80% of the training dataset is given to the attacker and the goal is to infer the membership of the remaining samples, similar to Rezaei & Liu (2021)\" -- Can you explain what this setting is in more detail? \n3. Are all models trained on complete training sets? Or on smaller splits of the training dataset (as is done in many MI papers to get the attack performance high)\n\n\n## Writing\n1. Please use \\citep for parenthesized citations.\n2. For graphs in the appendix, what is the y-axis? It appears to be the reverse of what was followed in the main paper.\n4. Sometimes the language of the paper is hand-wavy and unsubstantiated. Such as:\n“As they move from under-fitted region to overfitted region, they start to memorize features more specific to train samples Feldman (2020). Consequently, the average gap of correct agreement between train and test set widens. Hence, the wider *the* generalization gap of base learners is, the more effective the membership inference attack would be on *the* ensemble. “\nThere is no experiment that talks about memorization per se. And membership inference must not be taken as a proxy for memorization to describe itself.\n5. “In the black-box setting, the attacker only access the output of the ensemble “ --> has access to\n\n## Post Rebuttal\nI am keeping my score unchanged. The following is my response to authors:\n\n---\nThank you for your detailed comments! The paper started with an interesting premise -- some prior work says ensemble methods help reduce MI threat, while there is evidence for ensemble hurting MI. Let us understand what's going on and set things straight. Unfortunately, at the end of the paper, the discussion and experimentation do not live up to this initial expectation. Apart from a small curve for partitioning, bagging in Figure 5, there is hardly any discussion on when and what types of ensemble methods help against MI.\n\nThe goal of requesting additional experiments was not to make you do additional experiments and fill up the Appendix, but to highlight how this discussion is very useful for the reader. I thank you for the efforts you put in with the experiments in the Appendix. Unfortunately, they do not see any discussion in the paper. For example, while the authors show how confidence varies with ensemble size -- what happens when this ensemble is not just model randomization but partitioning -- what if this was 10% sets, or 50% sets. What is the tradeoff now? The size of these experiments may seem large, but I am trying to point out from a reader's view -- what types of discussions in the paper (and conclusions) will help resolve the initial dilemma -- should I do ensemble or not? If i want to protect myself from MI.\n\nI am sorry that I am delayed in responding to you. But I believe that while this work is beneficial to the community, it only partially caters to the expectation it built. Re-prioritization and minor additional experiments can make this paper a delight for the reader!\n", "summary_of_the_paper": "This work analyzes the accuracy-privacy trade-off in ensemble learning by performing model inference attacks. The key finding of the paper is that the presence of an ensemble (that averages the predictions of individual learners) exacerbates the disparity between the confidence distribution of samples that were seen during training v/s those that weren't. They highlight how the main reason for this observation is the reduced agreement between base models for data points that were not seen during training. There is some evaluation of prior membership inference defenses in the ensemble setting.", "summary_of_the_review": "While the paper does provide some interesting results about the privacy threat of deep ensemble models, it lacks in the discussion of \"deep ensemble models\" as a whole -- which has been discussed as a defense in prior work. Further, some of the claims can be further substantiated as highlighted in my review. Finally, it would be good to clarify if the finding of the change in confidence distribution (for deep ensembles) of train and test samples has been observed in prior work, or is a new contribution in this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V432_5m08Q7", "reviewer": "Reviewer_H6K7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors in the paper perform empirical studies to investigate the trade-off between accuracy and privacy (measured by membership inference attacks) in deep ensembles. They find out that the level of correct agreement among models is the most dominant factor that improves the performance of MI attacks in deep ensembles. They support their claim by visualizing the distribution shifts of correct agreement in train/test examples. They further implement a variety of existing defenses, such as differential privacy and regularizations, etc., to investigate the ​​effects of existing defense mechanisms. Overall, the paper is well-written and the experiments are well conducted.\n\nMy biggest concerns on the paper are its limitations of novelties and contributions in the literature of membership inference attacks and privacy-preserving machine learning. The biggest finding of the paper is that the improved advantage of deep ensembles on MI attacks mainly comes from the level of correct agreement among models in the deep ensembles. The finding is purely empirical without theoretical justifications. Besides, it is not sure whether the similar conclusions extend to ensemble learning methods, which limits its contributions to the community. \n\nBesides, the no defense mechanism targeted for the finding is proposed and the defense methods evaluated in the experiments already existed. The privacy-utility trade-offs are expected and it seems the trade-offs are similar to the settings where no deep ensembles are adopted. \n", "summary_of_the_paper": "The authors in the paper perform empirical studies to investigate the trade-off between accuracy and privacy (measured by membership inference attacks) in deep ensembles.", "summary_of_the_review": "Although the paper is well-written and the experiments are well conducted, I think the novelties and contributions of the paper is not enough for acceptance: the authors just find out the level of correct agreement among models in the deep ensembles improves MI attacks in a particular deep ensemble setting, no algorithms nor theoretical insights targeted for the setting is provided. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E2m8f28g47j", "reviewer": "Reviewer_McDt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The idea of studying the accuracy-privacy trade off is interestrting and relatively new.\n2. The conclusion on the effectivess of the level of correct agreement among models is important and likely to motivate more related future work towards this direction.\nWeakness:\n1. The expriments and conclusions are all based on a very simple ensemble setting: averaging the individual predictions.  However in practice, we usually use some more advanced aggregating stratgies such as weighted averaging. In this case, it is not clear whether the conclusions in the paper will still be applicable.\n2. Instead of using underfitting ensembles, which usually yield less satisfactory accuracies,  to reduce the MI attack, the authors could also try some advanced ensemble methods. such as those that have enhanced diversities. Some good examples include:\nThe Diversified Ensemble Neural Network, Deep Negative Correlation Learning\n", "summary_of_the_paper": "This paper provide a systemantic analysis on the accuracy-privacy trade off for deep ensmebles. They show that the effectiveness of\nmembership inference attacks is likely to increase when ensembling improves accuracy. The authors further study  the impact of various factors such as prediction confidence and agreement between models that constitute the ensemble.", "summary_of_the_review": "Generally speaking, this is an interesting paper because the ``accuraacy and privacy\" trade off is under-resaearched. However, the experimental setting is relatively resrective. It is interesting to see whether the conclusion still holds for more advanced and generic ensemble settings.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors in the paper perform empirical studies to investigate the trade-off between accuracy and privacy (measured by membership inference attacks) in deep ensembles. They find out that the level of correct agreement among models is the most dominant factor that improves the performance of MI attacks in deep ensembles. They support their claim by visualizing the distribution shifts of correct agreement in train/test examples. They further implement a variety of existing defenses, such as differential privacy and regularizations, etc., to investigate the ​​effects of existing defense mechanisms. Overall, the paper is well-written and the experiments are well conducted.
While these findings are interesting, they do not reveal something useful or surprising about deep ensemble learning. It is not clear what the contribution is to the membership inference attack literature and private machine learning literature. they do not propose anything new to make the attacks stronger or defenses stronger.
Decision: Reject