Accuracy-Privacy Trade-off in Deep Ensemble: A Membership Inference Perspective

1 INTRODUCTION . Ensemble learning has been shown to improve classification accuracy of neural networks in particular , and machine learning classifiers in general ( Kondratyuk et al. , 2020 ; Kuncheva & Whitaker , 2003 ; Sagi & Rokach , 2018 ) . The most commonly used approach for deep models involves averaging the output of multiple neural networks ( NN ) that are independently trained on the same dataset with different random initialization , called deep ensemble ( Lobacheva et al. , 2020 ) . Such a simple approach has been extensively used in practice to improve accuracy ( Lee et al. , 2015 ; Wang et al. , 2020 ) . Notably , a majority of the top performers in machine learning benchmarks , such as the ImageNet Large Scale Visual Recognition Challenge ( Russakovsky et al. , 2015 ) , have adopted some form of ensemble learning ( Lee et al. , 2015 ; Szegedy et al. , 2015 ; He et al. , 2016 ) . Other forms of ensemble learning ( different from deep ensembles ) , such as partitioning , has also been used to defend against privacy-harming membership inference ( MI ) attacks , where the goal of an attacker is to infer whether a sample has been used to train a model–i.e. , whether the sample belongs to the train set . Membership inference attacks generally use the prediction confidence of NN models to infer membership status of a sample ( Salem et al. , 2018 ; Shokri et al. , 2017 ; Truex et al. , 2019 ; Yeom et al. , 2018 ) by leveraging the insight that trained models may output higher prediction confidence on train samples than non-train samples ( Choo et al. , 2020 ) . The intuition behind using ensemble learning approaches , like partitioning , to defend against MI attacks is that training each model on a different subset of data makes the ensemble less prone to overfitting ( Salem et al. , 2018 ) . This idea of using ensemble learning to defend against MI attacks has since been discussed in the literature ( Huang et al. , 2020 ; Li et al. , 2021 ; Rahimian et al. , 2020 ; Yang et al. , 2020 ) . However , none of these papers theoretically or empirically demonstrate usefulness of ensemble learning as a defense mechanism . In this paper , we show that these two goals of ensemble learning , namely improving accuracy and defending against MI attack , do not trivially sum up in a unified solution . Figure 1 illustrates accuracy and privacy trade-off by plotting accuracy and membership inference attack effectiveness for ensembles comprising of varying number of base models ( 1 , 2 , 5 , and 10 ) that are trained for different numbers of epochs ( 5 , 45 , and 90 ) . We make two key observations here . First , there is an increase in both accuracy and MI attack effectiveness as we go from a single model to ensembles comprising of an increasing number of base models . The trade-off is more noticeable for more accurate models trained for a larger number of epochs . Second , we can adapt the design of ensembles to suitably navigate the trade-off between accuracy and privacy . Specifically , the two extreme cases are to : ( 1 ) maximize accuracy by using an ensemble of highly accurate models but at the cost of worse privacy1 ( purple arrow ) ; and ( 2 ) maximize privacy by intentionally using an ensemble of under-fitted models instead of a single model but at the cost of accuracy ( brown arrow ) . To understand the root cause of this trade-off , we show that using deep ensembles to improve accuracy exacerbates its susceptibility to membership inference attacks by making train and non-train samples more distinguishable . By analyzing the confidence averaging mechanism of deep ensembles , we investigate potential factors that enable membership inference . We show that the most influential factor is the level of correct agreement among models . Simply put , the number of models that correctly classify a train sample is often greater than the ones that correctly classify a test sample . This results in a wider confidence gap between train and non-train samples , when confidence values are averaged , enabling more effective membership inference attacks . We further show that the difference in the level of correct agreement between train and non-train samples is correlated with models ’ generalization gap . Hence , a natural question to ask is `` can deep ensembles that use less overfitted models mitigate privacy issues while achieving high accuracy ? '' . To answer this question , we study several regularization techniques , common membership inference defenses , and a few other ensembling approaches . We again observe a privacy-accuracy trade-off pattern similar to that shown in Figure 1 . Summary of contributions : In this paper , we perform a systematic empirical study of MI attacks on deep ensemble models . First , we show that when deep ensembles improve accuracy , it also leads to a different distribution shift in the prediction confidence of train and test samples , which in turn enables more effective membership inference . Second , we analyze various factors that potentially cause the prediction confidence of train and non-train samples to diverge . Among potential factors , we show that the most dominant factor is the level of correct agreement among models which indicates that more models in an ensemble agree on their prediction when a sample is a training sample . Hence , the aggregation of their prediction yields higher confidence output in comparison with non-train samples . We show that common defense mechanisms in membership inference literature , including differential privacy , MMD+Mixup , L1 and L2 regularization , as well as other ensemble training approaches , such as bagging , partitioning , and stacking ( Salem et al. , 2018 ) , can be used to mitigate 1Note that for complicated tasks , such as image classification , the common practice is to train deep models for a large number of epochs and avoid under-fitted models . That is because memorizing samples from long-tailed subpopulations are shown to be necessary to achieve close-to-optimal generalization error ( Feldman , 2020 ) . effectiveness of MI attacks but at the cost of accuracy . Although the main focus of the paper is on deep ensembles , we also cover bagging , partitioning , stacking ( Salem et al. , 2018 ) , logit averaging ( Appendix A.3 ) , weighted averaging ( Appendix A.4 ) , as well as more advanced and state-of-the-art ensembling techniques , such as snapshot ensembles ( Huang et al. , 2017 ) and diversified ensemble networks ( Zhang et al. , 2020 ) ( Appendix A.5 ) . We observe similar trade-off . 2 SYSTEM MODEL . 2.1 ENSEMBLE LEARNING . Background . In literature , ensemble learning refers to various approaches that combine multiple models to make a prediction . Models used to construct an ensemble are often called base learners . There are two main factors to construct an ensemble ( Sagi & Rokach , 2018 ) : 1 ) how base learners are trained to ensure diversity , such as random initialization , bagging , partitioning , etc. , and 2 ) how the output of base learners are fused to obtain the final output , including majority voting , confidence averaging , stacking , etc . Unlike ensemble of traditional machine learning algorithms , in a deep ensemble , the main source of diversity often comes only from random initialization of base learners ( Fort et al. , 2019 ) . In fact , other sources of diversity , such as bagging , have been shown to considerably degrade the overall accuracy of a deep ensemble ( Lee et al. , 2015 ; Lakshminarayanan et al. , 2017 ) . System Model . We mainly focus on the most widely used deep ensemble ( Kondratyuk et al. , 2020 ) unless otherwise specified . In this model , 1 ) base models are trained with random initialization on the same training dataset , and 2 ) their prediction confidence are fused through averaging . A less common approach is to average model logits which has been used in a few studies ( Webb et al. , 2020 ; Wang et al. , 2020 ) . See Appendix A.3 for experimental evaluation of logit averaging and A.4 for weighted averaging ensembles . We also evaluate two state-of-the-art deep ensembling approaches , namely snapshot ensemble and diversified ensemble network , in Appendix A.5 . Other general ensembling approaches , such as bagging , partitioning , and stacking ( Salem et al. , 2018 ) , are studied as defense mechanisms because they degrade accuracy but improve protection against MI attacks . 2.2 MEMBERSHIP INFERENCE . Background . Membership inference is a form of privacy leakage where the goal is to determine if a sample was used during the training of a target model . Samples used during training are often referred to as member or train samples , and other samples are referred to as non-member , non-train , or test samples . The first MI attack on neural networks was proposed in Shokri et al . ( 2017 ) where the attacker trains an attack classifier to predict the membership status . The attack classifier takes the prediction confidence of a target model as an input . Assuming that the attacker has access to a dataset with a similar distribution , she trains a set of shadow models to mimic the target model . Since the membership status of the data with which the shadow models are trained are known to the attacker , she can use the data to train the attack classifier . Many papers use the same idea with different variations or less restrictive assumptions ( Salem et al. , 2018 ; Liu et al. , 2019 ; Song et al. , 2019 ; Long et al. , 2017 ; Truex et al. , 2019 ; Long et al. , 2018 ; Yeom et al. , 2018 ; Rezaei & Liu , 2021 ; Zou et al. , 2020 ; Li & Zhang , 2020 ) . Most previous work built upon the idea of using prediction confidence to infer the membership status , except for Rezaei & Liu ( 2021 ) ; Choo et al . ( 2020 ) ; Rahimian et al . ( 2020 ) . In Rezaei & Liu ( 2021 ) , the authors assumed white-box access to the target model and launched a series of MI attack based on confidence values , distance to the decision boundary , gradient w.r.t model weight , and gradient w.r.t input . In Choo et al . ( 2020 ) , the authors proposed two attacks based on input transformation and distance to the boundary in a black-box setting . Similarly , in Rahimian et al . ( 2020 ) , the attacker randomly perturbs an input to obtain a set of random transformations of the input and uses the predicted labels to infer membership status . System Model . Since most existing attacks use confidence values , we first focus on changes of confidence values when using deep ensembles . We show that when using deep ensembles the distribution of confidence values becomes more distinguishable between train and non-train set in comparison with non-ensemble case . Consequently , any MI attack that relies on confidence values would be more effective on deep ensembles . Since our goal is to show a trade-off between accuracy and privacy , not to show which confidence-based attack can slightly outperforms another , we focus on a confidence-based attack proposed in Rezaei & Liu ( 2021 ) in both white-box and black-box settings . Here , white-box means the attacker has access to base-learners ’ output before aggregation , and black-box means the attacker has only access to the aggregated confidence output . Decision boundary-based attacks are extremely computational and query inefficient and it is not trivial how to adopt them for ensemble learning where essentially an input is copied n times and then fed to n models . Gradient-based approach of Rezaei & Liu ( 2021 ) also needs full knowledge of the entire deep ensemble . We consider adaptation of non-confidence-based attacks for deep ensembles as future work .

Summary: This paper provide a systemantic analysis on the accuracy-privacy trade off for deep ensmebles. They show that the effectiveness of membership inference attacks is likely to increase when ensembling improves accuracy. The authors further study  the impact of various factors such as prediction confidence and agreement between models that constitute the ensemble.