Privacy‐preserving generative framework for images against membership inference attacks

IET CommunicationsVolume 17, Issue 1 p. 45-62 ORIGINAL RESEARCHOpen Access Privacy-preserving generative framework for images against membership inference attacks Ruikang Yang, Ruikang Yang orcid.org/0000-0002-7621-270X State Key Laboratory of Integrated Service Networks (ISN), Xidian University, Xi'an, China School of Cyber Engineering, Xidian University, Xi'an, China Contribution: Conceptualization, Data curation, ​Investigation, Methodology, Project administration, Resources, Software, Validation, Visualization, Writing - original draft, Writing - review & editingSearch for more papers by this authorJianfeng Ma, Jianfeng Ma State Key Laboratory of Integrated Service Networks (ISN), Xidian University, Xi'an, China School of Cyber Engineering, Xidian University, Xi'an, China Contribution: Conceptualization, Formal analysis, Funding acquisition, ​Investigation, Methodology, Supervision, Validation, Writing - original draft, Writing - review & editingSearch for more papers by this authorYinbin Miao, Yinbin Miao State Key Laboratory of Integrated Service Networks (ISN), Xidian University, Xi'an, China School of Cyber Engineering, Xidian University, Xi'an, China Contribution: Funding acquisition, Supervision, Writing - original draftSearch for more papers by this authorXindi Ma, Corresponding Author Xindi Ma [email protected] State Key Laboratory of Integrated Service Networks (ISN), Xidian University, Xi'an, China School of Cyber Engineering, Xidian University, Xi'an, China Correspondence Xindi Ma, State Key Laboratory of Integrated Service Networks (ISN), Xidian University, Xi'an, China. Email: [email protected] Contribution: Conceptualization, Formal analysis, Funding acquisition, ​Investigation, Methodology, Supervision, Validation, Writing - original draft, Writing - review & editingSearch for more papers by this author Ruikang Yang, Ruikang Yang orcid.org/0000-0002-7621-270X State Key Laboratory of Integrated Service Networks (ISN), Xidian University, Xi'an, China School of Cyber Engineering, Xidian University, Xi'an, China Contribution: Conceptualization, Data curation, ​Investigation, Methodology, Project administration, Resources, Software, Validation, Visualization, Writing - original draft, Writing - review & editingSearch for more papers by this authorJianfeng Ma, Jianfeng Ma State Key Laboratory of Integrated Service Networks (ISN), Xidian University, Xi'an, China School of Cyber Engineering, Xidian University, Xi'an, China Contribution: Conceptualization, Formal analysis, Funding acquisition, ​Investigation, Methodology, Supervision, Validation, Writing - original draft, Writing - review & editingSearch for more papers by this authorYinbin Miao, Yinbin Miao State Key Laboratory of Integrated Service Networks (ISN), Xidian University, Xi'an, China School of Cyber Engineering, Xidian University, Xi'an, China Contribution: Funding acquisition, Supervision, Writing - original draftSearch for more papers by this authorXindi Ma, Corresponding Author Xindi Ma [email protected] State Key Laboratory of Integrated Service Networks (ISN), Xidian University, Xi'an, China School of Cyber Engineering, Xidian University, Xi'an, China Correspondence Xindi Ma, State Key Laboratory of Integrated Service Networks (ISN), Xidian University, Xi'an, China. Email: [email protected] Contribution: Conceptualization, Formal analysis, Funding acquisition, ​Investigation, Methodology, Supervision, Validation, Writing - original draft, Writing - review & editingSearch for more papers by this author First published: 17 October 2022 https://doi.org/10.1049/cmu2.12507AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Machine learning has become an integral part of modern intelligent systems in all aspects of life. Membership inference attacks (MIAs), as the significant model attacks, also jeopardize the privacy of the intelligent systems. Previous works on defending MIAs concentrate on the model output perturbation or tampering with the training process. However, data and model reuse are common in intelligent systems, which results in the lack of scalability of previous defending works. This paper proposes a new privacy-preserving framework for images to transform source data into synthetic data to train models against MIAs. The synthetic data makes it easy to defend MIAs during data and model reuse to improve the scheme's scalability. The framework generates synthetic data satisfying differential privacy through the variational autoencoder model's information extraction and data generation capabilities to improve model accuracy. A noise addition mechanism with metric privacy for the latent code generated from source data is proposed, where noise is the product of Γ-distribution and unit hyper-sphere samples. Moreover, it is proved that the synthetic data also satisfies metric privacy. The experimental evaluations demonstrate that the framework reduces MIAs' attack accuracy to about 0.5 and maintains higher utility than DP-SGD under the same setting. 1 INTRODUCTION With the widespread use of artificial intelligence (AI) and machine learning (ML) in intelligent systems, the need to ensure privacy protection in these systems has come to the fore, with an overarching goal that the output and decision do not leak specific secrets of true individual attributes. The benefits of ML and AI can be shown in every part of the intelligent systems, including intelligent driving [1], innovative medical services [2], smart home systems [3], and intelligent traffic [4]. For example, machine learning used in the innovative medical services can accelerate the diagnostic process and provide personalized medical and healthcare service. For intelligent driving, it is possible to reduce the need for drivers and improve driving safety. Nowadays, ML has been provided as a service (MLaaS) by many platforms. Smart devices access these models from the internet through prediction application programming interfaces (APIs) that return a prediction score vector. At the same time, many companies are willing to deploy these models in intelligent clients locally to minimize real-time prediction serving latency. Meanwhile, several architectures have been proposed [5, 6] for distributed and federated learning to protect the clients' training data privacy. Each client trains a local model on their data and periodically exchanges model parameters or partially constructed models with the other clients. However, a series of studies [7-16] have indicated that the adversary could exploit the prediction output (black-box attack mode) or the ML models' parameters (white-box attack mode) to perform membership inference attacks to get membership information of the ML models. Membership inference attacks try to determine whether the record is used during the training of the target model. These attacks cause severe privacy and security threats in intelligent systems, mainly when the training dataset contains sensitive attributes like diagnosis and location information. For example, if a hacker could determine a patient data is used to train a cancer prediction model deployed in a wearable device, he is able to judge that the patient have cancer with great probability. As a consequence, this patient's privacy would be disclosed. In intelligent driving systems, the location privacy can be disclosed. A hacker can determine whether some images with landmarks are used to train the image classification model deployed in an intelligent vehicle. Thus, he could judge the driver's location with a significant probability. As a consequence, the vehicle's privacy would also be disclosed. Therefore, the membership privacy of ML models is significant important and we want to propose a privacy-preserving method to protect the membership privacy for ML models. In ML, the influence of a user on the trained model should be erased at request as the learned model contains private information about its training data, following the spirit of the European Union's General Data Protection Regulation (GDPR). Meanwhile, the main techniques to achieve the privacy preservation in ML are secure multi-party computing (MPC) and fully homomorphic encryption (FHE). However, the MPC scheme applied to protect the security of ML cannot effectively resist similar inference attacks because it does not protect the final result. And the FHE scheme is not widely used in practical application environments because of its complex calculations. Therefore, design a practical and effective mechanism to resist the membership inference attack is still an unsolved challenge in ML. Overfitting is the primary cause for the feasibility of membership inference attacks as the trained model tends to memorize training inputs and perform better on them. As a result, the members and non-members are distinguishable according to the prediction scores or the parameters' gradients in the target model. There are several defences [7-11, 13, 17, 18] have been proposed and most of them try to reduce overfitting by various regularization technique, such as L2 regularization [7], min–max game based adversarial regularization [11, 18], and dropout [9]. In addition, Song et al. [13] showed that early stopping outperformed other overfitting-prevention counter-measures against membership inference attacks to classifiers. However, none of them directly reduce the distinguishability of members or non-members and make the big utility loss. Some other mechanisms [19-25] like differentially private-stochastic gradient descent (DP-SGD) leverage differential privacy mechanism [26] when training the target model. In addition, some research like MemGuard [12] and purification framework [27] leverage both the specific differential privacy mechanism and ML constraint method to the prediction output of target model. All of these works utilize some mechanisms in the training stage or the output of the target model. However, data and model reuse are common in practice in intelligent systems. It is common for intelligent systems that periodic data streams feed into an intelligent big data analytic system. For example, the same images can simultaneously be used to train object detection models, object recognition models, and image classification models in the intelligent driving system. Each time we use these data to train a new model or apply these ML models, we adjust the mechanism's settings accordingly, and the model convergence faces challenges. Meanwhile, the noise mechanism applied to the output will also reduce the accuracy of the target model to a certain extent. Therefore, we consider that some transformation mechanisms could fundamentally apply to the input data to prevent membership inference attacks and improve the scheme's scalability without much utility loss. We summarize our motivation as follows. Motivation: the main basis for membership inference attacks is the model's sensitivity to the training data. The attacker can distinguish the difference in the representation of training and non-training data on the target model by constructing an attack classification model. However, the goal of model training is to learn the common target features of a set of data. The model may inadvertently acquire features specific to the training data because the model's training is hard to interpret. These unique features become an essential basis for differentiating the training data. Therefore, we consider generating new training data while preserving the common features of the original data as much as possible. These new training data are used to train the target model to resist the membership inference attacks. In this paper, we propose a privacy-preserving modular generative framework in intelligent systems to defend against membership inference attacks by transforming the source data into synthetic data to train the target model for improving the scheme's scalability. While some data generation mechanisms [28-31] satisfying differential privacy have been proposed, all of these works consider that they leverage differential noise mechanisms in the training stage, as previously stated. The biggest problem of these works is that they should consider the model convergence and privacy budget costs in each iteration, making the model hard to converge and extend. On the other hand, their purpose is to construct generative models that satisfy differential privacy to generate lots of new data. However, this paper focuses on protecting the target model from membership inference attacks by transforming the original data into similar data that satisfy differential privacy to improve the scheme's scalability without compromising much utility. Membership inference attacks, including black-box attacks and white-box attacks, are considered the measurement to reflect the privacy leakage of the trained model about its training set. The black-box attacks represent a scenario where the model is accessed through APIs. The white-box attacks describe a federal learning scene where shared model parameters cause model information leakage or the scenario where locally deployed models are compromised. We achieve the generative framework by generating synthetic data through a well-trained variational autoencoder (VAE) [32] from source data. The framework treats VAE as a building component to protect the downstream task model. But many other schemes [33-37] set the goal of protection as a generative model, which is the difference between our work among them. We treat the VAE model as a specific data generation solution because the VAE model creates a continuous and complete latent space locally and the new never seen data could be generated. The traditional generation model randomly generates hidden vectors in the hidden space. But we generate new data through the hidden vector generated from the original data, which can maintain the data's specific basic distribution characteristics. Thus, our mechanism can generate new data that is more in line with the target data distribution when facing the scene of partial data generation. Notice that we choose to generate new data rather than keep the latent vectors because of the portability of downstream training tasks. The contributions of this paper are summarized as follows. We propose a privacy-preserving modular generative framework to generate robust representation of the source data for many ML tasks in intelligent systems, and this framework can be further easily fine-tuned for many different downstream tasks such as image classification as shown in our work. We adopt VAE as a specific extractor and reconstructor in our framework, because of the stochasticity of the generative model and the continuity of the VAE model. This is different from the aforementioned others that set the protection target as the generative model. Moreover, we adopt the metric privacy mechanism in the latent space of VAE rather than the data domain for improving model accuracy. Note that we do not apply the differential privacy mechanism in the model during the training process or the model output like many papers mentioned above because of the scheme's scalability. We evaluate our system framework against the state-of-art membership inference attack on three real-world image datasets. Our results show that our project can protect data resources without much efficiency loss. Specifically, our framework stays a similar prediction accuracy for the target model compared to DP-SGD under the much lower privacy budget. In the remainder of this article, we firstly describe related works in Section 2 and preliminaries in Section 3. Then, we present the system model and threat model in Section 4. After that, Section 5 proposes our privacy-preserving generative framework and the related algorithms. Besides, experiments and results are discussed in Section 6. Conclusions of this article with future work are presented in Section 7. 2 RELATED WORK Membership inference attack The first paper about the membership inference attack was published by Shokri et al. [7] under the assumption of the black-box model. The shadow model was used to train an ML model similar to the target model, and the shadow model was used to learn the differences in confidence vectors between members and non-members. Later, Nasr et al. [8] first proposed the membership inference attack under the white-box model and federated learning model. They classified between members and non-members by distinguishing model data such as the gradient changes of each layer or the output of each layer. Then researchers focused on designing effective membership inference attacks under the limited adversary's advantages and finding membership inference threats under different ML application scenarios. In order to enhance the membership inference attack, The article [13] set different thresholds, which were used to determine whether the prediction confidence belongs to the member, for different class labels. At the same time, the redefined prediction entropy was used to strengthen the membership inference attack. There are also some membership inference attacks where the adversary's advantages are limited. Hui et al. [14] proposed a membership inference attack when the adversary could not collect enough samples with output probabilities and labels as either members or non-members, and Choquette et al. [16] introduced label-only membership attacks with the adversary only got access to models' predicted labels. When the defensive mechanism was applied to prevent adversarial samples, Song et al. [10] discussed the practical membership inference threats of ML under this situation. The defensive measures against adversarial samples were to generate a robust ML model. However, this robust ML model was more sensitive to membership inference attacks. The paper [38] designed new membership inference algorithms against ML models and achieved significantly higher inference accuracy when the augmented data was also used in training, and the augmented mechanism was known to the adversary. However, the malicious attacker could only utilize the original image or randomly chosen transformations which yielded a significantly lower inference success rate. In addition, there was a new membership inference attack [15] to recommender systems where the adversary could only observe the ordered recommended items. The existing membership inference attacks mainly relied on the confidence vectors or other model output to build a binary classification model to determine whether the data was directly used to train the model. From this point, we consider generating new training data to replace the raw data to train the target application model. Protection mechanism against membership inference attack Membership inference attack defence mechanisms have also developed for many years. As mentioned in the article [9], they used the dropout mechanism and model stacking mechanism to avoid over-fitting of the target model and prevent membership inference attacks. Song et al. [11] introduced adversarial training in the training process, which was abstracted into a maximum and minimum problem. Minimizes the classification loss and maximizes the profit of the inference attack to find the best defensive model against the attack model. Jia et al. [12] studied to fuzzy the confidence score vector to evade the membership classification of the attack model. Wang et al. [17] reduced model storage and computational operation to defend membership inference attacks. Yin et al. [18] set definitions for utility and privacy of target classifier, and formulated the design goal of the defence method as an optimization problem and solved it. There are also many works to utilize differential privacy to prevent membership inference attacks and we discuss them as follows. 2.2.1 Differential privacy For the schemes that apply differential privacy technology to mitigate membership inference attacks, some schemes [20, 21, 24, 25, 39] added noise to the objective function, while others [19, 22, 23] called DP-SGD added noise to the gradient during training. Jayaraman et al. [25] added noise in the objective function, gradient vectors, and output vectors to analyse trade-offs between privacy and utility. Nasr et al. [39] instantiated the hypothetical adversary to analyse the differential private ML especially DP-SGD. There are also some schemes [33-37] that set the goal of protection as a generative model. Hayes et al. [33] first applied the membership inference attack on the generative model and utilized differential privacy during the training stage to protect the generative model. Chen et al. [34] Systematically analysed the membership inference attack on the generative model and made a detailed classification. However, we treat the generative model VAE as a building component to protect the downstream task model. The existing defence schemes basically add noise in the model training process or directly confuse the output results of the model. It is rarely considered to protect the privacy of the source data by processing the raw data. Besides, data reuse and model reuse is common in practice. Each time the users utilize these data to train a new model or apply these ML models, they have to accordingly adjust the mechanism's settings. Meanwhile, they should also consider more about the model convergence and privacy budget costs in each iteration, making the model hard to converge and extend. Therefore, we consider that some transformation mechanisms could fundamentally apply to the input data to prevent membership inference attacks and improve the scheme's scalability. Data generation mechanism Some data generation mechanisms [28-31] satisfying differential privacy have been proposed. Xie et al. [28] added noise on the gradient of the Wasserstein distance with respect to the training data during the training procedure. The DP-CGAN [31] made the training process private by injecting random Gaussian noise into the optimization process of the discriminator network. DP-GAN [30] was built upon the improved WGAN framework and enforced DP by injecting random noise in updating the discriminator. Chen et al. [29] proposed gradient-sanitized Wasserstein GAN replacing the typical training procedure with the DP-SGD. All of these previous works considered that leverage differential noise mechanism in the training process as previously stated and the membership inference attacks were not included. However, we consider that generating synthetic data to train the target model for source data protection and do not affect the training process of the generative model. On the one hand, we do not consider applying the differential privacy mechanism during the training stage because of the model convergence, privacy budget costs in each iteration, and extensions for scheme. On the other hand, their purpose is to construct generative models satisfying differential privacy to generate lots of new data, while this paper focuses on protecting the target model from membership inference attacks by transforming the original data into similar data that satisfy differential privacy. 3 PRELIMINARIES This section reviews three concepts used in our work: the generative model, the differential privacy, and the membership inference attack. We briefly introduce the variational autoencoder as a transformation function and the metric privacy applied to satisfy the privacy guarantee. Generative model 3.1.1 Variational autoencoder [32] Variational autoencoder (VAE) is a prevailing generative framework that consists of an encoder and a decoder, which are cascaded to reconstruct data with pre-defined similarity metrics. The encoder maps data into a latent space, and the decoder maps the encoded latent code back to the data space. The VAE's objective function is composed of the reconstruction loss and the prior regularization over the latent code distribution. Formally, min θ , ϕ − E q ϕ ( z | x ) [ log ( p θ ( x | z ) ) ] + K L ( q ϕ ( z | x ) | | P z ) $$\begin{equation} \min \limits _{\theta ,\phi }-\,\mathbb {E}_{q_{\phi }(z|x)}[\text{log}(p_{\theta }(x|z))] + KL(q_{\phi }(z|x)||P_z) \end{equation}$$ (1) where z denote the latent code, x denotes the input data, q ϕ ( z | x ) $q_{\phi }(z|x)$ is the probabilistic encoder parameterized by ϕ which is used to approximate the true posterior, p θ ( x | z ) $p_{\theta }(x|z)$ represents the probabilistic decoder parameterized by θ, and K L ( · | | · ) $KL(\cdot ||\cdot )$ denotes the KL divergence. In practice, q ϕ ( z | x ) $q_{\phi }(z|x)$ is always constrained to be Gaussian distribution N ( μ , σ 2 ) $\mathcal {N}(\mu , \sigma ^2)$ and is sampled via the reparameterization trick, which results in a closed-form derivation of the second term. Differential privacy Differential privacy [26] is one of the most popular paradigms dealing with information leaks in statistical databases. It provides a formal privacy guarantee, ensuring that sensitive information relative to individuals cannot be easily inferred by disclosing answers to aggregate queries. A mechanism K is ε-differential private if for any two adjacent databases x, x ′ $x^{\prime }$ , and any property Z, the probability distributions K ( x ) $K(x)$ , K ( x ′ ) $K(x^{\prime })$ differ on Z at most by e ε $e^{\epsilon }$ , which means K ( x ) ( Z ) ≤ e ε K ( x ′ ) ( Z ) $K(x)(Z) \le e^{\epsilon }K(x^{\prime })(Z)$ . 3.2.1 Metric privacy While the standard differential privacy is a rigorous privacy notion, it is only applicable to publishing aggregate statistics. We concentrate on generating data rather than aggregating statistics about the data. Therefore, it requires a more general privacy notion for data that belongs to an arbitrary domain of secrets. The authors [40] extended the principle of differential privacy and proposed a generalized notion, metric privacy. Essentially, it defines a distance metric between secrets and guarantees a level of indistinguishability proportional to the distance. Specifically, given an arbitrary set of secrets X $\mathcal {X}$ with a metric d X $d_{\mathcal {X}}$ : Definition 1. (Generalized Privacy [[40]])A mechanism from X $\mathcal {X}$ to Z $\mathcal {Z}$ is a (probabilistic) function K: X $\mathcal {X}$ → P ( Z ) $P(\mathcal {Z})$ which satisfies d X $d_{\mathcal {X}}$ -privacy, iff ∀ x , x ′ ∈ X : d P ( K ( x ) , K ( x ′ ) ) ≤ d X ( x , x ′ ) $\forall x,x^{\prime }\in \mathcal {X}:d_{\mathcal {P}}(K(x),K(x^{\prime }))\le d_{\mathcal {X}}(x,x^{\prime })$ , or equivalently: K ( x ) ( Z ) ≤ e d X ( x , x ′ ) K ( x ′ ) ( Z ) ∀ Z ∈ F Z $$\begin{equation} K(x)(Z) \le e^{d_{\mathcal {X}}(x,x^{\prime })}K(x^{\prime })(Z) \qquad \forall Z \in \mathcal {F}_{\mathcal {Z}} \end{equation}$$ (2)where Z $\mathcal {Z}$ is a set of query outcomes, F Z $\mathcal {F}_{\mathcal {Z}}$ is a σ-algebra over Z $\mathcal {Z}$ , and P ( Z ) $P(\mathcal {Z})$ is the set of probability measures over Z $\mathcal {Z}$ . Metric privacy guarantees that the output of a mechanism should be roughly the same, i.e. bounded by the distance d X ( x , x ′ ) $d_{\mathcal {X}}(x,x^{\prime })$ , between two inputs x and x ′ $x^{\prime }$ . An adversary who observes the output space, it is challenging to infer the exact input, thus the privacy of the input is protected. The metric d X $d_{\mathcal {X}}$ can be extended by scaling a standard metric d by a factor ε, i.e. d X = ε · d $d_{\mathcal {X}}=\epsilon \cdot d$ . As a result, the guarantee of metric privacy also relies on ε: lower ε indicates higher indistinguishability, hence stronger privacy. Membership inference attack We formulate the membership inference attack as a binary classification task where the attacker aims to classify whether a sample is used to train the victim ML model. Formally, we define A : ( x , M ( θ ) ) → { 0 , 1 } $$\begin{equation} \mathcal {A}:(x,\mathcal {M}(\theta )) \rightarrow \lbrace 0,1\rbrace \end{equation}$$ (3)where the attack model A $\mathcal {A}$ output 1 if the attacker infers that the sample x is included in the training set, and 0 otherwise. θ denotes the target model parameters while M $\mathcal {M}$ represents the general model publishing mechanism, i.e. type of access available to the attacker. For example, M $\mathcal {M}$ is an identity function for the white-box access case and can be the inference function for the black-box case. This work mainly focuses on the attack during the target model inference stage after the training is complete. 4 SYSTEM MODEL AND THREAT MODEL In this section, we describe the system model of our framework and then elaborate on the threat model defined in our framework. System model Because the source data directly are used to train the target application model and the well-trained application model is vulnerable with existing membership inference attacks [7-16], we will give the definition of system model as follows: Definition 2. (System model)Assume a user U $\mathcal {U}$ has the source dataset as D an