# Membership Inference Defense in Distributed Federated Learning Based on Gradient Differential Privacy and Trust Domain Division Mechanisms

Authors: Zhenpeng Liu; Ruilin Li; Dewei Miao; Lele Ren; Yonggang Zhao
Journal: Security and Communication Networks
 (2022)

---

## Abstract

Distributed federated learning models are vulnerable to membership inference attacks (MIA) because they remember information about their training data. Through a comprehensive privacy analysis of distributed federated learning models, we design an attack model based on generative adversarial networks (GAN) and member inference attacks (MIA). Malicious participants (attackers) utilize the attack model to successfully reconstruct training sets of other regular participants without any negative impact on the global model. To solve this problem, we apply the differential privacy method to the training process of the model, which effectively reduces the accuracy of member inference attacks by clipping the gradient and adding noise to it. In addition, we manage the participants hierarchically through the method of trust domain division to alleviate the performance degradation of the model caused by differential privacy processing. Experimental results show that in distributed federated learning, our designed scheme can effectively defend against member inference attacks in white-box scenarios and maintain the usability of the global model, realizing an effective trade-off between privacy and usability.


---

## Body

## 1. Introduction

In recent years, several Internet service providers, such as Google and Amazon, have launched “machine learning as a service (MLaaS)” platforms to make it easier for individuals or organizations to build machine learning applications. This platform allows users to build data processing models that meet their needs and deploy the models to their services by uploading their datasets, creating classification tasks, and running data analyses. Since trained models are usually stored on cloud platforms and cloud providers only provide query APIs for users, the risk of leaking datasets and related information is high, which can easily lead to member inference attacks.Member inference attack was proposed by Shokri et al. [1] in 2017. Moreover, they completed the systematic design of the member inference attack in the black-box scenario. The attacker obtains membership relation information of a given sample by inferring whether the given sample exists in the training set of the target model. For example, for a diagnostic model constructed from AIDS patients, if a person’s medical data are inferred to be the model’s training data, the person may have AIDS. The attacker captures the difference between the prediction vectors of member and nonmember data to implement member inference attacks. Therefore, overfitting is usually the main cause of member inference attacks on distributed federated learning models.With the development of machine learning technology, the application scenarios of distributed federated learning [2] are more and more extensive. Traditional centralized deep learning requires users to share their data with cloud service providers, who train deep neural networks on these cloud platforms with high computing power. However, these cloud platforms are not entirely trustworthy (honest but curious, HbC). Once they are attacked to cause data leakage or data abuse, it will bring great risks to users’ personal property. However, in distributed federated learning, multiple participants collaboratively train the global model without sharing their own training set. They only need to exchange parameters through the parameter server (PS) to train a higher accuracy model, which is almost the same as the training accuracy of centralized learning.Nonetheless, if some malicious actors in distributed federated learning can reconstruct the training set of the global model, then the privacy of the model cannot be guaranteed. With the successful research of Melis et al. [3] on membership inference attack schemes for distributed learning scenarios, membership privacy leakage in such scenarios has been thrust into the limelight. Later, Nasr et al. [4] used the SGD algorithm to construct an attack scheme and successfully extracted membership information of data instances in a distributed scenario. In the process of distributed learning, each participant performs collaborative learning by using the same deep learning framework and sharing parameters. Although these parameters do not directly leak the user’s privacy, they just provide the necessary basis for member inference attacks.In this paper, we propose a defense scheme based on gradient differential privacy and trust domain [5] division to reduce the risk of privacy leakage caused by member inference attacks on the federated learning model. In general, our contributions are as follows:(i)
First, we construct an attack model based on generative adversarial networks (GANs) and membership inference attacks (MIA), and conduct the attack experiments to demonstrate that the distributed federated learning systems are vulnerable to membership inference attacks by successfully extracting data from the training set.(ii)
We propose a defensive countermeasure for an attack model based on gradient differential privacy that effectively protects the parameters shared by participants and reduces the accuracy of member inference attacks to random guessing.(iii)
We propose an optimization scheme based on the trust domain division to address the performance loss to the model caused by the above differential privacy scheme. By choosing appropriate parameters, this scheme optimizes the accuracy of model training close to centralized deep learning while keeping the privacy budget constant.The remainder of this paper is structured as follows: Section2 introduces the research status of privacy protection in distributed federated learning; Section 3 explains the attack model combining GAN and MIA in detail; Section 4 introduces the privacy protection method based on differential privacy and trust domain division; Section 5 introduces some preparations related to the experiment; Section 6 conducts the experiments and evaluates the performance; Section 7 concludes the paper.


## 2. Related Works

This section presents the research status of defense methods for membership inference attacks and privacy protection methods in distributed scenarios, respectively.


### 2.1. Research Status of Member Inference Defense

The participants of distributed federated learning can collaboratively train a global model with excellent usability and scalability without sharing data with each other. However, participants may be attackers during the training process, which increases the risk of the model being attacked by MIA.Existing methods for defending against membership inference attacks can be divided into the following two categories. The first kind of defense method is realized by confusing information. For example, Juuti et al. [6] proposed a method of rounding the result vector returned by the target model to erase the data after the decimal point; Jia et al. [7] proposed a method of adding noise to the prediction vector to confuse the distribution of the prediction vector between target members and target nonmembers. However, Choquette-Choo et al. [8] successfully distinguish the membership relationship of the data only through its label, which indicates that adding noise to the prediction vector is not reliable because of the risk of data leakage. Another type of defense method is achieved by capturing the query behavior of the attacker. When an attacker implements a membership inference attack, the target model will be queried multiple times while the regular participants will have far fewer requests. By capturing the behavioral characteristics of an inquirer, we can distinguish whether the inquirer is a normal user. Moreover, the feature distribution of the query samples constructed by the attacker is significantly different from the standard samples. For example, Kesarwani et al. [9] proposed to assess the risk of a model being attacked by recording a user’s query samples and examining their feature distribution. However, this method can only be applied during the process of attack, and the detection accuracy is not ideal. Besides, in the process of distributed federated learning, the attacker pretending to be a normal participant can statically and passively obtain the parameters of the global model, which makes it difficult for us to obtain its behavioral characteristics.


### 2.2. Research Status of Privacy Protection Methods in Distributed Federated Learning

Homomorphic encryption [10] and local differential privacy [11] have been proposed and applied in distributed data collection to reduce the risk of data leakage. For example, Phong et al. [12] applied homomorphic encryption to the process of model learning and constructed a more secure distributed federated learning system, alleviating the data leakage from incompletely trusted third-party servers. Although this method can prevent the attacker from directly obtaining the local data of other parties, the attacker can still obtain the distribution of user data and reconstruct it. In addition, differential privacy [13] is also applied to machine learning models [14] to defend against membership inference attacks. This method can prevent the model from remembering the details of a particular user, thus ensuring the user’s privacy. For example, Avella–Medina [15] proposed a differential privacy protection scheme for uploaded parameters, which is achieved by randomly selecting parameters and adding Laplacian noise to the selected parameters. However, the repeated addition of noise by each participant will inevitably increase the complexity of parameter exchange [16], which will increase the computational overhead of the participants and lead to decline in the accuracy of the model. In conclusion, distributed federated learning systems lack an effective method to balance privacy and availability when encountering member inference attacks.The trust domain division method based on identity authentication has been applied in many fields of deep learning. It can be achieved through trust negotiation, which provides a digital certificate to help build trust relationships among participants who are unfamiliar with each other. Unlike conventional privacy methods, trust domain division reduces the frequency of interaction between attackers and normal users through certain constraints. It does not require much privacy processing on data and parameters and then reduces privacy overhead. Therefore, the trust domain division method provides an idea for us to achieve an effective trade-off between privacy and usability of models.


## 2.1. Research Status of Member Inference Defense

The participants of distributed federated learning can collaboratively train a global model with excellent usability and scalability without sharing data with each other. However, participants may be attackers during the training process, which increases the risk of the model being attacked by MIA.Existing methods for defending against membership inference attacks can be divided into the following two categories. The first kind of defense method is realized by confusing information. For example, Juuti et al. [6] proposed a method of rounding the result vector returned by the target model to erase the data after the decimal point; Jia et al. [7] proposed a method of adding noise to the prediction vector to confuse the distribution of the prediction vector between target members and target nonmembers. However, Choquette-Choo et al. [8] successfully distinguish the membership relationship of the data only through its label, which indicates that adding noise to the prediction vector is not reliable because of the risk of data leakage. Another type of defense method is achieved by capturing the query behavior of the attacker. When an attacker implements a membership inference attack, the target model will be queried multiple times while the regular participants will have far fewer requests. By capturing the behavioral characteristics of an inquirer, we can distinguish whether the inquirer is a normal user. Moreover, the feature distribution of the query samples constructed by the attacker is significantly different from the standard samples. For example, Kesarwani et al. [9] proposed to assess the risk of a model being attacked by recording a user’s query samples and examining their feature distribution. However, this method can only be applied during the process of attack, and the detection accuracy is not ideal. Besides, in the process of distributed federated learning, the attacker pretending to be a normal participant can statically and passively obtain the parameters of the global model, which makes it difficult for us to obtain its behavioral characteristics.


## 2.2. Research Status of Privacy Protection Methods in Distributed Federated Learning

Homomorphic encryption [10] and local differential privacy [11] have been proposed and applied in distributed data collection to reduce the risk of data leakage. For example, Phong et al. [12] applied homomorphic encryption to the process of model learning and constructed a more secure distributed federated learning system, alleviating the data leakage from incompletely trusted third-party servers. Although this method can prevent the attacker from directly obtaining the local data of other parties, the attacker can still obtain the distribution of user data and reconstruct it. In addition, differential privacy [13] is also applied to machine learning models [14] to defend against membership inference attacks. This method can prevent the model from remembering the details of a particular user, thus ensuring the user’s privacy. For example, Avella–Medina [15] proposed a differential privacy protection scheme for uploaded parameters, which is achieved by randomly selecting parameters and adding Laplacian noise to the selected parameters. However, the repeated addition of noise by each participant will inevitably increase the complexity of parameter exchange [16], which will increase the computational overhead of the participants and lead to decline in the accuracy of the model. In conclusion, distributed federated learning systems lack an effective method to balance privacy and availability when encountering member inference attacks.The trust domain division method based on identity authentication has been applied in many fields of deep learning. It can be achieved through trust negotiation, which provides a digital certificate to help build trust relationships among participants who are unfamiliar with each other. Unlike conventional privacy methods, trust domain division reduces the frequency of interaction between attackers and normal users through certain constraints. It does not require much privacy processing on data and parameters and then reduces privacy overhead. Therefore, the trust domain division method provides an idea for us to achieve an effective trade-off between privacy and usability of models.


## 3. Attack Models Based on Generative Adversarial Networks and Membership Inference Attacks

To obtain private data from the training set of regular participants, the attacker pretends to be a regular participant in a distributed federated learning model. The attacker secretly observes the collaborative learning process and tricks other participants into publishing their local models to build the attack model. Like other regular participants, the attacker does not generate any contact with other participants and only updates the local parameters according to the parameter exchange protocol. However, this protocol can provide the structure of the deep learning model and the data labels of other participants to the attacker. The attacker uses these parameters as a basis to locally train a higher accuracy GAN network [17] that can reconstruct the training set of the distributed federated learning system. At the same time, these parameters also help the attacker train the member inference attack model locally to infer which participant each sample belongs to.


### 3.1. GAN-Based Data Generation Attack Module

The GAN-based data generation attack module is shown in Figure1. A represents the malicious participant (GAN-based attacker), Vrepresents the other regular participants (victims), PS represents the parameter server, D-Net represents the discriminator network, and G-Net represents the generator network. As a participant in distributed federated learning, attacker A first builds and trains a GAN-based attack model. Instead of completing the training of the GAN network alone, A guides the generator network and the global network for continuous adversarial training. Before training, the G-Net is initialized with random noise, and the global parameters are copied from the parameter server PS’s parameter pool to the local. Then, A starts building the discriminator network locally and trains it adversarial with the generator network. Through continuous adversarial training, the generator network gradually reconstructs the training sets of other regular participants to achieve the optimization goal shown in equation (1).Figure 1
GAN-based data generation attack module.In equation (1), xi represents the original data, zj represents the randomly generated data, and fxi;θD is the discriminator deep neural network. θD represents the parameters of the discriminator network, and gzj;θG is the generator deep neural network, in which outputs samples similar to the original data set. θG represents the parameters of the generator network. When the data generated by G-Net can pass the test of the D-Net with high accuracy, we can conclude that the generator network G-Net has learned the distribution of the local data of V, and the GAN network has been well trained.(1)min︸θGmax︸θD∑i=1n+logfxi;θD+∑j=1n−logf1−fgzj;θG;θD.The GAN-based data generation attack flow can be described as Algorithm 1.θD and θG represent the discriminator network parameters and the generator network parameters, respectively, where θD is updated by the global parameter θd downloaded from PS. Attacker A first builds a GAN model with the same structure as the global model locally. Then, D-Net and G-Net train each other, and the new discriminator parameters θD obtained from the training will be uploaded to the parameter server PS. This process is repeated until the G-Net can generate samples from the same distribution as the training set or reach a preset number of steps. As the adversarial training of the GAN model continues, attacker A can obtain data samples from all participantsV, thus providing sufficient training data for the member inference attack model.Algorithm 1: GAN attack model in federated learning system.
(1)
Model initialization(2)
for (epoch=1 to n) do(3)
Download global parametersθd from PS(4)
Update local discriminator parameterθD with θd(5)
If (user = =  A), then(6)
G-Net and D-Net conduct confrontation training on GAN(7)
Update parametersθG and θD(8)
UploadθD to PS(9)
Else if(10)
TrainCNN on local datasets(11)
UploadθD to PS(12)
end for(13)
return records generated byG-Net(14)
Output: local datasets for eachV


### 3.2. Member Inference Attack (MIA) Module

The data generated by GAN are used as the training sets to build the member inference attacks (MIA) model. The GAN model can only reconstruct the training data of the distributed learning model, while the MIA model can find the membership of the training data to help attackerA steal more private information.Most of the research on member inference attacks (MIA) is conducted in a black-box scenario, where attackerA cannot obtain the structure and parameters of the target model. However, MIA is carried out in the white-box scenario in the distributed federated learning system. A pretends to be a regular participant, then A can see the parameters of the global model, and get information about the model update instantly. Therefore, attacker A can train the MIA model in a supervised manner. We summarize the knowledge of the MIA model into three points:(1)
We can obtain the output resultfx,w of any sample x, where w represents the intermediate parameters of the model.(2)
For all regular participantsV, we can obtain the algorithm and architecture of their model.(3)
For all regular participantsV, we can obtain the computation results of their model intermediate layer.The process of training the MIA model is shown in Figure2. Assuming that participant D2 of the distributed federated learning model is attacker A. We take the training sets of the regular participants V reconstructed by the GAN network as the shadow training set D’, and divide D’ into a training set D’train and a test set D’test to train the shadow model [18] Vi. Attacker A can use the local model of the other participant V as the shadow model because the parameters of the target model can be easily downloaded from the parameter server PS. According to this, whether sample x belongs to the target model, the samples trained by the shadow model are labeled with “in” or “out” to construct the MIA model’s training data set D∗. D∗ consists of n instances x1∗,y1∗,x2∗,y2∗,…,xn∗,yn∗, all of them are, respectively, composed of the sample xi∗ and their output vector yi∗, where yi∗ records the characteristics of sample xi∗.Figure 2
Construction of the member inference attack model.In this case, the membership inference is modeled as a binary classification [19] task. First, the attacker A inputs a given sample x into the target model to obtain the prediction vector PV and then inputs the PV into the member inference attack (MIA) model to obtain the result “in” or “out,” which can infer whether sample x belongs to the target model’s training set.


## 3.1. GAN-Based Data Generation Attack Module

The GAN-based data generation attack module is shown in Figure1. A represents the malicious participant (GAN-based attacker), Vrepresents the other regular participants (victims), PS represents the parameter server, D-Net represents the discriminator network, and G-Net represents the generator network. As a participant in distributed federated learning, attacker A first builds and trains a GAN-based attack model. Instead of completing the training of the GAN network alone, A guides the generator network and the global network for continuous adversarial training. Before training, the G-Net is initialized with random noise, and the global parameters are copied from the parameter server PS’s parameter pool to the local. Then, A starts building the discriminator network locally and trains it adversarial with the generator network. Through continuous adversarial training, the generator network gradually reconstructs the training sets of other regular participants to achieve the optimization goal shown in equation (1).Figure 1
GAN-based data generation attack module.In equation (1), xi represents the original data, zj represents the randomly generated data, and fxi;θD is the discriminator deep neural network. θD represents the parameters of the discriminator network, and gzj;θG is the generator deep neural network, in which outputs samples similar to the original data set. θG represents the parameters of the generator network. When the data generated by G-Net can pass the test of the D-Net with high accuracy, we can conclude that the generator network G-Net has learned the distribution of the local data of V, and the GAN network has been well trained.(1)min︸θGmax︸θD∑i=1n+logfxi;θD+∑j=1n−logf1−fgzj;θG;θD.The GAN-based data generation attack flow can be described as Algorithm 1.θD and θG represent the discriminator network parameters and the generator network parameters, respectively, where θD is updated by the global parameter θd downloaded from PS. Attacker A first builds a GAN model with the same structure as the global model locally. Then, D-Net and G-Net train each other, and the new discriminator parameters θD obtained from the training will be uploaded to the parameter server PS. This process is repeated until the G-Net can generate samples from the same distribution as the training set or reach a preset number of steps. As the adversarial training of the GAN model continues, attacker A can obtain data samples from all participantsV, thus providing sufficient training data for the member inference attack model.Algorithm 1: GAN attack model in federated learning system.
(1)
Model initialization(2)
for (epoch=1 to n) do(3)
Download global parametersθd from PS(4)
Update local discriminator parameterθD with θd(5)
If (user = =  A), then(6)
G-Net and D-Net conduct confrontation training on GAN(7)
Update parametersθG and θD(8)
UploadθD to PS(9)
Else if(10)
TrainCNN on local datasets(11)
UploadθD to PS(12)
end for(13)
return records generated byG-Net(14)
Output: local datasets for eachV


## 3.2. Member Inference Attack (MIA) Module

The data generated by GAN are used as the training sets to build the member inference attacks (MIA) model. The GAN model can only reconstruct the training data of the distributed learning model, while the MIA model can find the membership of the training data to help attackerA steal more private information.Most of the research on member inference attacks (MIA) is conducted in a black-box scenario, where attackerA cannot obtain the structure and parameters of the target model. However, MIA is carried out in the white-box scenario in the distributed federated learning system. A pretends to be a regular participant, then A can see the parameters of the global model, and get information about the model update instantly. Therefore, attacker A can train the MIA model in a supervised manner. We summarize the knowledge of the MIA model into three points:(1)
We can obtain the output resultfx,w of any sample x, where w represents the intermediate parameters of the model.(2)
For all regular participantsV, we can obtain the algorithm and architecture of their model.(3)
For all regular participantsV, we can obtain the computation results of their model intermediate layer.The process of training the MIA model is shown in Figure2. Assuming that participant D2 of the distributed federated learning model is attacker A. We take the training sets of the regular participants V reconstructed by the GAN network as the shadow training set D’, and divide D’ into a training set D’train and a test set D’test to train the shadow model [18] Vi. Attacker A can use the local model of the other participant V as the shadow model because the parameters of the target model can be easily downloaded from the parameter server PS. According to this, whether sample x belongs to the target model, the samples trained by the shadow model are labeled with “in” or “out” to construct the MIA model’s training data set D∗. D∗ consists of n instances x1∗,y1∗,x2∗,y2∗,…,xn∗,yn∗, all of them are, respectively, composed of the sample xi∗ and their output vector yi∗, where yi∗ records the characteristics of sample xi∗.Figure 2
Construction of the member inference attack model.In this case, the membership inference is modeled as a binary classification [19] task. First, the attacker A inputs a given sample x into the target model to obtain the prediction vector PV and then inputs the PV into the member inference attack (MIA) model to obtain the result “in” or “out,” which can infer whether sample x belongs to the target model’s training set.


## 4. Member Inference Defense Mechanism Based on Differential Privacy and Trust Domain Division

For the member inference attack model in chapter 3, we propose a defense method based on differential privacy and trust domain division. Firstly, we implement differential privacy for the gradient information shared by the participants, which alleviates the overfitting of the model and reduces the risk of gradient parameter leakage. However, we later found that differential privacy will inevitably harm the model’s accuracy. To solve this problem, we apply trust domain division to distributed federated learning, which reduces the frequency of gradient parameter interaction through identity authentication. Therefore, the method effectively alleviates the decline in the model’s accuracy. Experiments verify the feasibility of our defensive schemes. The target model can achieve an effective trade-off between availability and privacy when the privacy parameters are reasonably selected.


### 4.1. Differential Privacy in the Training Process

Participants strive to train models with the same structure in the training process. However, the global model structure is usually complex, which causes the local model trained by the participant to remember certain training samples and increases the risk of being attacked. In order to keep the loss value of the model as low as possible, the SGD algorithm keeps changing the model’s parameters and gets the gradient loss of the training sets closer and closer to zero.This means that each training sample leaves a clear footprint on the gradient of the parameter loss function. Based on this (footprint), an attacker can use the SGD algorithm to extract information about the training set in the distributed federated learning process. By increasing the gradient of the target samplex, we can distinguish whether x is a member or a nonmember of the target model. If x is the training set of other participants, the target model will minimize its loss by gradient descent. Conversely, the gradient of the model will not change significantly. In other words, if a training data point leads to a significant loss, the SGD algorithm will influence some parameters to adapt themselves towards reducing the loss on this point. If the model does not see the data point during training, the changes in the gradient on this point are gradual throughout the training. This is what we exploit in membership inference attacks.In the distributed learning, each participant has direct access to its own local training set and adjusts its model parameters during training based on the global parameters calculated by the parameter server (PS). If differential privacy is used during training, a single sample will not affect the target model’s gradient, and the global model will meet the differential privacy. In the process of stochastic gradient update, noise can be added according to the idea of DP-SGD. Specifically, the following two steps need to be taken:(1)
By clipping theL-2 norm of the gradient gxi, the size of gxi is limited to a specific range, reducing the sensitivity of each gradient. As in equation (2), gxi2 is the L-2 norm of the gradient gxi, C is the threshold for gradient clipping. If the gradient is less than or equal to C, the gradient remains unchanged; otherwise, the gradient is reduced to C. It prevents a single training sample from having a significant effect on the model parameters. In addition, the gradient clipping method also reduces the overfitting of the model so that the model can still effectively defend against external member reasoning attacks after training.(2)g¯xi=gximax1,gxi2/C.(2)
The noise sampled randomly from the Gaussian distribution is added to the clipped gradientgxi, which introduces some randomness into the algorithm.(3)g˜=1L∑ig¯xi+N0,σ2C2.In equation (3), L represents the data volume and σ indicates the size of the noise. σ is proportional to the privacy performance, and its variance is proportional to the clipping gradient. Intuitively, adding random noise to the gradient can effectively reduce the risk of member inference attacks by capturing parameter updates. The algorithm with randomness can effectively confuse the behaviour of the training and nontraining set. At this time, we only need to consider how to select parameters in the model’s training process reasonably. We only need to focus on which group of parameters the model prefers to select in the distributed training process.By introducing differential privacy into the model’s training process through the above method, the risk of member inference attack on the target sample by using the active gradient rise can be reduced. In other words, if a single target sample cannot cause significant changes in the model’s gradient, the training set data of the model will be better protected, which indicates that the model does not remember the sample, and the trained model has better generalization ability.


### 4.2. Model Performance Optimization Based on Trust Domain Division

Empirically, protection schemes based on differential privacy implementation will inevitably harm model accuracy and learning efficiency. Therefore, we propose a trust domain mechanism [20] to improve the model accuracy of federated learning after satisfying differential privacy protection. We will specify how the trust domain division improves the model accuracy after satisfying differential privacy protection.The method of trust domain division draws on the trust-based cross-domain security authentication model in cryptography, which can be implemented by trust negotiation [21]. Trust negotiation provides a digital certificate to help establish relative trust among participants who are unfamiliar with each other. At the same time, this digital certificate is also the identity proof of the certificate holder. In practical scenarios, the attacker usually does not have insufficient real training sets and computing resources, which avoids the attacker obtaining a higher-level digital certificate when dividing the trust domain. This also means that the attacker loses the possibility to interact directly with participants with large datasets, thus reducing the risk of data leakage for regular participants. However, participants within a trusted domain trust each other, and participants within each trust domain still follow the distributed federated learning protocol when performing joint learning. The parameter server only uses differential privacy for training results obtained from the trust domain when sharing parameters among trust domains. This reduces the amount of leaked information when parameters interact among trust domains and further reduces the risk of data leakage to an attacker.Figure3 shows the application process of trust domain dividing in federated learning. First, according to the size of the local training set and the scale of computing resources owned by the participants, we divide the participants of distributed federated learning into n trust domains of different levels and establish federated learning within each trust domain. As the certificate authority (CA) [22] , parameter server (PS) is the certification authority of the trust domain. All communication and authentication among participants rely on certificates issued by the CA and interact with other participants that are not trusted. After completing the update to the model, the participants will upload the training results to the parameter server (PS). Later, PS only provides the training results to other participants in the same trust domain (with the same level of digital certificates) for download. While sharing the gradient parameter gxi among trust domains, PS clips gxi and adds random noise to it. After differential privacy processing, the gradient parameter gxi will be shared with other trust domains.Figure 3
Privacy-preserving model for joint learning based on trust domain division.


## 4.1. Differential Privacy in the Training Process

Participants strive to train models with the same structure in the training process. However, the global model structure is usually complex, which causes the local model trained by the participant to remember certain training samples and increases the risk of being attacked. In order to keep the loss value of the model as low as possible, the SGD algorithm keeps changing the model’s parameters and gets the gradient loss of the training sets closer and closer to zero.This means that each training sample leaves a clear footprint on the gradient of the parameter loss function. Based on this (footprint), an attacker can use the SGD algorithm to extract information about the training set in the distributed federated learning process. By increasing the gradient of the target samplex, we can distinguish whether x is a member or a nonmember of the target model. If x is the training set of other participants, the target model will minimize its loss by gradient descent. Conversely, the gradient of the model will not change significantly. In other words, if a training data point leads to a significant loss, the SGD algorithm will influence some parameters to adapt themselves towards reducing the loss on this point. If the model does not see the data point during training, the changes in the gradient on this point are gradual throughout the training. This is what we exploit in membership inference attacks.In the distributed learning, each participant has direct access to its own local training set and adjusts its model parameters during training based on the global parameters calculated by the parameter server (PS). If differential privacy is used during training, a single sample will not affect the target model’s gradient, and the global model will meet the differential privacy. In the process of stochastic gradient update, noise can be added according to the idea of DP-SGD. Specifically, the following two steps need to be taken:(1)
By clipping theL-2 norm of the gradient gxi, the size of gxi is limited to a specific range, reducing the sensitivity of each gradient. As in equation (2), gxi2 is the L-2 norm of the gradient gxi, C is the threshold for gradient clipping. If the gradient is less than or equal to C, the gradient remains unchanged; otherwise, the gradient is reduced to C. It prevents a single training sample from having a significant effect on the model parameters. In addition, the gradient clipping method also reduces the overfitting of the model so that the model can still effectively defend against external member reasoning attacks after training.(2)g¯xi=gximax1,gxi2/C.(2)
The noise sampled randomly from the Gaussian distribution is added to the clipped gradientgxi, which introduces some randomness into the algorithm.(3)g˜=1L∑ig¯xi+N0,σ2C2.In equation (3), L represents the data volume and σ indicates the size of the noise. σ is proportional to the privacy performance, and its variance is proportional to the clipping gradient. Intuitively, adding random noise to the gradient can effectively reduce the risk of member inference attacks by capturing parameter updates. The algorithm with randomness can effectively confuse the behaviour of the training and nontraining set. At this time, we only need to consider how to select parameters in the model’s training process reasonably. We only need to focus on which group of parameters the model prefers to select in the distributed training process.By introducing differential privacy into the model’s training process through the above method, the risk of member inference attack on the target sample by using the active gradient rise can be reduced. In other words, if a single target sample cannot cause significant changes in the model’s gradient, the training set data of the model will be better protected, which indicates that the model does not remember the sample, and the trained model has better generalization ability.


## 4.2. Model Performance Optimization Based on Trust Domain Division

Empirically, protection schemes based on differential privacy implementation will inevitably harm model accuracy and learning efficiency. Therefore, we propose a trust domain mechanism [20] to improve the model accuracy of federated learning after satisfying differential privacy protection. We will specify how the trust domain division improves the model accuracy after satisfying differential privacy protection.The method of trust domain division draws on the trust-based cross-domain security authentication model in cryptography, which can be implemented by trust negotiation [21]. Trust negotiation provides a digital certificate to help establish relative trust among participants who are unfamiliar with each other. At the same time, this digital certificate is also the identity proof of the certificate holder. In practical scenarios, the attacker usually does not have insufficient real training sets and computing resources, which avoids the attacker obtaining a higher-level digital certificate when dividing the trust domain. This also means that the attacker loses the possibility to interact directly with participants with large datasets, thus reducing the risk of data leakage for regular participants. However, participants within a trusted domain trust each other, and participants within each trust domain still follow the distributed federated learning protocol when performing joint learning. The parameter server only uses differential privacy for training results obtained from the trust domain when sharing parameters among trust domains. This reduces the amount of leaked information when parameters interact among trust domains and further reduces the risk of data leakage to an attacker.Figure3 shows the application process of trust domain dividing in federated learning. First, according to the size of the local training set and the scale of computing resources owned by the participants, we divide the participants of distributed federated learning into n trust domains of different levels and establish federated learning within each trust domain. As the certificate authority (CA) [22] , parameter server (PS) is the certification authority of the trust domain. All communication and authentication among participants rely on certificates issued by the CA and interact with other participants that are not trusted. After completing the update to the model, the participants will upload the training results to the parameter server (PS). Later, PS only provides the training results to other participants in the same trust domain (with the same level of digital certificates) for download. While sharing the gradient parameter gxi among trust domains, PS clips gxi and adds random noise to it. After differential privacy processing, the gradient parameter gxi will be shared with other trust domains.Figure 3
Privacy-preserving model for joint learning based on trust domain division.


## 5. Experimental Setup

### 5.1. Datasets

We selected three datasets to conduct the experiments on:CIFAR100, Adult, and Purchase100 datasets. In principle, the attack model in this paper can be extended to other types of data or samples. CIFAR100 consists of 6000 color images of 32 × 32 pixels, commonly used to evaluate image recognition algorithms [23]; Adult contains instances containing continuous values such as age and discrete values such as education and marital status, described by 6,000 binary features. It can be found in the UCI machine learning library [24]; Purchase100 contains thousands of shoppers’ consumption records. Each record is a binary record containing a user’s shopping history. These records are divided into 100 categories according to the similarity of shopping styles. The experiments will set up each participant to train separately using the number of instances, as shown in Table 1.Table 1
Size of data sets used to train and test target classification models and membership inference models.
DatasetsTarget modelInference attack modelTrainingTestTraining set membersTraining nonmembersTest membersTest nonmembersCIFAR10050,00010,00025,0005,0005,0005,000Adult8,00060,0004,0006,0005,0005,000Puchase10020,00050,00010,00010,00010,00010,000


### 5.2. Target Model

The federated learning system disclosed by Shokri [25] is shown in Figure 4, which serves as the target model. We set up n participants, and each participant has a certain number of training sets to train its local model. Before training, all participants agreed on the structure and learning objectives of the model.Figure 4
Distributed federal learning system.In a distributed federated learning system, the participantVi trains the local model according to the parameter exchange protocol [26], including the maintenance, update, and learning of the local vector wi. PS maintains and updates the global parameter vector wglobal in the same way. We set the structure of all participants consistent as convolutional neural network (CNN) [27] and use a selective stochastic gradient descent [28] algorithm at the fully connected layers. After training, each participant can independently adjust the model for testing locally.The local training algorithm of the distributed federated learning system can be described as Algorithm 2. The local parameters are initialized for each participant before participating in the first training. Then, the latest parameter componentθd is downloaded from the parameter server (PS) to replace the local parameters. Later, participants start training the CNN on the local training set. During the training process, the participant calculates the gradient vector Δwi and uploads the selected parameter Δwsi back to the parameter server PS. Intuitively, the participants will select and replace the parameter vector for uploading within a specific loss range, making the model train more accurate and converge faster.Algorithm 2: Local training algorithms for federal learning system
(1)
Input: global model parameterswglobal(2)
Model initialization and parameterwi initialization(3)
for (epoch  = 1 to n) do(4)
Download global parametersθd from PS(5)
TrainCNN on local datasets(6)
Update local vectorswi(7)
ComputeΔwi(8)
UploadΔwsi to PSend


### 5.3. Parameter Setting

AttackerA uses the same CNN network as other regular participants V. The front end of the CNN network includes a convolution layer and a pooling layer. The back end of the CNN is a fully connected layer, which uses a back-propagation algorithm. When the local model selects the gradient to upload to PS, the upload ratio θμ of the parameters is set to 0.01. When downloading parameters, the download scale of θd of global parameters is 1. For the purchase 100 datasets, we use a CNN model with 600 nodes in the input layer, 100 nodes in the output layer, and four layers (1024, 512, 256, and 128) in the hidden layer. For the adult datasets, we use the CNN network. The model structure is 1024, 512, 256, 128, and 100 (the output layer). We set the learning rate as α = 1 × 10−2 and the learning rate decay value as 1 × 10−7 to introduce more randomness into the learning process and reduce overfitting.We built a distributed federated learning model with 29 regular participants and one attacker. All 30 participants (including the attacker) were connected to the parameter server. When the model accuracy region is stable or reaches the termination condition, the model stops iterating and selects the model with the highest test accuracy as the target model. Each participantV uses the SGD optimizer to train the local datasets. Attacker A conducts a member inference attack in a white-box scenario.


### 5.4. Experimental Environment

All experiments in this paper were performed on an Intel Core i5-10400F 2.90 GHz CPU, 16 GB (3200 MHz) of hardware. We use the PyTorch geometry library [29] and the Python 3.9 environment for model development and training. In addition, we use the machine learning algorithm library Scikitlearn and the data analysis software Pandas in the interactive development environment Jupyter Notebook for testing usability and privacy.


### 5.5. Assessment Indicators

#### 5.5.1. Accuracy Rate

The accuracy rate is the number of samples correctly predicted by the model as a percentage of the total number of samples predicted. This metric can be used to evaluate the training effectiveness of distributed federated learning models and the accuracy of GAN-generated data in attack models.


#### 5.5.2. Attack Accuracy

There are two types of attacker output: “Member” and “Nonmember.” Attack accuracy is the fraction of correct membership predictions (predicting members as members and nonmembers as nonmembers) for an unknown data point. The size of the member and nonmember sample sets that we use to evaluate attack accuracy is the same.


## 5.1. Datasets

We selected three datasets to conduct the experiments on:CIFAR100, Adult, and Purchase100 datasets. In principle, the attack model in this paper can be extended to other types of data or samples. CIFAR100 consists of 6000 color images of 32 × 32 pixels, commonly used to evaluate image recognition algorithms [23]; Adult contains instances containing continuous values such as age and discrete values such as education and marital status, described by 6,000 binary features. It can be found in the UCI machine learning library [24]; Purchase100 contains thousands of shoppers’ consumption records. Each record is a binary record containing a user’s shopping history. These records are divided into 100 categories according to the similarity of shopping styles. The experiments will set up each participant to train separately using the number of instances, as shown in Table 1.Table 1
Size of data sets used to train and test target classification models and membership inference models.
DatasetsTarget modelInference attack modelTrainingTestTraining set membersTraining nonmembersTest membersTest nonmembersCIFAR10050,00010,00025,0005,0005,0005,000Adult8,00060,0004,0006,0005,0005,000Puchase10020,00050,00010,00010,00010,00010,000


## 5.2. Target Model

The federated learning system disclosed by Shokri [25] is shown in Figure 4, which serves as the target model. We set up n participants, and each participant has a certain number of training sets to train its local model. Before training, all participants agreed on the structure and learning objectives of the model.Figure 4
Distributed federal learning system.In a distributed federated learning system, the participantVi trains the local model according to the parameter exchange protocol [26], including the maintenance, update, and learning of the local vector wi. PS maintains and updates the global parameter vector wglobal in the same way. We set the structure of all participants consistent as convolutional neural network (CNN) [27] and use a selective stochastic gradient descent [28] algorithm at the fully connected layers. After training, each participant can independently adjust the model for testing locally.The local training algorithm of the distributed federated learning system can be described as Algorithm 2. The local parameters are initialized for each participant before participating in the first training. Then, the latest parameter componentθd is downloaded from the parameter server (PS) to replace the local parameters. Later, participants start training the CNN on the local training set. During the training process, the participant calculates the gradient vector Δwi and uploads the selected parameter Δwsi back to the parameter server PS. Intuitively, the participants will select and replace the parameter vector for uploading within a specific loss range, making the model train more accurate and converge faster.Algorithm 2: Local training algorithms for federal learning system
(1)
Input: global model parameterswglobal(2)
Model initialization and parameterwi initialization(3)
for (epoch  = 1 to n) do(4)
Download global parametersθd from PS(5)
TrainCNN on local datasets(6)
Update local vectorswi(7)
ComputeΔwi(8)
UploadΔwsi to PSend


## 5.3. Parameter Setting

AttackerA uses the same CNN network as other regular participants V. The front end of the CNN network includes a convolution layer and a pooling layer. The back end of the CNN is a fully connected layer, which uses a back-propagation algorithm. When the local model selects the gradient to upload to PS, the upload ratio θμ of the parameters is set to 0.01. When downloading parameters, the download scale of θd of global parameters is 1. For the purchase 100 datasets, we use a CNN model with 600 nodes in the input layer, 100 nodes in the output layer, and four layers (1024, 512, 256, and 128) in the hidden layer. For the adult datasets, we use the CNN network. The model structure is 1024, 512, 256, 128, and 100 (the output layer). We set the learning rate as α = 1 × 10−2 and the learning rate decay value as 1 × 10−7 to introduce more randomness into the learning process and reduce overfitting.We built a distributed federated learning model with 29 regular participants and one attacker. All 30 participants (including the attacker) were connected to the parameter server. When the model accuracy region is stable or reaches the termination condition, the model stops iterating and selects the model with the highest test accuracy as the target model. Each participantV uses the SGD optimizer to train the local datasets. Attacker A conducts a member inference attack in a white-box scenario.


## 5.4. Experimental Environment

All experiments in this paper were performed on an Intel Core i5-10400F 2.90 GHz CPU, 16 GB (3200 MHz) of hardware. We use the PyTorch geometry library [29] and the Python 3.9 environment for model development and training. In addition, we use the machine learning algorithm library Scikitlearn and the data analysis software Pandas in the interactive development environment Jupyter Notebook for testing usability and privacy.


## 5.5. Assessment Indicators

### 5.5.1. Accuracy Rate

The accuracy rate is the number of samples correctly predicted by the model as a percentage of the total number of samples predicted. This metric can be used to evaluate the training effectiveness of distributed federated learning models and the accuracy of GAN-generated data in attack models.


### 5.5.2. Attack Accuracy

There are two types of attacker output: “Member” and “Nonmember.” Attack accuracy is the fraction of correct membership predictions (predicting members as members and nonmembers as nonmembers) for an unknown data point. The size of the member and nonmember sample sets that we use to evaluate attack accuracy is the same.


## 5.5.1. Accuracy Rate

The accuracy rate is the number of samples correctly predicted by the model as a percentage of the total number of samples predicted. This metric can be used to evaluate the training effectiveness of distributed federated learning models and the accuracy of GAN-generated data in attack models.


## 5.5.2. Attack Accuracy

There are two types of attacker output: “Member” and “Nonmember.” Attack accuracy is the fraction of correct membership predictions (predicting members as members and nonmembers as nonmembers) for an unknown data point. The size of the member and nonmember sample sets that we use to evaluate attack accuracy is the same.


## 6. Experiments and Analysis

### 6.1. Experiments to Demonstrate Gradient Leakage

In the first experiment, we reveal how it helps to compromise privacy by comparing the gradient norm of the last layer on the target model between member instances and nonmember instances (training set samples and nontraining set samples). The rationale is that an attacker involved in the training process can actively influence the target model to extract more information about its training set, which is very applicable to inference attacks on federated learning.Figure5 compares the target model’s last layer of gradient parameters at different data points. As shown in Figure 5(a), the gradient parameters for the target members will be very similar to the gradient parameters for the nontarget member instances for each training period as the attacker moves up the gradient of the target instance. On the other hand, this is not true for the nonmember instances as shown in Figure 5(b).Figure 5
Gradient norms of different types of target instances under attack.
(a)(b)(c)Applying the gradient ascent algorithm to member instance will trigger the target model to try to minimize its loss by descending in the direction of the model’s gradient for those instances (and therefore nullifying the effect of the attacker’s ascent). However, for target nonmember instances, the model does not explicitly modify their gradients since they do not affect the training loss function. The attacker repeats the gradient ascent throughout each epoch of training, so ensuring the gradient of the model will keep increasing on such nonmember instances. Figure6(c) displays the difference between the gradient parameters of target members and target nonmember instances. The active gradient ascent attacker induces the target model to behave drastically differentiate between target member and nonmember instances, which makes the membership inference attack easier.Figure 6
The effect of learning ratea on model accuracy: (a) CIFAR100; (b) adult; (c) purchase100.
(a)(b)(c)


### 6.2. Attacks Experiments Based on GAN and MIA

#### 6.2.1. Feasibility Tests for Distributed Federated Learning in the Presence of Attack Models

First, a GAN model is constructed in a distributed federated learning environment with 29 regular participants and one attacker. Examine the precision of the distributed federated learning model and evaluate the precision of centralized and independent training as a comparison. The experimental outcomes are depicted in Table2. The average training accuracy for federated learning is 96.3%, which is somewhat lower than centralized training but higher than independent training. Even if there are malevolent participants (attackers), the training effect of the model will not be negatively impacted, as demonstrated by the experiment results.Table 2
Accuracy of the three model training methods.
Training methodThe training accuracy of the modelCIFAR100 (%)Adult (%)Puchase100 (%)Intensive training98.698.697.8Independent training93.594.190.5Federal learning training96.197.595.5


#### 6.2.2. Accuracy of GAN and MIA Using Various Training Epochs

We selected different training epochs and input the data samples generated by the GAN-based attack model into the completed training neural network for testing. After passing through the neural network, the classification accuracy of the samples was recorded and compared with the actual test samples. At the same time, we also recorded the accuracy changes of member inference attacks in corresponding cases.Table3 shows the accuracy of GAN and MIA when using various training epochs (CIFAR100 dataset with Alexnet). From the experimental results, we can see that using the later epochs can significantly improve their accuracy. Because the model will learn more unconventional data points and member information [30] at the later epochs, which improves the accuracy of member inference attacks. The accuracy of the samples generated by GAN tends to be stable around the 400th epochs, so we take the five training periods (100, 200, 300, 400, and 500) to test the performance of GAN, It can be seen from the experimental results that the average accuracy of GAN reaches 95.3%, which shows that the attack model can effectively restore the participants’ training sets and infer their membership relationship.Table 3
Accuracy of attackers using various training epochs in the federated setting.
Observed epochsAccuracyGenerator network (%)Member inference attack (%)10, 30, 50, 70, 9077.155.6100, 125, 150, 200, 25081.777.1100, 150, 200, 250, 30088.281.4100, 200, 300, 400, 50095.392.0


#### 6.2.3. Data Availability Assessment

To test the usability of the dataset generated by the GAN, we compared the accuracy of the data in two cases: the test set data (test_set) from three real datasets (CIFAR100, adult, purchase100), and the data generated by the GAN (gan_set). The data from these two cases were tested on the trained local model to observe their classification accuracy, and the experimental conditions were kept consistent except that the training sets were different. As shown in Figure7, blue represents three real datasets and orange represents the data generated by the GAN model constructed from the local model without any processing.Figure 7
The average accuracy of real test set vs. GAN-generated datasets.As can be seen in Figure7, the data generated by GAN are only close to the original data distribution, which results in a reduction in the accuracy of the generated data However when tested by the neural network, the average accuracy still reaches 90.8%, indicating that the reconstructed data based on the GAN can deceive the neural network, helping to build the training sets and carry out the member inference attacks with higher accuracy.


### 6.3. Experiments and Analysis of Member Inference Defense

In order to train the model with higher accuracy, the experiments first investigate the effect of learning rate on the accuracy of the model and then use the differential privacy to explore the effect of different levels of privacy settings on the training accuracy of the target model and the accuracy of the membership inference attack. After completing the above experiments, we divided the participants into different trust domains. The training accuracy of the model and the success rate of the member inference attack is reevaluated to verify the feasibility of the defense scheme.


#### 6.3.1. Effect of Learning Rate on Model Accuracy

Before applying differential privacy to the model, we first explored the impact of the learning rate on the accuracy of the model through experiments. We continue to use the experimental setup in chapter 5.3, setting the gradient upload ratioθu=0.1; the proportion of global parameters downloaded by the parameter server (PS) θd=1. Keeping the above parameters unchanged, we set the learning rate α of the model to two cases, which are α=1×10−2 and α=1×10−3. At the same time, we added centralized learning as the control group and obtained the experimental results as shown in Figure 6.From the experimental results, it can be seen that when the learning rateα is set to 1 × 10−2, distributed federal learning trained by the CIFAR100 datasets can achieve an accuracy level almost identical to the centralized learning, and the adult datasets also perform similarly. As can be seen in Figures 6(a)–6(c), as the learning rate decreases α=1×10−3, the accuracy of distributed learning decreases significantly because the lower learning rate reduces the amount of parameter information the model obtains in each epoch. In addition, as can be seen in Figure 6(c), when the learning rate of the model α=1×10−2, compared with centralized learning, the accuracy of the distributed federated learning model trained by the purchase 100 data set has decreased significantly. We continue to reduce the learning rate α and set α=1×10−3. It can be seen from Figure 6(c) that the model has not converged when the training rounds reach 100, which may lead to the unavailability of the model.Based on the above experimental results, we can conclude that although the gradient upload proportion is small, such asθu= 0.1, it will not have a negative impact on the accuracy of the model. A higher learning rate helps the model complete the training faster and obtain higher accuracy. Therefore, in the subsequent experiments, we will continue to keep the learning rate α=1×10−2.


#### 6.3.2. Privacy Performance Evaluation

We use the proposed GAN & MIA attack model to conduct attack experiments to test the feasibility of the defense scheme based on differential privacy. The privacy parametersϵ/c were set to 10, 0.1, and 0.01, and then the training accuracy of the distributed federated learning model and the attack accuracy of the attack model were examined in each of these three cases.As shown in Table4, when ϵ/c= 10, the degree of privacy protection is weak. The accuracy of the distributed federated learning model trained by the three datasets (purchase100, adult, and cifar100) can reach 92.7% on average, and the accuracy of member inference attacks can reach an amazing 80.7% on average, which means that the data privacy of the participants has been seriously compromised. Continue to strengthen the degree of privacy protection (smaller value). When ϵ/c= 1, the attack model is close to failure (the average accuracy of MIA is 58.9%), but a strong degree of privacy protection has some negative impact on the accuracy of the global model, which is consistent with common sense. We continue to strengthen privacy protection (ϵ/c= 0.1 and ϵ/c= 0.01). Although this will make the attack model completely invalid, it will also lead to the failure of effective interaction between participants, resulting in the failure of global model training.Table 4
Training accuracy and attack accuracy of federal learning models under different privacy settings. (purchased 100, adult, CIFAR 100).
Privacy parameterϵ/cPurchase 100AdultCIFAR 100Federated learning (%)Member inference (%)Federated learning (%)Member inference (%)Federated learning (%)Member inference (%)1098.391.189.274.590.576.5181.265.562.350.079.961.10.159.451.043.550.063.153.70.0152.550.117.250.055.050.0


#### 6.3.3. Defense Program Evaluation

The experimental results in 6.3.2 show that the model’s accuracy will be significantly reduced if it is directly protected by differential privacy, which is in line with our expectations. In the following experiments, we applied the trust domain division designed in chapter 3.2 to the training process of the global model to mitigate the loss of model accuracy. The participants in the same trust domain keep following the distributed federated learning protocol and carry out collaborative learning. In this case, the federated learning system meets thek-anonymity rule. Therefore, even if the participants in the trust domain still trust each other, the attacker cannot know which domain the obtained parameters belong to. Moreover, the attacker will be classified into a lower-level trust domain due to its insufficient computational resources and training set, which avoids the attacker from interacting with other participants directly and significantly reduces the risk of leaking data.In the following experiments, we divided the 30 participants into five trust domains and ten trust domains, respectively. We trained the model iteratively to steady under the same privacy-preserving strength to observe whether the division of trust domains improved the training accuracy while adding centralized deep learning and federated learning without trust domains (k = 30) as control groups.The experimental results are shown in Figure8, when keeping the privacy budget unchanged, dividing participants into a reasonable number of trust domains can help train models with higher accuracy while ensuring the model’s privacy.Figure 8
Curves of model training accuracy for different number of trust domains and privacy budgets: (a) privacy budgetϵ/c=1; (b) privacy budget ϵ/c=0.1; (c) privacy budget ϵ/c=0.01.
(a)(b)(c)Whenϵ/c=1, it can be seen from Figure 8(a) that the accuracy of the model training is comparable to the centralized learning when participants are divided into five trust domains (k = 5). However, the model’s accuracy will be reduced when the participants are divided into ten trust domains (k = 10). This is because dividing more trust domains requires more frequent parameter interactions, and these parameters are handled by differential privacy. We continue to increase our privacy budget. According to the experimental results in Table 4, we know that when ϵ/c=0.01, the average accuracy of member inference attacks is 50%, which has completely failed. However, it can be seen from the results in Figure 8(c) that directly training the model under such a privacy budget will seriously damage the accuracy of the model, making the model unusable. When ϵ/c=0.1, it can be seen from Figure 8(b) that dividing participants into five trust domains (k = 5) significantly improves the accuracy of the model compared to the scheme with the no-trust domain (k = 30). The average accuracy of the model can be kept at 92.5%, which makes the model meet the availability. At the same time, the average accuracy of member inference attacks is 51.6%, indicating that it can effectively defend against member inference attacks.In conclusion, we demonstrate that the member inference defense scheme proposed in this paper is feasible. We can achieve an effective trade-off between privacy and usability of the distributed federated learning model by choosing various parameters reasonably.


## 6.1. Experiments to Demonstrate Gradient Leakage

In the first experiment, we reveal how it helps to compromise privacy by comparing the gradient norm of the last layer on the target model between member instances and nonmember instances (training set samples and nontraining set samples). The rationale is that an attacker involved in the training process can actively influence the target model to extract more information about its training set, which is very applicable to inference attacks on federated learning.Figure5 compares the target model’s last layer of gradient parameters at different data points. As shown in Figure 5(a), the gradient parameters for the target members will be very similar to the gradient parameters for the nontarget member instances for each training period as the attacker moves up the gradient of the target instance. On the other hand, this is not true for the nonmember instances as shown in Figure 5(b).Figure 5
Gradient norms of different types of target instances under attack.
(a)(b)(c)Applying the gradient ascent algorithm to member instance will trigger the target model to try to minimize its loss by descending in the direction of the model’s gradient for those instances (and therefore nullifying the effect of the attacker’s ascent). However, for target nonmember instances, the model does not explicitly modify their gradients since they do not affect the training loss function. The attacker repeats the gradient ascent throughout each epoch of training, so ensuring the gradient of the model will keep increasing on such nonmember instances. Figure6(c) displays the difference between the gradient parameters of target members and target nonmember instances. The active gradient ascent attacker induces the target model to behave drastically differentiate between target member and nonmember instances, which makes the membership inference attack easier.Figure 6
The effect of learning ratea on model accuracy: (a) CIFAR100; (b) adult; (c) purchase100.
(a)(b)(c)


## 6.2. Attacks Experiments Based on GAN and MIA

### 6.2.1. Feasibility Tests for Distributed Federated Learning in the Presence of Attack Models

First, a GAN model is constructed in a distributed federated learning environment with 29 regular participants and one attacker. Examine the precision of the distributed federated learning model and evaluate the precision of centralized and independent training as a comparison. The experimental outcomes are depicted in Table2. The average training accuracy for federated learning is 96.3%, which is somewhat lower than centralized training but higher than independent training. Even if there are malevolent participants (attackers), the training effect of the model will not be negatively impacted, as demonstrated by the experiment results.Table 2
Accuracy of the three model training methods.
Training methodThe training accuracy of the modelCIFAR100 (%)Adult (%)Puchase100 (%)Intensive training98.698.697.8Independent training93.594.190.5Federal learning training96.197.595.5


### 6.2.2. Accuracy of GAN and MIA Using Various Training Epochs

We selected different training epochs and input the data samples generated by the GAN-based attack model into the completed training neural network for testing. After passing through the neural network, the classification accuracy of the samples was recorded and compared with the actual test samples. At the same time, we also recorded the accuracy changes of member inference attacks in corresponding cases.Table3 shows the accuracy of GAN and MIA when using various training epochs (CIFAR100 dataset with Alexnet). From the experimental results, we can see that using the later epochs can significantly improve their accuracy. Because the model will learn more unconventional data points and member information [30] at the later epochs, which improves the accuracy of member inference attacks. The accuracy of the samples generated by GAN tends to be stable around the 400th epochs, so we take the five training periods (100, 200, 300, 400, and 500) to test the performance of GAN, It can be seen from the experimental results that the average accuracy of GAN reaches 95.3%, which shows that the attack model can effectively restore the participants’ training sets and infer their membership relationship.Table 3
Accuracy of attackers using various training epochs in the federated setting.
Observed epochsAccuracyGenerator network (%)Member inference attack (%)10, 30, 50, 70, 9077.155.6100, 125, 150, 200, 25081.777.1100, 150, 200, 250, 30088.281.4100, 200, 300, 400, 50095.392.0


### 6.2.3. Data Availability Assessment

To test the usability of the dataset generated by the GAN, we compared the accuracy of the data in two cases: the test set data (test_set) from three real datasets (CIFAR100, adult, purchase100), and the data generated by the GAN (gan_set). The data from these two cases were tested on the trained local model to observe their classification accuracy, and the experimental conditions were kept consistent except that the training sets were different. As shown in Figure7, blue represents three real datasets and orange represents the data generated by the GAN model constructed from the local model without any processing.Figure 7
The average accuracy of real test set vs. GAN-generated datasets.As can be seen in Figure7, the data generated by GAN are only close to the original data distribution, which results in a reduction in the accuracy of the generated data However when tested by the neural network, the average accuracy still reaches 90.8%, indicating that the reconstructed data based on the GAN can deceive the neural network, helping to build the training sets and carry out the member inference attacks with higher accuracy.


## 6.2.1. Feasibility Tests for Distributed Federated Learning in the Presence of Attack Models

First, a GAN model is constructed in a distributed federated learning environment with 29 regular participants and one attacker. Examine the precision of the distributed federated learning model and evaluate the precision of centralized and independent training as a comparison. The experimental outcomes are depicted in Table2. The average training accuracy for federated learning is 96.3%, which is somewhat lower than centralized training but higher than independent training. Even if there are malevolent participants (attackers), the training effect of the model will not be negatively impacted, as demonstrated by the experiment results.Table 2
Accuracy of the three model training methods.
Training methodThe training accuracy of the modelCIFAR100 (%)Adult (%)Puchase100 (%)Intensive training98.698.697.8Independent training93.594.190.5Federal learning training96.197.595.5


## 6.2.2. Accuracy of GAN and MIA Using Various Training Epochs

We selected different training epochs and input the data samples generated by the GAN-based attack model into the completed training neural network for testing. After passing through the neural network, the classification accuracy of the samples was recorded and compared with the actual test samples. At the same time, we also recorded the accuracy changes of member inference attacks in corresponding cases.Table3 shows the accuracy of GAN and MIA when using various training epochs (CIFAR100 dataset with Alexnet). From the experimental results, we can see that using the later epochs can significantly improve their accuracy. Because the model will learn more unconventional data points and member information [30] at the later epochs, which improves the accuracy of member inference attacks. The accuracy of the samples generated by GAN tends to be stable around the 400th epochs, so we take the five training periods (100, 200, 300, 400, and 500) to test the performance of GAN, It can be seen from the experimental results that the average accuracy of GAN reaches 95.3%, which shows that the attack model can effectively restore the participants’ training sets and infer their membership relationship.Table 3
Accuracy of attackers using various training epochs in the federated setting.
Observed epochsAccuracyGenerator network (%)Member inference attack (%)10, 30, 50, 70, 9077.155.6100, 125, 150, 200, 25081.777.1100, 150, 200, 250, 30088.281.4100, 200, 300, 400, 50095.392.0


## 6.2.3. Data Availability Assessment

To test the usability of the dataset generated by the GAN, we compared the accuracy of the data in two cases: the test set data (test_set) from three real datasets (CIFAR100, adult, purchase100), and the data generated by the GAN (gan_set). The data from these two cases were tested on the trained local model to observe their classification accuracy, and the experimental conditions were kept consistent except that the training sets were different. As shown in Figure7, blue represents three real datasets and orange represents the data generated by the GAN model constructed from the local model without any processing.Figure 7
The average accuracy of real test set vs. GAN-generated datasets.As can be seen in Figure7, the data generated by GAN are only close to the original data distribution, which results in a reduction in the accuracy of the generated data However when tested by the neural network, the average accuracy still reaches 90.8%, indicating that the reconstructed data based on the GAN can deceive the neural network, helping to build the training sets and carry out the member inference attacks with higher accuracy.


## 6.3. Experiments and Analysis of Member Inference Defense

In order to train the model with higher accuracy, the experiments first investigate the effect of learning rate on the accuracy of the model and then use the differential privacy to explore the effect of different levels of privacy settings on the training accuracy of the target model and the accuracy of the membership inference attack. After completing the above experiments, we divided the participants into different trust domains. The training accuracy of the model and the success rate of the member inference attack is reevaluated to verify the feasibility of the defense scheme.


### 6.3.1. Effect of Learning Rate on Model Accuracy

Before applying differential privacy to the model, we first explored the impact of the learning rate on the accuracy of the model through experiments. We continue to use the experimental setup in chapter 5.3, setting the gradient upload ratioθu=0.1; the proportion of global parameters downloaded by the parameter server (PS) θd=1. Keeping the above parameters unchanged, we set the learning rate α of the model to two cases, which are α=1×10−2 and α=1×10−3. At the same time, we added centralized learning as the control group and obtained the experimental results as shown in Figure 6.From the experimental results, it can be seen that when the learning rateα is set to 1 × 10−2, distributed federal learning trained by the CIFAR100 datasets can achieve an accuracy level almost identical to the centralized learning, and the adult datasets also perform similarly. As can be seen in Figures 6(a)–6(c), as the learning rate decreases α=1×10−3, the accuracy of distributed learning decreases significantly because the lower learning rate reduces the amount of parameter information the model obtains in each epoch. In addition, as can be seen in Figure 6(c), when the learning rate of the model α=1×10−2, compared with centralized learning, the accuracy of the distributed federated learning model trained by the purchase 100 data set has decreased significantly. We continue to reduce the learning rate α and set α=1×10−3. It can be seen from Figure 6(c) that the model has not converged when the training rounds reach 100, which may lead to the unavailability of the model.Based on the above experimental results, we can conclude that although the gradient upload proportion is small, such asθu= 0.1, it will not have a negative impact on the accuracy of the model. A higher learning rate helps the model complete the training faster and obtain higher accuracy. Therefore, in the subsequent experiments, we will continue to keep the learning rate α=1×10−2.


### 6.3.2. Privacy Performance Evaluation

We use the proposed GAN & MIA attack model to conduct attack experiments to test the feasibility of the defense scheme based on differential privacy. The privacy parametersϵ/c were set to 10, 0.1, and 0.01, and then the training accuracy of the distributed federated learning model and the attack accuracy of the attack model were examined in each of these three cases.As shown in Table4, when ϵ/c= 10, the degree of privacy protection is weak. The accuracy of the distributed federated learning model trained by the three datasets (purchase100, adult, and cifar100) can reach 92.7% on average, and the accuracy of member inference attacks can reach an amazing 80.7% on average, which means that the data privacy of the participants has been seriously compromised. Continue to strengthen the degree of privacy protection (smaller value). When ϵ/c= 1, the attack model is close to failure (the average accuracy of MIA is 58.9%), but a strong degree of privacy protection has some negative impact on the accuracy of the global model, which is consistent with common sense. We continue to strengthen privacy protection (ϵ/c= 0.1 and ϵ/c= 0.01). Although this will make the attack model completely invalid, it will also lead to the failure of effective interaction between participants, resulting in the failure of global model training.Table 4
Training accuracy and attack accuracy of federal learning models under different privacy settings. (purchased 100, adult, CIFAR 100).
Privacy parameterϵ/cPurchase 100AdultCIFAR 100Federated learning (%)Member inference (%)Federated learning (%)Member inference (%)Federated learning (%)Member inference (%)1098.391.189.274.590.576.5181.265.562.350.079.961.10.159.451.043.550.063.153.70.0152.550.117.250.055.050.0


### 6.3.3. Defense Program Evaluation

The experimental results in 6.3.2 show that the model’s accuracy will be significantly reduced if it is directly protected by differential privacy, which is in line with our expectations. In the following experiments, we applied the trust domain division designed in chapter 3.2 to the training process of the global model to mitigate the loss of model accuracy. The participants in the same trust domain keep following the distributed federated learning protocol and carry out collaborative learning. In this case, the federated learning system meets thek-anonymity rule. Therefore, even if the participants in the trust domain still trust each other, the attacker cannot know which domain the obtained parameters belong to. Moreover, the attacker will be classified into a lower-level trust domain due to its insufficient computational resources and training set, which avoids the attacker from interacting with other participants directly and significantly reduces the risk of leaking data.In the following experiments, we divided the 30 participants into five trust domains and ten trust domains, respectively. We trained the model iteratively to steady under the same privacy-preserving strength to observe whether the division of trust domains improved the training accuracy while adding centralized deep learning and federated learning without trust domains (k = 30) as control groups.The experimental results are shown in Figure8, when keeping the privacy budget unchanged, dividing participants into a reasonable number of trust domains can help train models with higher accuracy while ensuring the model’s privacy.Figure 8
Curves of model training accuracy for different number of trust domains and privacy budgets: (a) privacy budgetϵ/c=1; (b) privacy budget ϵ/c=0.1; (c) privacy budget ϵ/c=0.01.
(a)(b)(c)Whenϵ/c=1, it can be seen from Figure 8(a) that the accuracy of the model training is comparable to the centralized learning when participants are divided into five trust domains (k = 5). However, the model’s accuracy will be reduced when the participants are divided into ten trust domains (k = 10). This is because dividing more trust domains requires more frequent parameter interactions, and these parameters are handled by differential privacy. We continue to increase our privacy budget. According to the experimental results in Table 4, we know that when ϵ/c=0.01, the average accuracy of member inference attacks is 50%, which has completely failed. However, it can be seen from the results in Figure 8(c) that directly training the model under such a privacy budget will seriously damage the accuracy of the model, making the model unusable. When ϵ/c=0.1, it can be seen from Figure 8(b) that dividing participants into five trust domains (k = 5) significantly improves the accuracy of the model compared to the scheme with the no-trust domain (k = 30). The average accuracy of the model can be kept at 92.5%, which makes the model meet the availability. At the same time, the average accuracy of member inference attacks is 51.6%, indicating that it can effectively defend against member inference attacks.In conclusion, we demonstrate that the member inference defense scheme proposed in this paper is feasible. We can achieve an effective trade-off between privacy and usability of the distributed federated learning model by choosing various parameters reasonably.


## 6.3.1. Effect of Learning Rate on Model Accuracy

Before applying differential privacy to the model, we first explored the impact of the learning rate on the accuracy of the model through experiments. We continue to use the experimental setup in chapter 5.3, setting the gradient upload ratioθu=0.1; the proportion of global parameters downloaded by the parameter server (PS) θd=1. Keeping the above parameters unchanged, we set the learning rate α of the model to two cases, which are α=1×10−2 and α=1×10−3. At the same time, we added centralized learning as the control group and obtained the experimental results as shown in Figure 6.From the experimental results, it can be seen that when the learning rateα is set to 1 × 10−2, distributed federal learning trained by the CIFAR100 datasets can achieve an accuracy level almost identical to the centralized learning, and the adult datasets also perform similarly. As can be seen in Figures 6(a)–6(c), as the learning rate decreases α=1×10−3, the accuracy of distributed learning decreases significantly because the lower learning rate reduces the amount of parameter information the model obtains in each epoch. In addition, as can be seen in Figure 6(c), when the learning rate of the model α=1×10−2, compared with centralized learning, the accuracy of the distributed federated learning model trained by the purchase 100 data set has decreased significantly. We continue to reduce the learning rate α and set α=1×10−3. It can be seen from Figure 6(c) that the model has not converged when the training rounds reach 100, which may lead to the unavailability of the model.Based on the above experimental results, we can conclude that although the gradient upload proportion is small, such asθu= 0.1, it will not have a negative impact on the accuracy of the model. A higher learning rate helps the model complete the training faster and obtain higher accuracy. Therefore, in the subsequent experiments, we will continue to keep the learning rate α=1×10−2.


## 6.3.2. Privacy Performance Evaluation

We use the proposed GAN & MIA attack model to conduct attack experiments to test the feasibility of the defense scheme based on differential privacy. The privacy parametersϵ/c were set to 10, 0.1, and 0.01, and then the training accuracy of the distributed federated learning model and the attack accuracy of the attack model were examined in each of these three cases.As shown in Table4, when ϵ/c= 10, the degree of privacy protection is weak. The accuracy of the distributed federated learning model trained by the three datasets (purchase100, adult, and cifar100) can reach 92.7% on average, and the accuracy of member inference attacks can reach an amazing 80.7% on average, which means that the data privacy of the participants has been seriously compromised. Continue to strengthen the degree of privacy protection (smaller value). When ϵ/c= 1, the attack model is close to failure (the average accuracy of MIA is 58.9%), but a strong degree of privacy protection has some negative impact on the accuracy of the global model, which is consistent with common sense. We continue to strengthen privacy protection (ϵ/c= 0.1 and ϵ/c= 0.01). Although this will make the attack model completely invalid, it will also lead to the failure of effective interaction between participants, resulting in the failure of global model training.Table 4
Training accuracy and attack accuracy of federal learning models under different privacy settings. (purchased 100, adult, CIFAR 100).
Privacy parameterϵ/cPurchase 100AdultCIFAR 100Federated learning (%)Member inference (%)Federated learning (%)Member inference (%)Federated learning (%)Member inference (%)1098.391.189.274.590.576.5181.265.562.350.079.961.10.159.451.043.550.063.153.70.0152.550.117.250.055.050.0


## 6.3.3. Defense Program Evaluation

The experimental results in 6.3.2 show that the model’s accuracy will be significantly reduced if it is directly protected by differential privacy, which is in line with our expectations. In the following experiments, we applied the trust domain division designed in chapter 3.2 to the training process of the global model to mitigate the loss of model accuracy. The participants in the same trust domain keep following the distributed federated learning protocol and carry out collaborative learning. In this case, the federated learning system meets thek-anonymity rule. Therefore, even if the participants in the trust domain still trust each other, the attacker cannot know which domain the obtained parameters belong to. Moreover, the attacker will be classified into a lower-level trust domain due to its insufficient computational resources and training set, which avoids the attacker from interacting with other participants directly and significantly reduces the risk of leaking data.In the following experiments, we divided the 30 participants into five trust domains and ten trust domains, respectively. We trained the model iteratively to steady under the same privacy-preserving strength to observe whether the division of trust domains improved the training accuracy while adding centralized deep learning and federated learning without trust domains (k = 30) as control groups.The experimental results are shown in Figure8, when keeping the privacy budget unchanged, dividing participants into a reasonable number of trust domains can help train models with higher accuracy while ensuring the model’s privacy.Figure 8
Curves of model training accuracy for different number of trust domains and privacy budgets: (a) privacy budgetϵ/c=1; (b) privacy budget ϵ/c=0.1; (c) privacy budget ϵ/c=0.01.
(a)(b)(c)Whenϵ/c=1, it can be seen from Figure 8(a) that the accuracy of the model training is comparable to the centralized learning when participants are divided into five trust domains (k = 5). However, the model’s accuracy will be reduced when the participants are divided into ten trust domains (k = 10). This is because dividing more trust domains requires more frequent parameter interactions, and these parameters are handled by differential privacy. We continue to increase our privacy budget. According to the experimental results in Table 4, we know that when ϵ/c=0.01, the average accuracy of member inference attacks is 50%, which has completely failed. However, it can be seen from the results in Figure 8(c) that directly training the model under such a privacy budget will seriously damage the accuracy of the model, making the model unusable. When ϵ/c=0.1, it can be seen from Figure 8(b) that dividing participants into five trust domains (k = 5) significantly improves the accuracy of the model compared to the scheme with the no-trust domain (k = 30). The average accuracy of the model can be kept at 92.5%, which makes the model meet the availability. At the same time, the average accuracy of member inference attacks is 51.6%, indicating that it can effectively defend against member inference attacks.In conclusion, we demonstrate that the member inference defense scheme proposed in this paper is feasible. We can achieve an effective trade-off between privacy and usability of the distributed federated learning model by choosing various parameters reasonably.


## 7. Conclusion

Distributed federated learning effectively ensures data security and protects local data privacy when sharing data. Because the participants carry out collaborative training, they will inevitably access each other’s gradient parameters and model parameters, which contain information about the training set of the local model. So, an attacker can extract local training sets by computing the gradient parameter values for multiple iterations.In this paper, we first construct an attack model based on generative adversarial networks and membership inference attacks. We successfully extract information from the training set of participants to demonstrate the distributed federated learning models are vulnerable to membership inference attacks. To solve this problem, we propose a method of using differential privacy to protect the model and prove the feasibility of our proposed method through experiments. However, at the same time, we find that differential privacy leads to the decline of model accuracy, so we propose a scheme to optimize the performance of the model by dividing multiple trust domains. The results show that dividing the trust domains can significantly improve the model’s accuracy under the same privacy budget setting.Next, we will try to use the designed attack model to extract the private data of participants with sparse local data to further test and improve the performance of the attack model. In addition, we will apply the proposed member inference defense method to more scenarios and test its robustness.



---

*Source: 1615476-2022-07-14.xml*