# SSGD: A Safe and Efficient Method of Gradient Descent

Authors: Jinhuan Duan; Xianxian Li; Shiqi Gao; Zili Zhong; Jinyan Wang
Journal: Security and Communication Networks
 (2021)

---

## Abstract

With the vigorous development of artificial intelligence technology, various engineering technology applications have been implemented one after another. The gradient descent method plays an important role in solving various optimization problems, due to its simple structure, good stability, and easy implementation. However, in multinode machine learning system, the gradients usually need to be shared, which will cause privacy leakage, because attackers can infer training data with the gradient information. In this paper, to prevent gradient leakage while keeping the accuracy of the model, we propose the super stochastic gradient descent approach to update parameters by concealing the modulus length of gradient vectors and converting it or them into a unit vector. Furthermore, we analyze the security of super stochastic gradient descent approach and demonstrate that our algorithm can defend against the attacks on the gradient. Experiment results show that our approach is obviously superior to prevalent gradient descent approaches in terms of accuracy, robustness, and adaptability to large-scale batches. Interestingly, our algorithm can also resist model poisoning attacks to a certain extent.


---

## Body

## 1. Introduction

Gradient descent (GD) is a technique to minimize an objective function, which is parameterized by the parameters of a model, by updating the parameters with the opposite direction of the gradient of the objective function about the parameters [1]. It has widely been applied in solving various optimization problems because of its simplicity and impressive generalization ability [2], but it is born with a heart of revealing privacy. Mathematically, the gradient is the parametric derivative of the loss function, which is explicitly calculated from the given training data and its true label. Therefore, the attacker may extract the sensitive information of the original training data from the captured gradients. Recently, researches have shown that the attacker, which captures the gradient of a training sample, can successfully infer its attributes [3], label [4], class representation [5, 6], or the data input itself [4, 7–9], with high accuracy. In the actual deep learning system, the gradient of multiple samples is widely used to improve efficiency and performance, which can also be viewed as the per-coordinate average of the single-sample gradients. Is multisample gradient safer for the privacy of training data? Unfortunately, Pan et al. [9] gave the theoretical analysis to indicate that multisample gradient still leaks samples and labels under certain circumstances. Since the work of Zhu et al. [7] was proposed, there is a branch of research [4, 7–9] to explore a violent but universal method for successful data reconstruction attacks, and some meaningful empirical results are given on CIFAR-10 and ImageNet. These works are based on the same learning-based framework. First, a batch of unknown training samples are used as variables, and then the optimal training samples are searched by minimizing the distance between the ground-truth gradient and the gradient calculated by the variables. The main difference between them is the choice of minimizing distance function. L2 and cosine distances are used in [4, 7, 8], respectively. Although Zhao et al. [4] used the properties of neural networks to recover the label of a single sample before the learning-based attack, this technique is only suitable to single-point gradient. It is the same as [7] in the multisample case. Pan et al. [9] gave a theoretical explanation for information leakage of single sample in a fully connected neural network with ReLu activation function. Furthermore, they used the internal information between neurons to show that in some cases there is sample and label leakage in multiple samples and extended the model to ResNet-18 [10], VGG-11 [11], DenseNet-121 [12], AlexNet [13], ShuffleNet v2-x0-5 [14], InceptionV3 [15], GoogLeNet [16], and MobileNet-V2 [17].To solve the gradient safety problem, Bonawitz et al. [18] designed a secure aggregation protocol, which is a four-round interactive protocol. Xu et al. proposed VerifyNet [19] and VeriFL [20] by adding verifiability to [18] for ensuring the correctness of aggregation. Bell et al. [21, 22] introduced a secure aggregation protocol with multilogarithmic communication and computational complexity, which reduces one round of interaction compared with [18]. Fereidooni et al. [23] showed that only two rounds of communication can be safely aggregated. All of the above works use encryption algorithms to encrypt the entire data set or intermediate values during the training process. Different from them, Ma et al. [24] used secure verifiable computing delegation to privately label a public data set from locally trained model aggregation and then utilized public data sets to train local models. Phong et al. [25] used homomorphic encryption technology to encrypt the gradient before sending it. Abadi et al. [26] employed differential privacy to protect gradients. Yadav et al. [27] applied differential privacy to federated machine learning by directly adding noise to the gradient. In PrivateDL [28], it is allowed to effectively transfer relational knowledge from sensitive data to public data in a way of privacy protection and enables participants to jointly learn local models based on public data with noise protection labels. However, these methods also have their limitations. The main problem of the secure aggregation protocol is communication overhead and computational efficiency. For differential privacy technology, it needs to consider the tradeoff between privacy and utility. More noise will lead to poor performance, and less noise will not be enough to protect the gradient. PrivateDL [28] requires a public data set and reduces the performance of the algorithm.Therefore, this paper proposes a new gradient descent method, super stochastic gradient descent (SSGD), for achieving neuron-level security while maintaining the accuracy of model. Moreover, SSGD has stronger robustness. Phong et al. [25] analyzed the leakage of single-sample single-neuron input data in the single-layer perceptron by using the sigmoid activation function. Pan et al. [9] used the ReLu activation function to analyze the sample data leakage from the multilayer fully connected neural network gradient and indicated that multiple samples also reveal privacy. There are two neurons in the last layer which are only activated by the same single sample. Essentially, the leakage is caused by attacking the single-sample gradient. SSGD converts the neuron gradient into a unit vector, which makes that the gradient aggregation of neurons has superrandomness. Superrandomness may significantly worsen the performance of the algorithm and make it difficult to converge. We select multiple-sample gradient composition updates to increase stability. At the same time, the superrandomness also brings strong robustness because the attacker cannot know the true gradient. SSGD invalidates these attacks on the gradient model, including the attack by searching for the optimal training sample [4, 7, 8] based on minimizing the distance between the ground-truth gradient and the gradient calculated by the variable, and the attack by solving the equation system [9] to obtain the training data. Our contributions are summarized as follows.(1)
We propose a gradient descent algorithm, called super stochastic gradient descent. The main idea is to update the parameters by using the unit gradient vector. In neural networks, neuron parameters are updated by using the unit gradient vector of neurons.(2)
We analyze theoretically that SSGD can realize neuron-level security and defend against attacks on the gradient.(3)
Experimental results show our approach has better accuracy and robustness than prevalent gradient descent approaches. And it can resist model poisoning attacks to a certain extent.The rest of this paper is organized as follows. In Section2, we review the basic gradient descent methods and the data leakage by gradients. In Section 3, we describe the super stochastic gradient descent and analyze the safety of our approach. The experimental results are shown in Section 4. Finally, we conclude this paper and give the further work.


## 2. Preliminaries

In this section, we review some basic gradient descent algorithms [1], including batch gradient descent (BGD), stochastic gradient descent (SGD), and mini-batch gradient descent (MBGD). The difference among them is that how much data is used to calculate the gradient of the objective function. Then, we describe the information leakage caused by gradients [19].


### 2.1. Basic Gradient Descent Algorithms

The BGD is an ordinary form of gradient descent, which takes the entire training samples into account to calculate the gradient of the cost functionℓθ about the parameters θ and then update the parameters by(1)θ=θ−η⋅∇θℓθ,where η is the learning rate and ∇θℓθ represents the gradient of function ℓθ with respect to the parameters θ. The BGD uses the entire training set in each iteration. Therefore, the update is proceeded in the right direction, and finally BGD is guaranteed to converge to the extreme point. On the contrary, the SGD considers a training sample xi and label yi randomly selected from the training set in each iteration to perform the update of parameters by(2)θ=θ−η⋅∇θℓθ;xi;yi.The BGD and SGD are two extremes: one uses all training samples and the other uses one sample for gradient descent. Naturally, their advantages and disadvantages are very prominent. For the training speed, the SGD is very fast, and the BGD cannot be satisfactory when the size of training sample set is large. For accuracy, the SGD determines the direction of the gradient with only one sample, resulting in a solution which may not be optimal. For the convergence rate, because the SGD considers one sample in each iteration and the gradient direction changes greatly, it cannot quickly converge to the local optimal solution.The MBGD is a compromise between BGD and SGD, which performs an update with a randomly sampled mini-batch ofN training samples by(3)θ=θ−η⋅∇θℓθ;xi;i+N;yi;i+N,where N is the number of batches. MBGD decreases the variance of the updates for parameter, so it has more stable convergence. Moreover, the computing of gradient about a mini-batch is very efficient by using highly optimized matrix optimizations that existed in advanced deep learning libraries.


### 2.2. Analysis of Gradient Information Leakage

Phong et al. [25] illustrated that how gradients leak the data information based on a single neuron shown in Figure 1. Assume that x^∈Rd represents data input with a label value y∈R. w∈Rd is the weight parameter and b∈R is the bias, represented uniformly by θ=w,b∈Rd+1. g∈Rd+1 is the gradient vector of the parameter θ, f is an activation function, and the loss function isℓfx^,w,b,y=hw,bx^−y2, where hw,bx^=f∑i=1dwix^i+b. Let g=σ1,…,σk,…,σd,σ and k∈1,…,d. We have(4)σk=∂ℓfx^,w,b,y∂wk=2hw,bx^−yf′∑i=1dwix^i+b⋅x^k,σ=∂ℓfx^,w,b,y∂b=2hw,bx^−yf′∑i=1dwix^i+b.Figure 1
Single neuron structure.Therefore, we obtainσk=σ⋅x^k. By solving the system of equations, we can easily get x^ and y. Also, we know that g is determined by x^,y. Therefore, g and x^,y are bijective. In distributed training, w and b usually are the parameters that need to be updated and known. Then, it can infer x^,y from g.Based on [9], the single-sample analysis of multilayer neural networks by using ReLu activation function, there is also data leakage problem. Although there is no such simple and intuitive leakage of data in a multilayer neural network, we can still know x^ and y by analyzing the internal relationship of the neural network and find that x^,y and g are still bijective.


## 2.1. Basic Gradient Descent Algorithms

The BGD is an ordinary form of gradient descent, which takes the entire training samples into account to calculate the gradient of the cost functionℓθ about the parameters θ and then update the parameters by(1)θ=θ−η⋅∇θℓθ,where η is the learning rate and ∇θℓθ represents the gradient of function ℓθ with respect to the parameters θ. The BGD uses the entire training set in each iteration. Therefore, the update is proceeded in the right direction, and finally BGD is guaranteed to converge to the extreme point. On the contrary, the SGD considers a training sample xi and label yi randomly selected from the training set in each iteration to perform the update of parameters by(2)θ=θ−η⋅∇θℓθ;xi;yi.The BGD and SGD are two extremes: one uses all training samples and the other uses one sample for gradient descent. Naturally, their advantages and disadvantages are very prominent. For the training speed, the SGD is very fast, and the BGD cannot be satisfactory when the size of training sample set is large. For accuracy, the SGD determines the direction of the gradient with only one sample, resulting in a solution which may not be optimal. For the convergence rate, because the SGD considers one sample in each iteration and the gradient direction changes greatly, it cannot quickly converge to the local optimal solution.The MBGD is a compromise between BGD and SGD, which performs an update with a randomly sampled mini-batch ofN training samples by(3)θ=θ−η⋅∇θℓθ;xi;i+N;yi;i+N,where N is the number of batches. MBGD decreases the variance of the updates for parameter, so it has more stable convergence. Moreover, the computing of gradient about a mini-batch is very efficient by using highly optimized matrix optimizations that existed in advanced deep learning libraries.


## 2.2. Analysis of Gradient Information Leakage

Phong et al. [25] illustrated that how gradients leak the data information based on a single neuron shown in Figure 1. Assume that x^∈Rd represents data input with a label value y∈R. w∈Rd is the weight parameter and b∈R is the bias, represented uniformly by θ=w,b∈Rd+1. g∈Rd+1 is the gradient vector of the parameter θ, f is an activation function, and the loss function isℓfx^,w,b,y=hw,bx^−y2, where hw,bx^=f∑i=1dwix^i+b. Let g=σ1,…,σk,…,σd,σ and k∈1,…,d. We have(4)σk=∂ℓfx^,w,b,y∂wk=2hw,bx^−yf′∑i=1dwix^i+b⋅x^k,σ=∂ℓfx^,w,b,y∂b=2hw,bx^−yf′∑i=1dwix^i+b.Figure 1
Single neuron structure.Therefore, we obtainσk=σ⋅x^k. By solving the system of equations, we can easily get x^ and y. Also, we know that g is determined by x^,y. Therefore, g and x^,y are bijective. In distributed training, w and b usually are the parameters that need to be updated and known. Then, it can infer x^,y from g.Based on [9], the single-sample analysis of multilayer neural networks by using ReLu activation function, there is also data leakage problem. Although there is no such simple and intuitive leakage of data in a multilayer neural network, we can still know x^ and y by analyzing the internal relationship of the neural network and find that x^,y and g are still bijective.


## 3. Super Stochastic Gradient Descent

In this section, we propose our super stochastic gradient descent approach for preventing gradient leakage while keeping the accuracy and then analyze in detail the safety of our approach.


### 3.1. Approach

It was confirmed that the gradient leaks privacy [7, 25]. For solving the security problem caused by the exchange gradient in stochastic gradient descent or mini-batch gradient descent, we propose the super stochastic gradient descent approach, which can protect the gradient information without losing accuracy by hiding part of the gradient information. The gradient is the first-order partial derivative of the objective function, so it is a vector with both magnitude and direction. We seek the gradient of the objective function to find the fastest descent direction. But it is a little related to the modulus length of the gradient vector. Therefore, we hide the modulus length of the gradient vector and convert the gradient vector into a unit vector.The superrandomness, caused by the aggregation of multiple unit gradient vectors, may lead to poor results. To guarantee that this kind of randomness is friendly, we utilize the following approaches to reduce the uncertainty caused by superrandomness.For single-sample training samplexi and label yi, we use unit gradient vector to update parameter θ:(5)θ=θ−η⋅∇θℓθ;xi;yi∇θℓθ;xi;yi.For multiple samples, the parameter is updated to(6)θ=θ−ηm⋅∑j=1m∇θℓθ;xi;i+n;yi;i+nj∇θℓθ;xi;i+n;yi;i+nj,where xi+n represents n samples and yi+n denotes their labels. The gradient ∇ℓθ;xi;i+n;yi;i+n of n samples is considered as a basic gradient, and m is the number of basic gradients. Aggregating the unit gradient vectors of m basic gradients on average is to further enhance the stability of the algorithm. The algorithm has higher performance with strong randomness. It is secure to share this unit basic gradient in a distributed environment.Neuron is the smallest information carrier in the neural network structure. In the neural network, we choose to convert each neuron parameter gradient vector into a unit vector. Therefore, the single-layer neural network parameter is updated to(7)θr=θr−ηm⋅∑j=1m∇θℓθ;xi;i+n;yi;i+nrj∇θℓθ;xi;i+n;yi;i+nrj,where θr represents the rth column or rth row of the parameter matrix in the fully connected layer or convolutional layer (the convolution kernel is regarded as a neuron). In the fully connected layer, ∇θℓθ;xi;i+n;yi;i+nr is expressed as the rth column of the gradient matrix. And in the convolutional layer, it represents the rth row of the gradient matrix of the convolution kernel. Therefore, each row or column of the gradient matrix is a unit vector. Then, we obtain an average gradient matrix by using m such gradient matrices to update the parameters.


### 3.2. The Safety of SSGD

By analyzing the multilayer neural network with ReLu activation function on a training sample, the following relationship is obtained in [9]:(8)G¯i=∑cg¯cDiWi−1⋯W0XWHcTDH⋯Wi+1Di+1,in which X=x1,x2,…,xn is the input data, where xi∈Rd and X∈Rd×n. g¯c represents the cth dimension of the loss vector g¯, T is the number of layers of neural network, Di is the activation pattern of the ith layer of neural network, and G¯i and Wi denote the gradients and parameters of the ith layer of neural network, respectively. In fact, the attack gradient models are all solutions to the above equations. In the distributed training model that needs to share the gradient, the participants know G¯i, Wi, and Di. For data reconstruction attacks, it can infer g¯c and solve X by equation (8).The left side of equation (8) is the ith layer gradient matrix:(9)G¯i=σ1,1iσ1,2i⋯σ1,wiiσ2,1iσ2,2i⋯σ2,wii⋮⋮⋱⋮σwi−1,1iσwi−1,2i⋯σwi−1,wiiwi−1×wi,where wi is the number of neurons in the ith layer. The gradient matrix of our SSGD is(10)G^i=σ1,1iσ1,2i⋯σ1,wiiσ2,1iσ2,2i⋯σ2,wii⋮⋮⋱⋮σwi−1,1iσwi−1,2i⋯σwi−1,wiiwi−1×wi1μ1i0⋯001μ2i⋯0⋮⋮⋱⋮00⋯1μwiiwi×wi.Each column ofG^i is a unit vector, and μ1i is the modulus length of the 1st column vector of the ith layer gradient matrix, i.e., the modulus length of the 1st neuron gradient of the ith layer neural network. Essentially, the parameter matrix of a layer of neural network is multiplied by a diagonal matrix Ui on the right, and the value of the diagonal matrix is the reciprocal of the modulus length of the gradient vector of each neuron. By using our SSGD, equation (8) is represented as(11)G^i=∑cg¯cDiWi−1⋯W0XWHcTDH⋯Wi+1Di+1Ui,where Ui is unknown and is not uniquely determined when the loss functions are nonconvex and nonconcave functions. According to [29], we know that the loss function of multilayer neural networks are nonconvex and nonconcave functions. Due to the dynamicity of Ui, even if g¯c, G^i, Wi, and Di are known, X is not obtained.Our method hides the correlation between the gradient and the sample, eliminates the information between neurons, and achieves neuron-level security. SSGD is a multisample training; there is no information leakage problem like in [19], which is a single-sample leakage of privacy. SSGD can defend against attacks on the gradient.Since training a model requires rounds of iterations, is it safe to use multiple rounds of iterations? We previously analyzed that the gradientg and the training data x^,y are bijective in terms of parameter θ, i.e., g=∇θfθ|x^,y, where f is a functional relationship. We use θi and θi+1 to denote the training parameters of the ith and i + 1st rounds, respectively. Then, we haveθi+1=θi−η⋅gi. The ith gradient gi=∇θfθi|x^,y. Therefore, we have θi+1=θi−η⋅∇θfθi|x^,y. Furthermore, we obtain gi+1=∇θfθi−η⋅∇fθi|x^,y|x^,y. By comparing gi+1 with gi, we can see that there is not additional information in gi+1. The information of the model is only related to the training samples, initial parameters, and learning rate. Therefore, the iteration operation does not cause the information leakage.


## 3.1. Approach

It was confirmed that the gradient leaks privacy [7, 25]. For solving the security problem caused by the exchange gradient in stochastic gradient descent or mini-batch gradient descent, we propose the super stochastic gradient descent approach, which can protect the gradient information without losing accuracy by hiding part of the gradient information. The gradient is the first-order partial derivative of the objective function, so it is a vector with both magnitude and direction. We seek the gradient of the objective function to find the fastest descent direction. But it is a little related to the modulus length of the gradient vector. Therefore, we hide the modulus length of the gradient vector and convert the gradient vector into a unit vector.The superrandomness, caused by the aggregation of multiple unit gradient vectors, may lead to poor results. To guarantee that this kind of randomness is friendly, we utilize the following approaches to reduce the uncertainty caused by superrandomness.For single-sample training samplexi and label yi, we use unit gradient vector to update parameter θ:(5)θ=θ−η⋅∇θℓθ;xi;yi∇θℓθ;xi;yi.For multiple samples, the parameter is updated to(6)θ=θ−ηm⋅∑j=1m∇θℓθ;xi;i+n;yi;i+nj∇θℓθ;xi;i+n;yi;i+nj,where xi+n represents n samples and yi+n denotes their labels. The gradient ∇ℓθ;xi;i+n;yi;i+n of n samples is considered as a basic gradient, and m is the number of basic gradients. Aggregating the unit gradient vectors of m basic gradients on average is to further enhance the stability of the algorithm. The algorithm has higher performance with strong randomness. It is secure to share this unit basic gradient in a distributed environment.Neuron is the smallest information carrier in the neural network structure. In the neural network, we choose to convert each neuron parameter gradient vector into a unit vector. Therefore, the single-layer neural network parameter is updated to(7)θr=θr−ηm⋅∑j=1m∇θℓθ;xi;i+n;yi;i+nrj∇θℓθ;xi;i+n;yi;i+nrj,where θr represents the rth column or rth row of the parameter matrix in the fully connected layer or convolutional layer (the convolution kernel is regarded as a neuron). In the fully connected layer, ∇θℓθ;xi;i+n;yi;i+nr is expressed as the rth column of the gradient matrix. And in the convolutional layer, it represents the rth row of the gradient matrix of the convolution kernel. Therefore, each row or column of the gradient matrix is a unit vector. Then, we obtain an average gradient matrix by using m such gradient matrices to update the parameters.


## 3.2. The Safety of SSGD

By analyzing the multilayer neural network with ReLu activation function on a training sample, the following relationship is obtained in [9]:(8)G¯i=∑cg¯cDiWi−1⋯W0XWHcTDH⋯Wi+1Di+1,in which X=x1,x2,…,xn is the input data, where xi∈Rd and X∈Rd×n. g¯c represents the cth dimension of the loss vector g¯, T is the number of layers of neural network, Di is the activation pattern of the ith layer of neural network, and G¯i and Wi denote the gradients and parameters of the ith layer of neural network, respectively. In fact, the attack gradient models are all solutions to the above equations. In the distributed training model that needs to share the gradient, the participants know G¯i, Wi, and Di. For data reconstruction attacks, it can infer g¯c and solve X by equation (8).The left side of equation (8) is the ith layer gradient matrix:(9)G¯i=σ1,1iσ1,2i⋯σ1,wiiσ2,1iσ2,2i⋯σ2,wii⋮⋮⋱⋮σwi−1,1iσwi−1,2i⋯σwi−1,wiiwi−1×wi,where wi is the number of neurons in the ith layer. The gradient matrix of our SSGD is(10)G^i=σ1,1iσ1,2i⋯σ1,wiiσ2,1iσ2,2i⋯σ2,wii⋮⋮⋱⋮σwi−1,1iσwi−1,2i⋯σwi−1,wiiwi−1×wi1μ1i0⋯001μ2i⋯0⋮⋮⋱⋮00⋯1μwiiwi×wi.Each column ofG^i is a unit vector, and μ1i is the modulus length of the 1st column vector of the ith layer gradient matrix, i.e., the modulus length of the 1st neuron gradient of the ith layer neural network. Essentially, the parameter matrix of a layer of neural network is multiplied by a diagonal matrix Ui on the right, and the value of the diagonal matrix is the reciprocal of the modulus length of the gradient vector of each neuron. By using our SSGD, equation (8) is represented as(11)G^i=∑cg¯cDiWi−1⋯W0XWHcTDH⋯Wi+1Di+1Ui,where Ui is unknown and is not uniquely determined when the loss functions are nonconvex and nonconcave functions. According to [29], we know that the loss function of multilayer neural networks are nonconvex and nonconcave functions. Due to the dynamicity of Ui, even if g¯c, G^i, Wi, and Di are known, X is not obtained.Our method hides the correlation between the gradient and the sample, eliminates the information between neurons, and achieves neuron-level security. SSGD is a multisample training; there is no information leakage problem like in [19], which is a single-sample leakage of privacy. SSGD can defend against attacks on the gradient.Since training a model requires rounds of iterations, is it safe to use multiple rounds of iterations? We previously analyzed that the gradientg and the training data x^,y are bijective in terms of parameter θ, i.e., g=∇θfθ|x^,y, where f is a functional relationship. We use θi and θi+1 to denote the training parameters of the ith and i + 1st rounds, respectively. Then, we haveθi+1=θi−η⋅gi. The ith gradient gi=∇θfθi|x^,y. Therefore, we have θi+1=θi−η⋅∇θfθi|x^,y. Furthermore, we obtain gi+1=∇θfθi−η⋅∇fθi|x^,y|x^,y. By comparing gi+1 with gi, we can see that there is not additional information in gi+1. The information of the model is only related to the training samples, initial parameters, and learning rate. Therefore, the iteration operation does not cause the information leakage.


## 4. Experiments

Data. We use MNIST (https://yann.lecun.com/exdb/mnist) and Fashion-MNIST (https://fashion-mnist.s3-website.eu-central-1.amazonaws.com) datasets to assess the performance of our algorithm. The MNIST contains 60000 training images and 10000 test images, where every image is a 28 × 28 grayscale image, and each pixel is an octet. The Fashion-MNIST [30] is composed of 28 × 28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training set and test set contain 60,000 images and 10,000 images, respectively.Model. The lenet-5 [31] contains two convolutional layers, two pooling layers, and three fully connected layers. The activation function is ReLu. The input dimensions are 784, and output dimensions are 10.Evaluation Index (The Test Accuracy).We use 60000 training images to train model. The test accuracy is the average value of ten experimental results, and every experiment obtains the average test accuracy of randomly selecting 1000 samples from the test set. The number of iterations is 10,000. The highest test accuracy of these compared algorithms in the same experimental environment is shown in bold.


### 4.1. Accuracy and Efficiency

We compare SSGD with SGD, SGDm [32], and Adam [33], which are widely used gradient descent algorithms. The batch size (N=m×n) is set to 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, and 8192, where n is set to 1, 4, 8, 16, 32, 64, and 128 and m is set to 4, 8, 16, 32, and 64. When m = 1, it is the MBGD. There is not set same learning rate as a good experimental result, because SGD and SGDm have poor adaptability in large batches.For MNIST data set, the momentum of SGDm is set to 0.999. For the experimental parameters of Adam, the learning rate is set to5×10−4, and β1 and β2 are set to 0.9 and 0.999, respectively. For SSGD, the learning rate in this experiment is set to 10−1. For Fashion-MNIST data set, the momentum of SGDm is set to 0.99. For the experimental parameters of Adam, the learning rate is set to 10−3, and β1 and β2 are set to 0.9 and 0.999, respectively. For SSGD, the learning rate in this experiment is set to 10−2/1.50.002j, where j is the number of iterations.The comparative experimental results of SGD, SGDm, Adam, and SSGD are shown in Tables1 and 2, where the numbers in bracket in the second and third columns denote the learning rates of SGD and SGDm, respectively, and the number in bracket in the fifth column is the value of m. From Tables 1 and 2, we can see that the performance of our algorithm is better than that of SGD, SGDm, and Adam for large batches of data. In this case, SGD and SGDm need to reduce the learning rate to adapt to it. And Adam also has obvious overfitting in large batches of data. SSGD has always maintained high precision. On the whole, our algorithm on test accuracy is better and more stable than SGD, SGDm, and Adam.Table 1
The test accuracy of compared algorithms on MNIST.
N=m×nSGD (η)SGDm (η)AdamSSGD (m)160.9781 (5×10−4)0.9778 (5×10−4)0.97670.9832 (4)320.9702 (5×10−4)0.9768 (5×10−4)0.98500.9876 (8)640.9794 (5×10−4)0.9792 (5×10−4)0.98420.9900 (8)1280.9676 (5×10−4)0.9780 (5×10−4)0.98960.9901 (16)2560.9755 (10−4)0.9804 (5×10−4)0.98550.9877 (16)5120.9665 (10−4)0.9789 (5×10−4)0.98140.9861 (16)10240.9738 (10−5)0.9749 (10−4)0.97780.9869 (16)20480.9763 (10−5)0.9717 (10−4)0.98940.9886 (64)40960.9703 (10−5)0.9806 (10−4)0.97880.9878 (64)81920.9785 (2×10−6)0.8994 (10−4)0.97530.9855 (64)Table 2
The test accuracy of compared algorithms on fashion-MNIST.
N=m×nSGD (η)SGDm (η)AdamSSGD (m)160.8253 (10−4)0.7795 (10−3)0.71750.8035 (4)320.8241 (10−4)0.8031 (10−3)0.75130.8046 (4)640.8468 (10−4)0.8163 (10−3)0.76740.8344 (4)1280.8629 (10−4)0.8331 (10−3)0.78350.8437 (4)2560.8527 (10−4)0.8511 (10−3)0.82520.8602 (4)5120.8682 (10−4)0.8569 (10−3)0.85660.8590 (4)10240.8569 (10−5)0.8612 (10−3)0.85470.8668 (16)20480.8457 (10−5)0.8351 (10−4)0.85110.8652 (32)40960.8629 (10−6)0.8518 (10−4)0.83210.8704 (32)81920.8325 (10−6)0.8278 (10−4)0.81440.8648 (64)Tables3 and 4 show the running results of our SSGD approach in different numbers of training batches. We can see that the larger the number of training batches (N=m×n) is, the better the test accuracy is. When the number of training batches is too small, the effect of n on performance is greater than that of m. The distribution of m values in Tables 1 and 2 also shows this point.Table 3
Test accuracy of SSGD on MNIST.
n = 1n = 4n = 8n = 16n = 32n = 64n = 128m = 40.97170.98320.98420.98460.98950.98860.9888m = 80.98150.98760.99000.98840.98550.98610.9849m = 160.98010.98540.99010.98770.98610.98690.9832m = 320.98510.98040.99010.98330.98190.98670.9828m = 640.98300.98490.98310.98330.98860.98780.9855Table 4
Test accuracy of SSGD on Fashion-MNIST.
n = 1n = 4n = 8n = 16n = 32n = 64n = 128m = 40.77390.80350.80460.83440.84370.86020.8590m = 80.81520.81360.81500.82710.84010.85640.8655m = 160.81370.81760.82460.84460.84460.86680.8663m = 320.81890.81250.81980.83530.85740.86520.8704m = 640.82150.81960.82380.82890.85770.86550.8648The convergence rate graphs on MNIST and Fashion-MNIST are shown in Figures2(a) and 2(b), respectively. The value in longitudinal axis is the average accuracy of every 10 iterations. The SSGDm is SSGD with momentum. We choose the intermediate value 256 as the batch number in the convergence experiment, where n = 16 and m = 16 for SSGD and SSGDm. In Figure 2(a), the learning rates of SGD and SGDm are 10−4 and 5×10−4, respectively. The momentum of SGDm is set to 0.999. The learning rate of SSGDm is 10/1.0002j, where j is the number of iterations, and its momentum is 0.99. The other parameters are consistent with the above experiment on MNIST. In Figure 2(b), we choose the larger batch number 1024 as the batch number in the convergence experiment, where n = 64 and m = 16 for SSGD and SSGDm. The learning rates of SGD and SGDm are 10−5 and 10−3, respectively. The momentum of SGDm is set to 0.99. The other parameters are consistent with the above experiment on Fashion-MNIST. The learning rate of SSGDm is 1/1.50.002j, where j is the number of iterations, and its momentum is 0.9. From Figure 2, we can see that the convergence speed of our algorithm is faster and more stable than SGD, SGDm, and Adam.Figure 2
The convergence speed of test accuracy (a) on MNIST and (b) on Fashion-MNIST.
(a)(b)


### 4.2. Robustness

Robustness is the robustness of the system, which refers to the characteristic that the system maintains a certain performance under certain parameter perturbations. To check the robustness of our algorithm, we add random noise to the gradient. At the same time, we noticed that differential privacy is a way to protect gradient information by adding random noise that meets a certain distribution. To compare the performances of our algorithm and the model with differential privacy, we choose the model in the robustness experiment to add noise that satisfies differential privacy. In this section, we compare the performances of the traditional gradient descent algorithm and SSGD with noises. In [20], the large gradient does not participate in the update, which will seriously affect the gradient descent performance. However, the large gradient participating in the update will cause the noise scale to be too large, which makes the algorithm effect extremely poor or even unable to converge. Different from cutting gradient value in [20], we strictly define sensitivity as the maximum value minus the minimum value in the gradient matrix. We add Laplacian noises of the same scale on comparing algorithms and set privacy budget ε = 4 and ε = 2 on MNIST and Fashion-MNIST, respectively.We use SGDm and Adam as the compared algorithms. Also, we have tested SGD algorithm. When noise or the number of batches is large, the gradient explosion will occur and the SGD cannot converge on MNIST. SGDm and Adam algorithms have better robustness. Because both SGDm and Adam have momentum, SSGDm is chosen as our comparison algorithm. We adjust hyper parameters to get more performance for SGDm and Adam with noises. To make SGDm, Adam, and SSGDm experiments in the same environment, the batch number isN=n×m, where n is set to 4, 8, 16, 32, and 64, and m is set to 4, 8, 16, 32, and 64. For each iteration, after the n vectors are added, the Laplace noises of ε = 4 or ε = 2 that strictly meet the differential privacy are added. The sensitivity is set to the maximum value minus the minimum value of the gradient matrix of the same batch. Then, we use SGDm, Adam, and SSGDm algorithms to update their parameters, respectively. For SGDm, the momentum is 0.99. The learning rate is 10−2 and 10−3 on MNIST and Fashion-MNIST, respectively. For Adam, the learning rate is 10−3 on MNIST and Fashion-MNIST, β1 = 0.9 and β2 = 0.999. For SSGDm, we use the average of multiple-unit gradient vectors to update the gradient. Therefore, the module length of the update gradient vector decreases very slowly, and dynamic learning rates need to be set. The learning rate of SSGDm is set to 10/1.0002j and 1/1.50.002jon MNIST and Fashion-MNIST, respectively. The momentum = 0.9.From Tables5 and 6, all three algorithms comply with the law of acquaintance; that is, the larger the batch size is, the better the accuracy is. We can see that SSGDm is more robust than the SGDm and Adam algorithms when the noises of the same scale are added in gradients on test accuracy. On MNIST, compared with SGDm and Adam, the average test accuracy of SSGDm is increased by 4.12% and 1.60%, respectively. On Fashion-MNIST, compared with SGDm and Adam, the average test accuracy of SSGDm is increased by 5.24% and 1.64%, respectively.Table 5
Test accuracy withε = 4 on MINST.
SGDm\Adam\SSGDmn = 4n = 8n = 16n = 32n = 64m = 40.8714\0.9321\0.97300.9299\0.9515\0.98160.9508\0.95920.98220.9559\0.9567\0.97740.9518\0.97150.9851m = 80.9242\0.9570\0.97850.9337\0.9657\0.98290.9582\0.9737\0.97900.9569\0.95690.98320.9625\0.9797\0.9802m = 160.9471\0.9607\0.96840.9390\0.9606\0.98390.9662\0.97230.98330.9674\0.9726\0.98440.9699\0.97960.9860m = 320.9203\0.9580\0.97800.9333\0.9693\0.98050.9594\0.9658\0.98590.9647\0.97630.98380.9758\0.9806\0.9837m = 640.9163\0.9545\0.97720.9514\0.9771\0.98610.9560\0.97150.9870.9710\0.9702\0.98530.9678\0.98330.9885Table 6
Test accuracy withε = 2 on Fashion-MNIST.
SGDm\Adam\SSGDmn = 4n = 8n = 16n = 32n = 64m = 40.7089\0.7177\0.77320.7623\0.7729\0.78750.7894\0.7813\0.79470.7941\0.81800.82330.8108\0.8192\0.8343m = 80.7123\0.7402\0.78850.7658\0.7548\0.80400.7806\0.80220.81750.8044\0.8242\0.84120.8159\0.83730.8459m = 160.7127\0.7723\0.79030.7763\0.8052\0.81390.7724\0.8173\0.83890.7993\0.82910.84740.8199\0.8386\0.8455m = 320.7159\0.8013\0.81770.7669\0.8066\0.83330.7735\0.83390.84670.8103\0.8540\0.86110.8299\0.85530.8597m = 640.7158\0.8140\0.82350.7432\0.8309\0.84120.7853\0.8497\0.85240.8064\0.85860.86470.8283\0.8663\0.8655Where is the limit of the robustness of our algorithm? On MINST, we try to increase the scale of noises and makeε be 0.2, 0.5, 1, 2, and 4. The experimental environment is the same as the robustness experiment above, and the parameter settings are also the same. The batch number is set to n = 16 and m = 16. On Fashion-MNIST, we make ε be 0.5, 1, 2, and 4. The batch number is set to n = 64 and m = 16. From Tables 7 and 8, it is clear that our SSGDm has obvious advantages in robustness. The greater the scale of noises is, the more obvious the advantage of our algorithm is.Table 7
The test accuracy by varyingε on MINST.
ε = 0.2ε = 0.5ε = 1ε = 2ε = 4SGDm0.09700.37450.81660.93440.9662Adam0.80740.88130.91400.94160.9723SSGDm0.84810.93950.96710.97190.9833Table 8
The test accuracy by varyingε on fashion-MNIST.
ε = 0.5ε = 1ε = 2ε = 4SGDm0.09950.59210.81990.8363Adam0.64740.79340.83860.8524SSGDm0.75870.81060.84550.8618


### 4.3. Poisoning Attack

The goal of poisoning attack is to destroy the integrity and availability of data. The robustness experiment results show that our algorithm can resist the poisoning attack added to the gradient to a certain extent. According to the previous analysis, the gradient is a kind of mapping of the training data. Then, our algorithm should be effective against data poisoning attacks. This part of the experiment is to verify the performance of our algorithm in data poisoning attacks.SGD is a more basic gradient descent method. In this experiment, we chose SGD as compared algorithm. This experiment compares the performance of SGD, Adam, and SSGD on the same data set with noises. To determine the scale of added noises, the differential privacy mechanisms still are used to add noises with the same methods as the robustness experiment. We add Laplacian noises of different scales to 60,000 training samples. The evaluation method of the experiment result is the same as the above experiment. On MNIST, the batch number is set ton = 64 and m = 4. The learning rate of SGD, Adam, and SSGD is 10−4, 10−4, and 10−2, respectively. On Fashion-MNIST, the batch number is set to n = 64 and m = 16. The learning rate of SGD, Adam, and SSGD is 10−4, 10−4, and 10−2/1.50.002j, respectively. The other settings are the same as the above experiment.Figure3 is the effect picture after adding different noise scales. From Tables 9 and 10, we can see that SSGD is significantly better than SGD and Adam in test accuracy. Also, our algorithm still maintains a higher test accuracy while continuously increasing the scale of noises. Therefore, SSGD can resist gradient poisoning attacks and parametric poisoning attacks to a certain extent.Figure 3
Training sample image of MNIST. (a) The original image. (b) From left to right are the poisoned images withε = 5, 2, and 1, respectively.
(a)(b)Table 9
Test accuracy by varyingε on MNIST.
ε = 1ε = 2ε = 5SGD0.10950.91710.9679Adam0.48590.81600.8890SSGD0.94460.96210.9827Table 10
Test accuracy by varyingε on Fashion-MNIST.
ε = 1ε = 2ε = 5SGD0.10120.58130.7969Adam0.28880.34520.6296SSGD0.46380.76510.8290


### 4.4. Data Reconstruction Attack

Zhu et al. [7] presented an approach which shows the possibility of obtaining private training data from the publicly shared gradients. In their deep leakage from gradient (DLG) method, they synthesized the dummy data and corresponding labels with the supervision of shared gradients. Specifically, they start with random initialization of pseudodata and labels. Virtual gradients are computed on the current shared model in the distributed setup. By minimizing the difference between the virtual gradient and the shared real gradient, they iteratively update the virtual data and labels simultaneously. iDLG [4] is an improvement based on DLG. The following experimental diagrams include the experimental results of DLG [7] and iDLG [4] attacking SGD and SSGD algorithms on MINST datasets and Fashion-MNIST datasets. Figures 4 and 5 are the experimental results of DLG and iDLG attacking SGD. Figures 6 and 7 are about the experimental results of DLG and iDLG attacking SSGD. The number of iterations is 300, and the iteration is stopped if the predetermined accuracy is reached. We can see that our algorithm can defend against DLG and iDLG.Figure 4
DLG attacks SGD (a) on MINST dataset and (b) on Fashion-MNIST dataset.
(a)(b)Figure 5
iDLG attacks SGD (a) on MINST dataset and (b) on Fashion-MNIST dataset.
(a)(b)Figure 6
DLG attacks SSGD (a) on MINST dataset and (b) on Fashion-MNIST dataset.
(a)(b)Figure 7
iDLG attacks SSGD (a) on MINST dataset and (b) on Fashion-MNIST dataset.
(a)(b)


## 4.1. Accuracy and Efficiency

We compare SSGD with SGD, SGDm [32], and Adam [33], which are widely used gradient descent algorithms. The batch size (N=m×n) is set to 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, and 8192, where n is set to 1, 4, 8, 16, 32, 64, and 128 and m is set to 4, 8, 16, 32, and 64. When m = 1, it is the MBGD. There is not set same learning rate as a good experimental result, because SGD and SGDm have poor adaptability in large batches.For MNIST data set, the momentum of SGDm is set to 0.999. For the experimental parameters of Adam, the learning rate is set to5×10−4, and β1 and β2 are set to 0.9 and 0.999, respectively. For SSGD, the learning rate in this experiment is set to 10−1. For Fashion-MNIST data set, the momentum of SGDm is set to 0.99. For the experimental parameters of Adam, the learning rate is set to 10−3, and β1 and β2 are set to 0.9 and 0.999, respectively. For SSGD, the learning rate in this experiment is set to 10−2/1.50.002j, where j is the number of iterations.The comparative experimental results of SGD, SGDm, Adam, and SSGD are shown in Tables1 and 2, where the numbers in bracket in the second and third columns denote the learning rates of SGD and SGDm, respectively, and the number in bracket in the fifth column is the value of m. From Tables 1 and 2, we can see that the performance of our algorithm is better than that of SGD, SGDm, and Adam for large batches of data. In this case, SGD and SGDm need to reduce the learning rate to adapt to it. And Adam also has obvious overfitting in large batches of data. SSGD has always maintained high precision. On the whole, our algorithm on test accuracy is better and more stable than SGD, SGDm, and Adam.Table 1
The test accuracy of compared algorithms on MNIST.
N=m×nSGD (η)SGDm (η)AdamSSGD (m)160.9781 (5×10−4)0.9778 (5×10−4)0.97670.9832 (4)320.9702 (5×10−4)0.9768 (5×10−4)0.98500.9876 (8)640.9794 (5×10−4)0.9792 (5×10−4)0.98420.9900 (8)1280.9676 (5×10−4)0.9780 (5×10−4)0.98960.9901 (16)2560.9755 (10−4)0.9804 (5×10−4)0.98550.9877 (16)5120.9665 (10−4)0.9789 (5×10−4)0.98140.9861 (16)10240.9738 (10−5)0.9749 (10−4)0.97780.9869 (16)20480.9763 (10−5)0.9717 (10−4)0.98940.9886 (64)40960.9703 (10−5)0.9806 (10−4)0.97880.9878 (64)81920.9785 (2×10−6)0.8994 (10−4)0.97530.9855 (64)Table 2
The test accuracy of compared algorithms on fashion-MNIST.
N=m×nSGD (η)SGDm (η)AdamSSGD (m)160.8253 (10−4)0.7795 (10−3)0.71750.8035 (4)320.8241 (10−4)0.8031 (10−3)0.75130.8046 (4)640.8468 (10−4)0.8163 (10−3)0.76740.8344 (4)1280.8629 (10−4)0.8331 (10−3)0.78350.8437 (4)2560.8527 (10−4)0.8511 (10−3)0.82520.8602 (4)5120.8682 (10−4)0.8569 (10−3)0.85660.8590 (4)10240.8569 (10−5)0.8612 (10−3)0.85470.8668 (16)20480.8457 (10−5)0.8351 (10−4)0.85110.8652 (32)40960.8629 (10−6)0.8518 (10−4)0.83210.8704 (32)81920.8325 (10−6)0.8278 (10−4)0.81440.8648 (64)Tables3 and 4 show the running results of our SSGD approach in different numbers of training batches. We can see that the larger the number of training batches (N=m×n) is, the better the test accuracy is. When the number of training batches is too small, the effect of n on performance is greater than that of m. The distribution of m values in Tables 1 and 2 also shows this point.Table 3
Test accuracy of SSGD on MNIST.
n = 1n = 4n = 8n = 16n = 32n = 64n = 128m = 40.97170.98320.98420.98460.98950.98860.9888m = 80.98150.98760.99000.98840.98550.98610.9849m = 160.98010.98540.99010.98770.98610.98690.9832m = 320.98510.98040.99010.98330.98190.98670.9828m = 640.98300.98490.98310.98330.98860.98780.9855Table 4
Test accuracy of SSGD on Fashion-MNIST.
n = 1n = 4n = 8n = 16n = 32n = 64n = 128m = 40.77390.80350.80460.83440.84370.86020.8590m = 80.81520.81360.81500.82710.84010.85640.8655m = 160.81370.81760.82460.84460.84460.86680.8663m = 320.81890.81250.81980.83530.85740.86520.8704m = 640.82150.81960.82380.82890.85770.86550.8648The convergence rate graphs on MNIST and Fashion-MNIST are shown in Figures2(a) and 2(b), respectively. The value in longitudinal axis is the average accuracy of every 10 iterations. The SSGDm is SSGD with momentum. We choose the intermediate value 256 as the batch number in the convergence experiment, where n = 16 and m = 16 for SSGD and SSGDm. In Figure 2(a), the learning rates of SGD and SGDm are 10−4 and 5×10−4, respectively. The momentum of SGDm is set to 0.999. The learning rate of SSGDm is 10/1.0002j, where j is the number of iterations, and its momentum is 0.99. The other parameters are consistent with the above experiment on MNIST. In Figure 2(b), we choose the larger batch number 1024 as the batch number in the convergence experiment, where n = 64 and m = 16 for SSGD and SSGDm. The learning rates of SGD and SGDm are 10−5 and 10−3, respectively. The momentum of SGDm is set to 0.99. The other parameters are consistent with the above experiment on Fashion-MNIST. The learning rate of SSGDm is 1/1.50.002j, where j is the number of iterations, and its momentum is 0.9. From Figure 2, we can see that the convergence speed of our algorithm is faster and more stable than SGD, SGDm, and Adam.Figure 2
The convergence speed of test accuracy (a) on MNIST and (b) on Fashion-MNIST.
(a)(b)


## 4.2. Robustness

Robustness is the robustness of the system, which refers to the characteristic that the system maintains a certain performance under certain parameter perturbations. To check the robustness of our algorithm, we add random noise to the gradient. At the same time, we noticed that differential privacy is a way to protect gradient information by adding random noise that meets a certain distribution. To compare the performances of our algorithm and the model with differential privacy, we choose the model in the robustness experiment to add noise that satisfies differential privacy. In this section, we compare the performances of the traditional gradient descent algorithm and SSGD with noises. In [20], the large gradient does not participate in the update, which will seriously affect the gradient descent performance. However, the large gradient participating in the update will cause the noise scale to be too large, which makes the algorithm effect extremely poor or even unable to converge. Different from cutting gradient value in [20], we strictly define sensitivity as the maximum value minus the minimum value in the gradient matrix. We add Laplacian noises of the same scale on comparing algorithms and set privacy budget ε = 4 and ε = 2 on MNIST and Fashion-MNIST, respectively.We use SGDm and Adam as the compared algorithms. Also, we have tested SGD algorithm. When noise or the number of batches is large, the gradient explosion will occur and the SGD cannot converge on MNIST. SGDm and Adam algorithms have better robustness. Because both SGDm and Adam have momentum, SSGDm is chosen as our comparison algorithm. We adjust hyper parameters to get more performance for SGDm and Adam with noises. To make SGDm, Adam, and SSGDm experiments in the same environment, the batch number isN=n×m, where n is set to 4, 8, 16, 32, and 64, and m is set to 4, 8, 16, 32, and 64. For each iteration, after the n vectors are added, the Laplace noises of ε = 4 or ε = 2 that strictly meet the differential privacy are added. The sensitivity is set to the maximum value minus the minimum value of the gradient matrix of the same batch. Then, we use SGDm, Adam, and SSGDm algorithms to update their parameters, respectively. For SGDm, the momentum is 0.99. The learning rate is 10−2 and 10−3 on MNIST and Fashion-MNIST, respectively. For Adam, the learning rate is 10−3 on MNIST and Fashion-MNIST, β1 = 0.9 and β2 = 0.999. For SSGDm, we use the average of multiple-unit gradient vectors to update the gradient. Therefore, the module length of the update gradient vector decreases very slowly, and dynamic learning rates need to be set. The learning rate of SSGDm is set to 10/1.0002j and 1/1.50.002jon MNIST and Fashion-MNIST, respectively. The momentum = 0.9.From Tables5 and 6, all three algorithms comply with the law of acquaintance; that is, the larger the batch size is, the better the accuracy is. We can see that SSGDm is more robust than the SGDm and Adam algorithms when the noises of the same scale are added in gradients on test accuracy. On MNIST, compared with SGDm and Adam, the average test accuracy of SSGDm is increased by 4.12% and 1.60%, respectively. On Fashion-MNIST, compared with SGDm and Adam, the average test accuracy of SSGDm is increased by 5.24% and 1.64%, respectively.Table 5
Test accuracy withε = 4 on MINST.
SGDm\Adam\SSGDmn = 4n = 8n = 16n = 32n = 64m = 40.8714\0.9321\0.97300.9299\0.9515\0.98160.9508\0.95920.98220.9559\0.9567\0.97740.9518\0.97150.9851m = 80.9242\0.9570\0.97850.9337\0.9657\0.98290.9582\0.9737\0.97900.9569\0.95690.98320.9625\0.9797\0.9802m = 160.9471\0.9607\0.96840.9390\0.9606\0.98390.9662\0.97230.98330.9674\0.9726\0.98440.9699\0.97960.9860m = 320.9203\0.9580\0.97800.9333\0.9693\0.98050.9594\0.9658\0.98590.9647\0.97630.98380.9758\0.9806\0.9837m = 640.9163\0.9545\0.97720.9514\0.9771\0.98610.9560\0.97150.9870.9710\0.9702\0.98530.9678\0.98330.9885Table 6
Test accuracy withε = 2 on Fashion-MNIST.
SGDm\Adam\SSGDmn = 4n = 8n = 16n = 32n = 64m = 40.7089\0.7177\0.77320.7623\0.7729\0.78750.7894\0.7813\0.79470.7941\0.81800.82330.8108\0.8192\0.8343m = 80.7123\0.7402\0.78850.7658\0.7548\0.80400.7806\0.80220.81750.8044\0.8242\0.84120.8159\0.83730.8459m = 160.7127\0.7723\0.79030.7763\0.8052\0.81390.7724\0.8173\0.83890.7993\0.82910.84740.8199\0.8386\0.8455m = 320.7159\0.8013\0.81770.7669\0.8066\0.83330.7735\0.83390.84670.8103\0.8540\0.86110.8299\0.85530.8597m = 640.7158\0.8140\0.82350.7432\0.8309\0.84120.7853\0.8497\0.85240.8064\0.85860.86470.8283\0.8663\0.8655Where is the limit of the robustness of our algorithm? On MINST, we try to increase the scale of noises and makeε be 0.2, 0.5, 1, 2, and 4. The experimental environment is the same as the robustness experiment above, and the parameter settings are also the same. The batch number is set to n = 16 and m = 16. On Fashion-MNIST, we make ε be 0.5, 1, 2, and 4. The batch number is set to n = 64 and m = 16. From Tables 7 and 8, it is clear that our SSGDm has obvious advantages in robustness. The greater the scale of noises is, the more obvious the advantage of our algorithm is.Table 7
The test accuracy by varyingε on MINST.
ε = 0.2ε = 0.5ε = 1ε = 2ε = 4SGDm0.09700.37450.81660.93440.9662Adam0.80740.88130.91400.94160.9723SSGDm0.84810.93950.96710.97190.9833Table 8
The test accuracy by varyingε on fashion-MNIST.
ε = 0.5ε = 1ε = 2ε = 4SGDm0.09950.59210.81990.8363Adam0.64740.79340.83860.8524SSGDm0.75870.81060.84550.8618


## 4.3. Poisoning Attack

The goal of poisoning attack is to destroy the integrity and availability of data. The robustness experiment results show that our algorithm can resist the poisoning attack added to the gradient to a certain extent. According to the previous analysis, the gradient is a kind of mapping of the training data. Then, our algorithm should be effective against data poisoning attacks. This part of the experiment is to verify the performance of our algorithm in data poisoning attacks.SGD is a more basic gradient descent method. In this experiment, we chose SGD as compared algorithm. This experiment compares the performance of SGD, Adam, and SSGD on the same data set with noises. To determine the scale of added noises, the differential privacy mechanisms still are used to add noises with the same methods as the robustness experiment. We add Laplacian noises of different scales to 60,000 training samples. The evaluation method of the experiment result is the same as the above experiment. On MNIST, the batch number is set ton = 64 and m = 4. The learning rate of SGD, Adam, and SSGD is 10−4, 10−4, and 10−2, respectively. On Fashion-MNIST, the batch number is set to n = 64 and m = 16. The learning rate of SGD, Adam, and SSGD is 10−4, 10−4, and 10−2/1.50.002j, respectively. The other settings are the same as the above experiment.Figure3 is the effect picture after adding different noise scales. From Tables 9 and 10, we can see that SSGD is significantly better than SGD and Adam in test accuracy. Also, our algorithm still maintains a higher test accuracy while continuously increasing the scale of noises. Therefore, SSGD can resist gradient poisoning attacks and parametric poisoning attacks to a certain extent.Figure 3
Training sample image of MNIST. (a) The original image. (b) From left to right are the poisoned images withε = 5, 2, and 1, respectively.
(a)(b)Table 9
Test accuracy by varyingε on MNIST.
ε = 1ε = 2ε = 5SGD0.10950.91710.9679Adam0.48590.81600.8890SSGD0.94460.96210.9827Table 10
Test accuracy by varyingε on Fashion-MNIST.
ε = 1ε = 2ε = 5SGD0.10120.58130.7969Adam0.28880.34520.6296SSGD0.46380.76510.8290


## 4.4. Data Reconstruction Attack

Zhu et al. [7] presented an approach which shows the possibility of obtaining private training data from the publicly shared gradients. In their deep leakage from gradient (DLG) method, they synthesized the dummy data and corresponding labels with the supervision of shared gradients. Specifically, they start with random initialization of pseudodata and labels. Virtual gradients are computed on the current shared model in the distributed setup. By minimizing the difference between the virtual gradient and the shared real gradient, they iteratively update the virtual data and labels simultaneously. iDLG [4] is an improvement based on DLG. The following experimental diagrams include the experimental results of DLG [7] and iDLG [4] attacking SGD and SSGD algorithms on MINST datasets and Fashion-MNIST datasets. Figures 4 and 5 are the experimental results of DLG and iDLG attacking SGD. Figures 6 and 7 are about the experimental results of DLG and iDLG attacking SSGD. The number of iterations is 300, and the iteration is stopped if the predetermined accuracy is reached. We can see that our algorithm can defend against DLG and iDLG.Figure 4
DLG attacks SGD (a) on MINST dataset and (b) on Fashion-MNIST dataset.
(a)(b)Figure 5
iDLG attacks SGD (a) on MINST dataset and (b) on Fashion-MNIST dataset.
(a)(b)Figure 6
DLG attacks SSGD (a) on MINST dataset and (b) on Fashion-MNIST dataset.
(a)(b)Figure 7
iDLG attacks SSGD (a) on MINST dataset and (b) on Fashion-MNIST dataset.
(a)(b)


## 5. Conclusions

In this paper, we propose a new gradient descent approach, called super stochastic gradient descent. The SSGD enhances the randomness of gradients to protect against gradient-based attacks. Simultaneously, we use multisample aggregation to enhance stability and eliminate the uncertainty brought about by superrandomness. Our approach achieves neuron-level security and can defend against attacks on the gradient. Experimental results demonstrate that SSGD has good accuracy and strong robustness because its stability and randomness are enhanced. SSGD can also resist model poisoning attacks to a certain extent. But for attacks with the same degree of poisoning, data poisoning has a greater impact on performance. In the future, we will continue to find a more suitable method for resisting data poisoning attacks.



---

*Source: 5404061-2021-08-10.xml*