# ePMLF: Efficient and Privacy-Preserving Machine Learning Framework Based on Fog Computing

Authors: Ruoli Zhao; Yong Xie; Hong Cheng; Xingxing Jia; Syed Hamad Shirazi
Journal: International Journal of Intelligent Systems
 (2023)

---

## Abstract

With the continuous improvement of computation and communication capabilities, the Internet of Things (IoT) plays a vital role in many intelligent applications. Therefore, IoT devices generate a large amount of data every day, which lays a solid foundation for the success of machine learning. However, the strong privacy requirements of the IoT data make its machine learning very difficult. To protect data privacy, many privacy-preserving machine learning schemes have been proposed. At present, most schemes only aim at specific models and lack general solutions, which is not an ideal solution in engineering practice. In order to meet this challenge, we propose an efficient and privacy-preserving machine learning training framework (ePMLF) in a fog computing environment. The ePMLF framework can let the software service provider (SSP) perform privacy-preserving model training with the data on the fog nodes. The security of the data on the fog nodes can be protected and the model parameters can only be obtained by SSP. The proposed secure data normalization method in the framework further improves the accuracy of the training model. Experimental analysis shows that our framework significantly reduces the computation and communication overhead compared with the existing scheme.


---

## Body

## 1. Introduction

At present, the application of the IoT can realize the real-time collection of user data. The large amount of data generated every day provides a good basis for training high-quality machine learning models. However, privacy issues hinder the application of machine learning [1, 2]. On the one hand, training data contains people’s sensitive information, which is not allowed to be disclosed. On the other hand, the trained machine learning model is a valuable property of SSP and the leakage of the model will bring serious economic losses to SSP. Therefore, the training process faces a serious problem of privacy disclosure. The strong privacy requirements of data make training very difficult.In order to solve the problem of privacy disclosure in the training process, many privacy-preserving machine learning training schemes have been proposed. There are two main solutions: secure collaborative training and secure outsourcing training. The main application scenario of secure collaborative training is federated learning [3]. In federated learning, clients complete the training locally. Then, upload and download model parameters to the server. Therefore, frequent interaction leads to a high communication overhead of federated learning [4]. In addition, the goal of federated learning is to train a global model for each data provider, which is different from ours [5]. In this paper, the goal of our proposed framework is to train a private model for SSP by using the ciphertext data of data providers (fog nodes). Secure outsourcing training is mainly based on cloud computing. Cloud servers provide a lot of storage and computation resources. Data providers use homomorphic encryption or secret sharing technologies to convert their data into ciphertext data and outsource them to cloud servers. In [6–8], they all adopt the dual cloud server model and assume that the two cloud servers do not collude. However, this assumption has potential security hazards [5].Through the above analysis, the existing privacy-preserving schemes mainly have two problems. First, these schemes are only for specific machine learning algorithms and cannot meet all machine learning algorithms. The second is the lack of global normalization processing of data. Normalization without disclosing the data can make the training process converge to the optimal solution more easily. To solve these problems, Zhu et al. [5] proposed a privacy-preserving ML training framework, which contains multiple secure training protocols for the aggregation scenario and defends security under collusion situations. However, the scheme in [5] does not make a global data normalization. In addition, there are still some shortcomings to be improved, including the function of building blocks is not comprehensive and high communication and computation overhead, which makes the framework impractical.Therefore, we propose an efficient and practical privacy-preserving machine learning training framework in a fog computing environment. Specifically, with the continuous increase of data, it brings a heavy storage burden to IoT devices with limited resources. Therefore, deploying fog nodes with higher configurations near IoT devices has become an effective solution [9]. IoT devices will store the collected data in nearby fog nodes. We assume that a SSP wants to use the data in the fog nodes to train its own private model. In the training process, the data in the fog nodes will not be leaked to SSP and other fog nodes. The model of SSP will not be leaked to fog nodes. Our contributions are as follows:(1)
Based on the requirements of data privacy-preserving in the IoT environment and the characteristics of fog computing, we propose ePMLF. Through ePMLF, fog nodes can protect the privacy of the data and let an untrusted SSP train different machine-learning models. At the same time, ePMLF provides the function of model updating.(2)
The ePMLF implements secure data processing. We propose two secure data normalization methods (secure z-score and secure min-max), which can normalize the data distribution of all fog nodes and form high-quality training data. Based on the OU cryptosystem, we propose a method to encrypt negative numbers. Unlike the existing scheme [10], which can only encrypt negative numbers by the party holding the private key, our proposed method can allow any party to encrypt negative numbers.(3)
In ePMLF, we define the ciphertext as the encrypted data and its precision, which can prevent the resulting error caused by inconsistent precision in ciphertext computation. Then, we design a precision control protocol, which can avoid the plaintext overflow caused by multiple homomorphic operations. Based on the ciphertext defined by ePMLF, we propose some secure algorithms as the basic building blocks of the framework. Experimental results show that these algorithms are helpful to improve training efficiency.(4)
We implement the proposed framework. Strict security analysis shows that our framework meets the strong privacy-preserving requirements of all participants. Through comparative evaluation, our scheme significantly reduces the communication and computation overhead.


## 2. Related Work

The existing privacy-preserving machine learning training scenarios are mainly divided into two categories: secure collaborative training and secure outsourcing training.In the scenario of secure collaborative training, each participant has some computation resources and their own data. Therefore, each participant undertakes some computation and communication tasks and cooperatively trains a global machine-learning model on the joint training data. Data owners should keep their data confidential during the training [11–13]. Dani et al. [14] proposed protocols for solving the secure multi-party computation problem. Mehnaz et al. [15] proposed a secure sum protocol with strong security guarantees and used this protocol to propose two secure gradient-descent algorithms. Saha et al. [16] proposed a fog-enabled federated learning framework-FogFL to facilitate distributed learning and reduce communication latency and energy consumption of resource-constrained edge devices. Xu et al. [17] presented a secure and verifiable federated learning scheme, with which federated deep learning is achieved and the final learning results are verifiable. Wang et al. [18] proposed a privacy-preserving federated learning scheme for regression training, which is noninteractive in the whole training process. Zhao et al. [19] proposed a collaborative architecture based on orbital edge computing and low-orbit satellite network communication.In the scenario of secure outsourcing training, data owners with limited computation resources outsource their ciphertext data to cloud servers. The cloud servers perform the privacy-preserving machine learning training process and train a private machine learning model for the training service requester. Liu et al. [6] designed a system for privacy-preserving decision tree training and evaluation in a twin-cloud architecture. Liu et al. [7] proposed a secure ML-kNN training and classification scheme. Wang et al. [8] proposed an efficient privacy-preserving outsourced SVM scheme, which protects the privacy of training data and the SVM model. Zhang et al. [20] proposed a secure deep computation model by offloading the expensive operations to the cloud. Li et al. [21] proposed an outsourced privacy-preserving C4.5 algorithm over horizontally and vertically partitioned data for multiple parties. Li et al. [22] proposed a privacy-preserving multi-party machine learning framework and the data owners do not need to participate in the training process. Liu et al. [23] proposed a privacy-preserving clinical decision support system in the outsourced cloud computing environment. Deploying machine learning services in the cloud has become a flexible training solution. However, this approach generally relies on the assumption of noncollusion, which is a serious security risk.


## 3. Preliminaries

### 3.1. Homomorphic Encryption

#### 3.1.1. Okamoto-Uchiyama (OU) Cryptosystem

OU cryptosystem is a public key cryptosystem with additive homomorphism [24]. We will introduce OU cryptosystem as follows:(i)
Key Generation: choose two big primesp,q,p=q=l,n=p2⋅q,Lx=x−1/p. Generate a random number g∈Zn∗. Compute h=gnmodn. The public key is pk=n,g,h,l. The private key is sk=p,q.(ii)
Encryption: givenm∈0,2l−1. The message m will be encrypted with pk. The ciphertext is c=gmhrmodn, where r∈Zn is a random number.(iii)
Decryption: given a ciphertextc. Compute m=Lcp−1modp2/Lgp−1modp2modp.(iv)
Homomorphic computation: given two ciphertextsm1,m2 under the same public key pk. The homomorphic computations are defined as m1+m2=m1⋅m2,m1⋅m2=m2m1.


### 3.2. Cloud-ElGamal Cryptosystem

In this paper, we select an enhanced version of ElGamal called Cloud-ElGamal [25], which supports multiplicative homomorphism and resists confidentiality attacks.(i)
Key Generation: choose a big primep. Find a generator a of Zp∗. Generate a random integer d,1<d<p−1 and compute y=admodp. The evaluation key is ek=p. The private key is sk=a,d,y.(ii)
Encryption: givenm∈Zp. Generate a random integer k,1<k<p−1 and compute c1=akmodp,c2=ykmmodp. The ciphertext is c=c1,c2.(iii)
Decryption: Given a ciphertextc. Compute m=c2⋅c1d−1modp.(iv)
Homomorphic computation: given two ciphertextsm1,m2 under the same private key. The homomorphic computations are defined as m1⋅m2=m1⋅m2.


### 3.3. Machine Learning Training

Machine learning training consists of data preprocessing and model training.For data preprocessing, we focus on data normalization for continuous data (such as age and height). There are two main methods of data normalization: the min–max method and the z-score method.The min-max method maps the values of attributes between 0 and 1 according to the maximum and minimum values of attributes in the dataset. For attributef, the min-max method will compute as follows:(1)x′=x−minfmaxf−minf.The z-score method is to standardize attributes based on the average and standard deviation of attributes in the dataset. For attributef, we assume that the average of f is μf and the standard deviation of f is σf. The z-score method will compute as follows:(2)x′=x−μfσf.Through data normalization, high-quality training data can be formed.For model training, there are many model training algorithms, which include many linear and nonlinear operations. For example, logistic regression, SVM, and naive Bayes. To train a logistic regression model, the parameterw will be updated by computing w=w−learn_rate⋅x1/1+e−wx−y. To train a SVM model, the parameter w will be updated by computing w=w−learn_rate⋅z⋅w−z⋅yx. For training a naive Bayes model, we should compute the class prior probability Py and the conditional probability Px|y.


## 3.1. Homomorphic Encryption

### 3.1.1. Okamoto-Uchiyama (OU) Cryptosystem

OU cryptosystem is a public key cryptosystem with additive homomorphism [24]. We will introduce OU cryptosystem as follows:(i)
Key Generation: choose two big primesp,q,p=q=l,n=p2⋅q,Lx=x−1/p. Generate a random number g∈Zn∗. Compute h=gnmodn. The public key is pk=n,g,h,l. The private key is sk=p,q.(ii)
Encryption: givenm∈0,2l−1. The message m will be encrypted with pk. The ciphertext is c=gmhrmodn, where r∈Zn is a random number.(iii)
Decryption: given a ciphertextc. Compute m=Lcp−1modp2/Lgp−1modp2modp.(iv)
Homomorphic computation: given two ciphertextsm1,m2 under the same public key pk. The homomorphic computations are defined as m1+m2=m1⋅m2,m1⋅m2=m2m1.


## 3.1.1. Okamoto-Uchiyama (OU) Cryptosystem

OU cryptosystem is a public key cryptosystem with additive homomorphism [24]. We will introduce OU cryptosystem as follows:(i)
Key Generation: choose two big primesp,q,p=q=l,n=p2⋅q,Lx=x−1/p. Generate a random number g∈Zn∗. Compute h=gnmodn. The public key is pk=n,g,h,l. The private key is sk=p,q.(ii)
Encryption: givenm∈0,2l−1. The message m will be encrypted with pk. The ciphertext is c=gmhrmodn, where r∈Zn is a random number.(iii)
Decryption: given a ciphertextc. Compute m=Lcp−1modp2/Lgp−1modp2modp.(iv)
Homomorphic computation: given two ciphertextsm1,m2 under the same public key pk. The homomorphic computations are defined as m1+m2=m1⋅m2,m1⋅m2=m2m1.


## 3.2. Cloud-ElGamal Cryptosystem

In this paper, we select an enhanced version of ElGamal called Cloud-ElGamal [25], which supports multiplicative homomorphism and resists confidentiality attacks.(i)
Key Generation: choose a big primep. Find a generator a of Zp∗. Generate a random integer d,1<d<p−1 and compute y=admodp. The evaluation key is ek=p. The private key is sk=a,d,y.(ii)
Encryption: givenm∈Zp. Generate a random integer k,1<k<p−1 and compute c1=akmodp,c2=ykmmodp. The ciphertext is c=c1,c2.(iii)
Decryption: Given a ciphertextc. Compute m=c2⋅c1d−1modp.(iv)
Homomorphic computation: given two ciphertextsm1,m2 under the same private key. The homomorphic computations are defined as m1⋅m2=m1⋅m2.


## 3.3. Machine Learning Training

Machine learning training consists of data preprocessing and model training.For data preprocessing, we focus on data normalization for continuous data (such as age and height). There are two main methods of data normalization: the min–max method and the z-score method.The min-max method maps the values of attributes between 0 and 1 according to the maximum and minimum values of attributes in the dataset. For attributef, the min-max method will compute as follows:(1)x′=x−minfmaxf−minf.The z-score method is to standardize attributes based on the average and standard deviation of attributes in the dataset. For attributef, we assume that the average of f is μf and the standard deviation of f is σf. The z-score method will compute as follows:(2)x′=x−μfσf.Through data normalization, high-quality training data can be formed.For model training, there are many model training algorithms, which include many linear and nonlinear operations. For example, logistic regression, SVM, and naive Bayes. To train a logistic regression model, the parameterw will be updated by computing w=w−learn_rate⋅x1/1+e−wx−y. To train a SVM model, the parameter w will be updated by computing w=w−learn_rate⋅z⋅w−z⋅yx. For training a naive Bayes model, we should compute the class prior probability Py and the conditional probability Px|y.


## 4. System Overview

In this section, we will introduce our proposed framework ePMLF, including the system model, design goal, and threat model.


### 4.1. System Model

In our system model, the goal is to train a private machine-learning model for SSP. The training task is completed by fog nodes and SSP. At the same time, the privacy data of each fog node will not be leaked to SSP and other fog nodes. SSP’s trained model will not be leaked to fog nodes. Therefore, our system model is designed as shown in Figure1.Figure 1
System model.There are four participants in our system model, which are trusted authority (TA), IoT devices, fog nodes (FNs), and software service provider (SSP).(i)
Trusted authority (TA): TA is a trusted authority and generates system parameters for all participants.(ii)
IoT devices: IoT devices will produce a large amount of IoT data. However, their computation and storage resources are very limited.(iii)
Fog nodes (FNs): fog nodes have strong computation and storage capacity. They collect, store and manage the data generated by IoT devices. The data owned by FNs belongs to sensitive information and cannot be leaked.(iv)
Software service provider (SSP): SSP wants to train machine learning models through the data in the FNs. SSP is not trusted and will try to obtain the data of FNs during the training processing.


### 4.2. Design Goal

Based on the system model, our design goal includes privacy-preserving, training high-accuracy machine learning models, and low computation and communication overhead.


#### 4.2.1. Privacy Preserving

(1)
The data of FNs should be preserved(2)
The trained model of SSP should be preserved


#### 4.2.2. Training High Accuracy Machine Learning Model

When using ciphertext data from FNs to train machine learning models, it is very important to ensure the high accuracy of the trained models. Therefore, we need to generate high-quality training data through data normalization and design a general computing framework.


#### 4.2.3. Low Computation and Communication Overhead

For achieving privacy-preserving training, the proposed ePMLF is designed based on cryptography technology. Cryptography technology will bring significant computation and communication overhead, so we should improve the framework efficiency as much as possible.


### 4.3. Threat Model

In our proposed ePMLF, we assume that FNs and SSPs are honest-but-curious (TA is trusted). They will implement the protocol honestly, but they try to obtain the data of other participants by analyzing the results of the processing.At the same time, we allowm−1 FNs at most collude with each other to analyze the privacy of other participants and m−2 FNs at most collude with SSP to analyze the privacy of other participants, which is the same as [5]. To prove our proposed scheme is secure, we define an adversary. The adversary can eavesdrop and analyze data during the data transmission. The data transmission process is the interaction between participants in protocol implementation.


## 4.1. System Model

In our system model, the goal is to train a private machine-learning model for SSP. The training task is completed by fog nodes and SSP. At the same time, the privacy data of each fog node will not be leaked to SSP and other fog nodes. SSP’s trained model will not be leaked to fog nodes. Therefore, our system model is designed as shown in Figure1.Figure 1
System model.There are four participants in our system model, which are trusted authority (TA), IoT devices, fog nodes (FNs), and software service provider (SSP).(i)
Trusted authority (TA): TA is a trusted authority and generates system parameters for all participants.(ii)
IoT devices: IoT devices will produce a large amount of IoT data. However, their computation and storage resources are very limited.(iii)
Fog nodes (FNs): fog nodes have strong computation and storage capacity. They collect, store and manage the data generated by IoT devices. The data owned by FNs belongs to sensitive information and cannot be leaked.(iv)
Software service provider (SSP): SSP wants to train machine learning models through the data in the FNs. SSP is not trusted and will try to obtain the data of FNs during the training processing.


## 4.2. Design Goal

Based on the system model, our design goal includes privacy-preserving, training high-accuracy machine learning models, and low computation and communication overhead.


### 4.2.1. Privacy Preserving

(1)
The data of FNs should be preserved(2)
The trained model of SSP should be preserved


### 4.2.2. Training High Accuracy Machine Learning Model

When using ciphertext data from FNs to train machine learning models, it is very important to ensure the high accuracy of the trained models. Therefore, we need to generate high-quality training data through data normalization and design a general computing framework.


### 4.2.3. Low Computation and Communication Overhead

For achieving privacy-preserving training, the proposed ePMLF is designed based on cryptography technology. Cryptography technology will bring significant computation and communication overhead, so we should improve the framework efficiency as much as possible.


## 4.2.1. Privacy Preserving

(1)
The data of FNs should be preserved(2)
The trained model of SSP should be preserved


## 4.2.2. Training High Accuracy Machine Learning Model

When using ciphertext data from FNs to train machine learning models, it is very important to ensure the high accuracy of the trained models. Therefore, we need to generate high-quality training data through data normalization and design a general computing framework.


## 4.2.3. Low Computation and Communication Overhead

For achieving privacy-preserving training, the proposed ePMLF is designed based on cryptography technology. Cryptography technology will bring significant computation and communication overhead, so we should improve the framework efficiency as much as possible.


## 4.3. Threat Model

In our proposed ePMLF, we assume that FNs and SSPs are honest-but-curious (TA is trusted). They will implement the protocol honestly, but they try to obtain the data of other participants by analyzing the results of the processing.At the same time, we allowm−1 FNs at most collude with each other to analyze the privacy of other participants and m−2 FNs at most collude with SSP to analyze the privacy of other participants, which is the same as [5]. To prove our proposed scheme is secure, we define an adversary. The adversary can eavesdrop and analyze data during the data transmission. The data transmission process is the interaction between participants in protocol implementation.


## 5. Our Proposed Framework

In this section, we describe our proposed framework in detail. The ePMLF mainly includes system initialization, data normalization, basic building blocks, privacy-preserving machine learning training, and machine learning model updating. The workflow of our proposed framework is shown in Figure2.Figure 2
System workflow.In order to accurately describe our proposed scheme, we give the description of used notations in Table1.Table 1
Notation and definition.
NotationDefinitionlThe key length (bit)lxThe key length ofxPKiO,SKiOOU public-private key pair ofFNiPKsO,SKsOOU public-private key pair of SSPEKiE,SKiECloud-ElGamal evaluation-private key pair ofFNiEThe precision of floating-point numbersxPKThe ciphertext ofx under PK


### 5.1. System Initialization

In the system initialization phase, TA generates system parameters for all participants.Improved OU encryption.Zhang et al. [10] achieved the negative integers encryption based on the OU cryptosystem. They divided the plaintext space into two parts as follows: 0,p/2 represents positive integers and p/2,p−1 represents negative. For a negative integer −m, it should be converted to p−m. It should be noted that p is the private key of OU cryptosystem.Through the above analysis, we can find that the negative integers encryption method proposed by Zhang et al. can only be implemented by the participants with the private key, which is not practical. Therefore, we propose a new method to realize that any participants can encrypt negative integers based on the OU cryptosystem without disclosing the private key. Specifically, TA generates a large primeq′ after generating p,q. TA computes N=p⋅q′ and publics N. When a participant without a private key needs to encrypt the negative integer −m, computing N−m and encrypting it. In decryption, N is eliminated by modulus p. The range of encrypted integers is −p/2,p/2. We prove the correctness of our proposed method as follows.Negative integers encryption(3)c=gN−mhrmodn.Negative integers decryption(4)m=Lcp−1modp2Lgp−1modp2modp=Lgp−1)N−m⋅gp−1)rnmodp2Lgp−1modp2modp=Lgp−1N−mmodp2+Lgp−1rnmodp2Lgp−1modp2modp=L1+kpN−mmodp2+L1+kprnmodp2L1+kpmodp=L1+N−mkpmodp2+L1L1+kpmodp=L1+pq′−mkpmodp2+L1L1+kpmodp=L1+p2kq′−mkpmodp2+L1L1+kpmodp=L1−mkp+L1L1+kpmodp.Generate system parameters.(1)
We assume that there arem fog nodes in our system. TA generates a OU public-private key pair PKiO=ni,gi,hi,Ni,l,SKiO=pi,qi and a Cloud-ElGamal evaluation-private key pair EKiE=pie,SKiE=ai,di,yi for FN FNi.(2)
TA generates two random integersα, β, lα=lβ<l and splits α, β to m random integers, α=α1+α2+⋯+αm, β=β1+β2+⋯+βm. Then, TA distributes αi, βi to FNi and α, β to SSP by a secure communication channel.(3)
TA generates a OU public-private key pairPKsO=ns,gs,hs,Ns,l,SKsO=ps,qs for SSP.


### 5.2. Privacy Preserving Data Normalization

In our proposed framework, SSP uses the encrypted data of all FNs to train a machine-learning model. Therefore, we need to normalize the data by all FNs, which can improve the quality of training data.The data format ofFNi is x1ij,x2ij,…,xdij,yij,j=1,2,…,ni. In the processing of data normalization, any participants cannot know the data of FNi.Secure z-score as follows:(1)
For thek-th dimension data, FNi computes xki=∑j=1nixkij. Then, FNi encrypts xki with PKsO and sends xki+βiPKsO,i=1,2,…,m and ni to SSP.(2)
SSP computesxk¯=∑i=1mxki−β/∑mi=1ni and sends it to each FN.(3)
FNi computes σ=∑j=1nixkij−xk¯2/ni and xkij=xkij−xk¯/σ.Secure min–max as follows:In our proposed secure min-max, the (3) and (4) computation method is the same as [18].(1)
Each FN computes the maximum and minimum values of each dimensional data.FNii=1,2,…,m can obtain x1i_max,x2i_max,…,xdi_max and x1i_min,x2i_min,…,xdi_min.(2)
Forxji_maxxji_min, setting pos=1. Starting with i=2, FNpos and FNi input xjpos_maxxjpos_min and xji_maxxji_min into the comparison protocol [26] respectively to compare. If xjpos_max<xji_maxxjpos_min>xji_min, pos=i.(3)
Forxjpos_maxxjpos_min, FNpos generates a random integer RjmaxRjmin, lRjmax=lxjpos_maxlRjmin=lxjpos_min and computes xjpos_max+Rjmaxxjpos_min−Rjmin. Then, FNpos publics xjpos_max+Rjmax (xjpos_min−Rjmin).(4)
After computing the global maximum and minimum values, the data will be standardized according tox1max+R1max,x2max+R2max,…,xdmax and x1min−R1min,x2min−R2min,…,xdmin−Rdmin. FNi computes as follows:(5)xkij=xkij−xkmin−Rkminxkmax+Rkmax−xkmin−Rkmin.


### 5.3. Basic Building Blocks

In order to make our framework realize the privacy-preserving machine learning training, we will modularize the proposed framework by designing some general building blocks.


#### 5.3.1. Precision Control

In the machine learning training process, many data are floating-point numbers. Therefore, it is necessary to convert floating-point numbers to integers before encrypting. Generally, the conversion method is to multiply the floating-point number by2E or 10E [26], E is the precision. This method will significantly expand the original data. However, the plaintext space of the encryption algorithm is limited. Using the expanded data for homomorphic operation will lead to the problem of plaintext overflow and the pression of the data will change. For example, x2E⋅y2E⟶xy22E, xy22E+z2E⟶xy2E+z2E≠xy+z2E.To solve this problem, we propose a method called precision control. We express the encrypted data asc,t. The c is the ciphertext data and the t is the precision of the data. For example, xy⋅22E is expressed as xy⋅22E,2. Through the method, we can know the current precision of the data. When E⋅t+1≥l, we need to reduce the precision bits of encrypted data to avoid plaintext overflow. In order to better control the precision, we require that any participant must set t=1 before encrypting the data. The detailed method is as follows and is described in Algorithm 1.(1)
SSP chooses a random integerR,lR<E. For x2t⋅EPKiO,t, SSP sends x2t⋅E+R2t⋅E,t to FNi. For x2t⋅ESKiE,t, SSP sends x2t⋅ERSKiE,t to FNi.(2)
Forx2t⋅E+R2t⋅EPKiO,t,FNi computes x2E+R2E=x2t⋅E+R2t⋅E/2t−1E. For x2t⋅ERSKiE,t, FNi computes x2ER=x2t⋅ER/2t−1E. Then, FNi encrypts x2E+R2E (or x2ER) and sends x2E+R2EPKiE,1 (or x2ERSKiE,1) to SSP.(3)
Forx2E+R2EPKiE,1, SSP computes x2EPKiE,1←x2E+R2EPKiE⋅Ni−R2EPKiE. For x2ERSKiE,1, SSP computes the inverse R−1 of R modulo pie and obtains x2ESKiE,1←x2ERSKiE⋅R−1.Algorithm 1: Precision control.
Input:x2t⋅EPKiO,t or x2t⋅ESKiE,t,i∈1,mOutput:x2EPKiO,1 or x2ESKiE,1SSP:(1)
Sendx2t⋅E+R2t⋅EPKiO,torx2t⋅ERSKiE,t to FNiFNi:(2)
Decryptx2t⋅E+R2t⋅EPKiO,torx2t⋅ERSKiE,t(3)
Computex2E+R2E=x2t⋅E+R2t⋅E/2t−1Eorx2ER=x2t⋅ER/2t−1E and encrypt x2E(4)
Sendx2E+R2EPKiO,1 or x2ERSKiE,1 to SSPSSP:(5)
Computex2EPKiO,1←x2E+R2EPKiO⋅Ni−R2EPKiEorx2ESKiE,1←x2ERSKiE⋅R−1


#### 5.3.2. Secure Addition

Given two ciphertext data encrypted withPKiO, x2t⋅EPKiO,t and y2t⋅EPKiO,t. SSP will obtain x+y2t⋅EPKiO,t=x2t⋅EPKiO⋅y2t⋅EPKiO,t.


#### 5.3.3. Secure Subtraction

Given two ciphertext data encrypted withPKiO, x2t⋅EPKiO,t and y2t⋅EPKiO,t. SSP will obtain x−y2t⋅EPKiO,t=x2t⋅EPKiO⋅y2t⋅EPKiONi−1,t.


#### 5.3.4. Secure Multiplication (OU)

Given a ciphertext data encrypted withPKiO, x2t⋅EPKiO,t and a plaintext data of SSP, y⋅2E. SSP will obtain xy2t+1EPKiO,t+1=x2t⋅EPKiOy⋅2E,t+1.


#### 5.3.5. Secure Multiplication (Cloud-ElGamal):

Given two ciphertext data encrypted withSKiE, x2t1⋅ESKiE,t1 and y2t2⋅ESKiE,t2. SSP will obtain xy2t1+t2ESKiE,t1+t2=x2t1⋅ESKiE⋅y2t2⋅ESKiE,t1+t2.


#### 5.3.6. Secure Division:

The algorithm is inspired by [27]. Given two ciphertext data encrypted with PKiO, x2t1⋅EPKiO,t1 and y2t2⋅EPKiO,t2. SSP will obtain x/y22EPKiO,2. The computation process is as follows:(1)
SSP chooses two random integersR1,R2∈0,2l/2 and computes(6)x2t1⋅ER1PKiO=x2t1⋅EPKiOR1,y2t2⋅ER2PKiO=y2t2⋅EPKiOR2.Then, SSP sends them toFNi(2)
FNi decrypts x2t1⋅ER1PKiO, y2t2⋅ER2PKiO and obtains xR1/yR2. Sending xR1/yR22EPKiO,1 to SSP(3)
SSP computesx/y22EPKiO,2←xR1/yR22EPKiOR2/R12ESecure power computationGiven a ciphertext data encrypted withSKiE, x2t⋅ESKiE,t and a plaintext data of SSP, y⋅2E. SSP will obtain xy2E,1. The computation process is as follows:(1)
SSP computes(7)xy2E2y2Et⋅ESKiE=aiky2Emodpie,yikx2t⋅E)y2Emodpie.Choosing a random integerR1∈0,2l/3, R2,R2′=R22E∈0,2l/3 and computes xy2E2y2Et⋅ER1SKiE=xy2E2y2Et⋅ESKiE⋅R1, 2y2Et⋅ER1R2′. Then, sending xy2E2y2Et⋅ER1SKiE and 2y2Et⋅ER1R2′SKiE to FNi.(2)
FNi decrypts and computes(8)xyR2=2Exy2E2y2Et⋅ER12y2Et⋅ER1R2′.Then, sendingxy/R22ESKiE,1 to SSP(3)
SSP computesxy2ESKiE,1←xy/R22ESKiE⋅R2


#### 5.3.7. Secure Inner Product

Given an encrypted vector ofFNi, xPKiO=x12t⋅EPKiO,t,x22t⋅EPKiO,t,…,xd2t⋅EPKiO,t and a plaintext vector of SSP, y=y12E,y22E,…,yd2E. SSP will obtain(9)xy2t+1EPKiO,t+1=∏j=1dxj2t⋅EPKiOyj2E,t+1.


#### 5.3.8. Secure Summation

SSP wants to obtain the summation of∑j=1msi and si belongs to FNi. The process is described in Algorithm 2.(1)
FNi encrypts si′=si+αi with PKsO and sends si′PKsO to SSP(2)
SSP computess′PKsO=∏i=1msi′PKsO and decrypts s′PKsO. Then, SSP can obtain s=s′−αAlgorithm 2: Secure summation.
Input:si,i=1,2,…,mOutput:sFNs:fori=1⟶m:(1)FNi computes si′=si+αi(2) Encryptsi′ with PKsO and send si′PKsO to SSPendforSSP:(3) Computes′PKsO=∏i=1msi′PKsO(4) Decrypts′PKsO and compute s=s′−α


#### 5.3.9. Secure Sigmoid Function:

To complete the nonlinear computation of the sigmoid function, we propose an algorithm called Secure Sigmoid Function. The process is described in Algorithm3.(1)
FNi has ex=ex1,ex2,…,exd. FNi encrypts exj2E with SKiE and sends it to SSP(2)
SSP hasw=w1,w2,…,wd. SSP computes(10)ewjxj2E2wj2EESKiE=aik)wj2Emodpie,yikexj2E)wj2Emodpie2wj2EE=2Ewj2E,ewx⋅2E2ws2EESKiE=∏j=1dewjxj2E2wj2EESKiE∏j=1d2wj2EE,wherews=w1+w2+⋯+wd. Then, SSP chooses two random integers R1,R2, R1,R2′=R22E∈0,2l/4 and sends ewx⋅2E2ws2EER1SKiE, 2ws2EER1R2′ to FNi(3)
FNi decrypts and computes(11)ewxR2=2Eewx⋅2E2ws2EER12ws2EER1R2′.Then, sendingewx/R22EPKiO to SSP(4)
SSP chooses two random integersR3,R4∈0,2l/4 computes as follows:(12)ewx2ER3PKiO=ewxR22EPKiOR2R3,ewx+12ER3R4PKiO=ewx2EPKiO⋅2EPKiOR3R4.Then, SSP sendsewx2ER3PKiO and ewx+12ER3R4PKiO to FNi(5)
FNi decrypts ewx2ER3PKiO, ewx+12ER3R4PKiO and obtains(13)xjewxewx+1R4=xjewxR3ewx+1R3R4.Then, SSP sendsxjewx/ewx+1R42EPKiO to SSP(6)
SSP computes(14)xjewxewx+12EPKiO,1←xjewxewx+1R42EPKiOR4.Algorithm 3: Secure sigmoid function.
Input:ex=ex1,ex2,…,exd,w=w1,w2,…,wdOutput:ewxFNi:forj=1⟶d:(1)
Sendexj2ESKiE to SSPendforSSP:(2)
ewjxj2E2wj2EESKiE=aik)wj2Emodpie,yikexj2E)wj2Emodpie(3)
2wj2EE=2Ewj2E(4)
ewx⋅2E2ws2EESKiE=∏j=1dewjxj2E2wj2EESKiE(5)
2ws2EE=∏j=1d2wj2EE(6)
Choose two random integersR1,R2, R1,R2′=R22E∈0,2l/4(7)
Sendewx⋅2E2ws2EER1SKiE, 2ws2EER1R2′ to FNiFNi:(8)
Decrypt and computeewx/R2=2Eewx⋅2E2ws2EER1/2ws2EER1R2′(9)
Sendewx/R22EPKiO to SSPSSP:(10)
Choose two random integersR3,R4∈0,2l/4(11)
ewx2ER3PKiO=ewx/R22EPKiOR2R3(12)
ewx+12ER3R4PKiO=ewx2EPKiO⋅2EPKiOR3R4(13)
Sendewx2ER3PKiO and ewx+12ER3R4PKiO to FNiFNi:(14)
Decryptewx2ER3PKiO, ewx+12ER3R4PKiO(15)
xjewx/ewx+1R4=xjewxR3/ewx+1R3R4(16)
Sendxjewx/ewx+1R42EPKiO to SSPSSP:(17)
xjewx/ewx+12EPKiO,1←xjewx/ewx+1R42EPKiOR4


#### 5.3.10. Secure Sign Computation

Given encrypted data that is computed by SSP,x2t⋅EPKiO,tx2t⋅ESKiE,t. FNi needs to know if x2t⋅E<0 but cannot know the value of x2t⋅E.SSP flips a coins and chooses a random integer R∈0,2l/2−t⋅E. If s=1, SSP computes tmpPKiO=x2t⋅ERPKiOtmpSKiE=x2t⋅ERSKiE. If s=0, SSP computes tmpPKiO=−x2t⋅ERPKiOtmpSKiO=−x2t⋅ERSKiE. Then, SSP sends the computed data to FNi. FNi decrypts and obtains tmp. Let judge=1 if tmp<0 else judge=0. FNi sends judgePKsO. If s=1 and judge=1, SSP will know x2t⋅E<0. If s=1 and judge=0, x2t⋅E≥0. If s=0 and judge=1, x2t⋅E≥0. If s=0 and judge=0, x2t⋅E<0.Converting Cloud-ElGamal to OUFor this building block, we cite building block 7 of [5].Converting OU:For this building block, we cite building block 8 of [5].


### 5.4. Privacy Preserving Machine Learning Training

In this section, we achieve four training protocols based on our proposed building blocks, which are popular machine learning models. We assume that the training data has been normalized before training.


### 5.5. Secure Logistic Regression (LR) Training

We use the stochastic gradient descent (SGD) algorithm to train a Logistic Regression model. SSP randomly selectsT data of FNs (all data of FNs have been numbered). The process is described in Algorithm 4.Algorithm 4: Secure logistic regression training.
Input: the selected data of FNs, iterationsT, learning rate LOutput: model parametersW=w1,w2,…,wdFNs:fori=1⟶m:(1)
FNi encrypts the selected data with PKiOSuch asxjj∈1,d, send xjy2EPKiO,1 to SSPendforSSP:forit=1⟶T:(2)
xjewx/ewx+12EPKiO,1←SecureSigmoidFunctionforj=1⟶d:(3)
SSP performsSecureAddition and SecureMultiplicationOU(4)
wj22EPKiO,2←wj22EPKiOxjewx/ewx+12EPKiONi−L2E⋅xjy2EPKiOL2E(5)
Converting OU:wj22EPKsO←wj22EPKiO(6)
wj2E=wj22E/2Eendforendfor


### 5.6. Secure SVM Training

For training a SVM model, we also use the SGD algorithm. The process is described in Algorithm5.Algorithm 5: Secure SVM training.
Input: the selected data of FNs,iterationsT, learning rate L, regularization parameter zOutput: model parametersW=w1,w2,…,wdFNs:fori=1⟶m:(1)
FNi encrypts the selected data with PKiOSuch asxjj∈1,d,sendxj2EPKiO,1, xjy2EPKiO,1 and y to SSPendforSSP:forit=1⟶T:(2)
wx22EPKiO,2←SecureInnerProductxPKiO,w(3)
t1PKiO=wx22EPKiO⋅Ni−1PKiOy=1t2PKiO=wx22EPKiONi−1⋅Ni−1PKiOy=−1(4)
SSP andFNi perform SecureSignComputationfort1PKiO, t2PKiOFNi:(5)
In the process ofSecureSignComputation:ify=1: FNi will judge t1PKiOelse: FNi will judge t2PKiOSSP:(7)
To computewj22EPKiO,2, SSP performsSecureAddition and SecureMultiplicationOUifs=1,judge=1ors=0,judge=0:forj=1⟶d:wj22EPKiO,2←wj22EPKiO⋅wj2EPKiONi−L⋅z2E⋅xjy2EPKiOL⋅z2EConverting OU:wj22EPKsO←wj22EPKiOwj2E=wj2E/2Eendforelse:forj=1⟶d:wj22EPKiO,2⟵wj22EPKiO⋅wj2EPKiONi−L⋅z2EConverting OU:wj22EPKsO⟵wj22EPKiOwj2E=wj2E/2Eendforendfor


### 5.7. Secure Naive Bayes (NB) Training

To train a naive Bayes model, the user needs to aggregate the class prior probabilityPy and the conditional probability Px|y from FNs. We assume that it is a binary classification problem. The process is described in Algorithm 6.Algorithm 6: Secure naive Bayes training.
Input: the data of FNsOutput: model parametersPy=1,Py=0,Pxi=0|y=0,Pxi=0|y=1,Pxi=1|y=0,Pxi=0|y=1FNs:fori=1⟶m:(1)
FNi computes Sy=1i+αi=∑j=1niyj+αi,Sxk=0|y=0i+αi,Sxk=0|y=1i+αi,k=1,2,…,dEncrypt them withPKsO and send them to SSPendforSSP:(2)
ComputeSy=1+αPKsO=∏i=1mSy=1i+αiPKsO,Sxk=0|y=0+αSxk=0|y=0+αSxk=0|y=0+αPKsO=∏i=1mSxk=0|y=0i+αiPKsOSxk=0|y=1+αSxk=0|y=1+αSxk=0|y=1+αPKsO=∏i=1mSxk=0|y=1i+αiPKsO(3)
Decrypt them withSKsOfork=1⟶d:(4)
Pxk=0|y=0=Sxk=0|y=0/n−Sy=1Pxk=0|y=1=Sxk=0|y=1/Sy=1Pxk=1|y=0=1−Pxk=0|y=0Pxk=1|y=1=1−Pxk=0|y=1endfor(5)
Py=1=Sy=1/n,Py=0=1−Py=1


### 5.8. Secure Deep Neural Network Training

The training process of deep neural networks contains nonlinear computations such as ReLU. To compute a nonlinear activation function, we will make an approximation method. The approximation method is proposed in [28] and the nonlinear activation functions will be converted to polynomial functions. Then, the training process can be completed with our proposed building blocks. In addition, we use the training method proposed in [29] to train the deep neural network models.


### 5.9. Privacy Preserving Machine Learning Model Updating

With the continuous increase of fog node data in the system or the addition of new fog nodes in the system, the quantity and quality of the overall data will be significantly improved. Therefore, it is very important to update the SSP’s trained model. For different types of machine learning models, we propose different updating methods. Specifically, we divide the machine learning model into the model trained by the gradient-descent method and the model trained by a nongradient-descent method. In order to prevent a differential attack, the model trained by the nongradient-descent method can be updated only when the data of the original fog nodes increases to more than 50%.A model trained by the gradient-descent method:For the model trained based on the gradient-descent method (e.g., LR and SVM), the updating process of the model is as follows:(1)
TA determines the fog nodes and SSP participating in the updating process and redistributes the system parameters for them (in the way of 5.1).(2)
FNs and SSP train a new model on the added data. SSP will obtain the new modelw′.(3)
Based on thew′ and the original model w, SSP will compute the updated model wi=n/n+n′wi+n′/n+n′wi′,i=1,2,…,d (n is the number of original data and n′ is the number of added data).


### 5.10. Model Trained by Nongradient-Descent Method

For the model trained by the nongradient-descent method (for example, naive Bayes), the updating process of the model is as follows:(1)
TA redistributes system parameters for all fog nodes and SSP to update the model (in the way of 5.1)(2)
FNs and SSPs use all the data to retrain a model. SSP takes the model as the updated model


## 5.1. System Initialization

In the system initialization phase, TA generates system parameters for all participants.Improved OU encryption.Zhang et al. [10] achieved the negative integers encryption based on the OU cryptosystem. They divided the plaintext space into two parts as follows: 0,p/2 represents positive integers and p/2,p−1 represents negative. For a negative integer −m, it should be converted to p−m. It should be noted that p is the private key of OU cryptosystem.Through the above analysis, we can find that the negative integers encryption method proposed by Zhang et al. can only be implemented by the participants with the private key, which is not practical. Therefore, we propose a new method to realize that any participants can encrypt negative integers based on the OU cryptosystem without disclosing the private key. Specifically, TA generates a large primeq′ after generating p,q. TA computes N=p⋅q′ and publics N. When a participant without a private key needs to encrypt the negative integer −m, computing N−m and encrypting it. In decryption, N is eliminated by modulus p. The range of encrypted integers is −p/2,p/2. We prove the correctness of our proposed method as follows.Negative integers encryption(3)c=gN−mhrmodn.Negative integers decryption(4)m=Lcp−1modp2Lgp−1modp2modp=Lgp−1)N−m⋅gp−1)rnmodp2Lgp−1modp2modp=Lgp−1N−mmodp2+Lgp−1rnmodp2Lgp−1modp2modp=L1+kpN−mmodp2+L1+kprnmodp2L1+kpmodp=L1+N−mkpmodp2+L1L1+kpmodp=L1+pq′−mkpmodp2+L1L1+kpmodp=L1+p2kq′−mkpmodp2+L1L1+kpmodp=L1−mkp+L1L1+kpmodp.Generate system parameters.(1)
We assume that there arem fog nodes in our system. TA generates a OU public-private key pair PKiO=ni,gi,hi,Ni,l,SKiO=pi,qi and a Cloud-ElGamal evaluation-private key pair EKiE=pie,SKiE=ai,di,yi for FN FNi.(2)
TA generates two random integersα, β, lα=lβ<l and splits α, β to m random integers, α=α1+α2+⋯+αm, β=β1+β2+⋯+βm. Then, TA distributes αi, βi to FNi and α, β to SSP by a secure communication channel.(3)
TA generates a OU public-private key pairPKsO=ns,gs,hs,Ns,l,SKsO=ps,qs for SSP.


## 5.2. Privacy Preserving Data Normalization

In our proposed framework, SSP uses the encrypted data of all FNs to train a machine-learning model. Therefore, we need to normalize the data by all FNs, which can improve the quality of training data.The data format ofFNi is x1ij,x2ij,…,xdij,yij,j=1,2,…,ni. In the processing of data normalization, any participants cannot know the data of FNi.Secure z-score as follows:(1)
For thek-th dimension data, FNi computes xki=∑j=1nixkij. Then, FNi encrypts xki with PKsO and sends xki+βiPKsO,i=1,2,…,m and ni to SSP.(2)
SSP computesxk¯=∑i=1mxki−β/∑mi=1ni and sends it to each FN.(3)
FNi computes σ=∑j=1nixkij−xk¯2/ni and xkij=xkij−xk¯/σ.Secure min–max as follows:In our proposed secure min-max, the (3) and (4) computation method is the same as [18].(1)
Each FN computes the maximum and minimum values of each dimensional data.FNii=1,2,…,m can obtain x1i_max,x2i_max,…,xdi_max and x1i_min,x2i_min,…,xdi_min.(2)
Forxji_maxxji_min, setting pos=1. Starting with i=2, FNpos and FNi input xjpos_maxxjpos_min and xji_maxxji_min into the comparison protocol [26] respectively to compare. If xjpos_max<xji_maxxjpos_min>xji_min, pos=i.(3)
Forxjpos_maxxjpos_min, FNpos generates a random integer RjmaxRjmin, lRjmax=lxjpos_maxlRjmin=lxjpos_min and computes xjpos_max+Rjmaxxjpos_min−Rjmin. Then, FNpos publics xjpos_max+Rjmax (xjpos_min−Rjmin).(4)
After computing the global maximum and minimum values, the data will be standardized according tox1max+R1max,x2max+R2max,…,xdmax and x1min−R1min,x2min−R2min,…,xdmin−Rdmin. FNi computes as follows:(5)xkij=xkij−xkmin−Rkminxkmax+Rkmax−xkmin−Rkmin.


## 5.3. Basic Building Blocks

In order to make our framework realize the privacy-preserving machine learning training, we will modularize the proposed framework by designing some general building blocks.


### 5.3.1. Precision Control

In the machine learning training process, many data are floating-point numbers. Therefore, it is necessary to convert floating-point numbers to integers before encrypting. Generally, the conversion method is to multiply the floating-point number by2E or 10E [26], E is the precision. This method will significantly expand the original data. However, the plaintext space of the encryption algorithm is limited. Using the expanded data for homomorphic operation will lead to the problem of plaintext overflow and the pression of the data will change. For example, x2E⋅y2E⟶xy22E, xy22E+z2E⟶xy2E+z2E≠xy+z2E.To solve this problem, we propose a method called precision control. We express the encrypted data asc,t. The c is the ciphertext data and the t is the precision of the data. For example, xy⋅22E is expressed as xy⋅22E,2. Through the method, we can know the current precision of the data. When E⋅t+1≥l, we need to reduce the precision bits of encrypted data to avoid plaintext overflow. In order to better control the precision, we require that any participant must set t=1 before encrypting the data. The detailed method is as follows and is described in Algorithm 1.(1)
SSP chooses a random integerR,lR<E. For x2t⋅EPKiO,t, SSP sends x2t⋅E+R2t⋅E,t to FNi. For x2t⋅ESKiE,t, SSP sends x2t⋅ERSKiE,t to FNi.(2)
Forx2t⋅E+R2t⋅EPKiO,t,FNi computes x2E+R2E=x2t⋅E+R2t⋅E/2t−1E. For x2t⋅ERSKiE,t, FNi computes x2ER=x2t⋅ER/2t−1E. Then, FNi encrypts x2E+R2E (or x2ER) and sends x2E+R2EPKiE,1 (or x2ERSKiE,1) to SSP.(3)
Forx2E+R2EPKiE,1, SSP computes x2EPKiE,1←x2E+R2EPKiE⋅Ni−R2EPKiE. For x2ERSKiE,1, SSP computes the inverse R−1 of R modulo pie and obtains x2ESKiE,1←x2ERSKiE⋅R−1.Algorithm 1: Precision control.
Input:x2t⋅EPKiO,t or x2t⋅ESKiE,t,i∈1,mOutput:x2EPKiO,1 or x2ESKiE,1SSP:(1)
Sendx2t⋅E+R2t⋅EPKiO,torx2t⋅ERSKiE,t to FNiFNi:(2)
Decryptx2t⋅E+R2t⋅EPKiO,torx2t⋅ERSKiE,t(3)
Computex2E+R2E=x2t⋅E+R2t⋅E/2t−1Eorx2ER=x2t⋅ER/2t−1E and encrypt x2E(4)
Sendx2E+R2EPKiO,1 or x2ERSKiE,1 to SSPSSP:(5)
Computex2EPKiO,1←x2E+R2EPKiO⋅Ni−R2EPKiEorx2ESKiE,1←x2ERSKiE⋅R−1


### 5.3.2. Secure Addition

Given two ciphertext data encrypted withPKiO, x2t⋅EPKiO,t and y2t⋅EPKiO,t. SSP will obtain x+y2t⋅EPKiO,t=x2t⋅EPKiO⋅y2t⋅EPKiO,t.


### 5.3.3. Secure Subtraction

Given two ciphertext data encrypted withPKiO, x2t⋅EPKiO,t and y2t⋅EPKiO,t. SSP will obtain x−y2t⋅EPKiO,t=x2t⋅EPKiO⋅y2t⋅EPKiONi−1,t.


### 5.3.4. Secure Multiplication (OU)

Given a ciphertext data encrypted withPKiO, x2t⋅EPKiO,t and a plaintext data of SSP, y⋅2E. SSP will obtain xy2t+1EPKiO,t+1=x2t⋅EPKiOy⋅2E,t+1.


### 5.3.5. Secure Multiplication (Cloud-ElGamal):

Given two ciphertext data encrypted withSKiE, x2t1⋅ESKiE,t1 and y2t2⋅ESKiE,t2. SSP will obtain xy2t1+t2ESKiE,t1+t2=x2t1⋅ESKiE⋅y2t2⋅ESKiE,t1+t2.


### 5.3.6. Secure Division:

The algorithm is inspired by [27]. Given two ciphertext data encrypted with PKiO, x2t1⋅EPKiO,t1 and y2t2⋅EPKiO,t2. SSP will obtain x/y22EPKiO,2. The computation process is as follows:(1)
SSP chooses two random integersR1,R2∈0,2l/2 and computes(6)x2t1⋅ER1PKiO=x2t1⋅EPKiOR1,y2t2⋅ER2PKiO=y2t2⋅EPKiOR2.Then, SSP sends them toFNi(2)
FNi decrypts x2t1⋅ER1PKiO, y2t2⋅ER2PKiO and obtains xR1/yR2. Sending xR1/yR22EPKiO,1 to SSP(3)
SSP computesx/y22EPKiO,2←xR1/yR22EPKiOR2/R12ESecure power computationGiven a ciphertext data encrypted withSKiE, x2t⋅ESKiE,t and a plaintext data of SSP, y⋅2E. SSP will obtain xy2E,1. The computation process is as follows:(1)
SSP computes(7)xy2E2y2Et⋅ESKiE=aiky2Emodpie,yikx2t⋅E)y2Emodpie.Choosing a random integerR1∈0,2l/3, R2,R2′=R22E∈0,2l/3 and computes xy2E2y2Et⋅ER1SKiE=xy2E2y2Et⋅ESKiE⋅R1, 2y2Et⋅ER1R2′. Then, sending xy2E2y2Et⋅ER1SKiE and 2y2Et⋅ER1R2′SKiE to FNi.(2)
FNi decrypts and computes(8)xyR2=2Exy2E2y2Et⋅ER12y2Et⋅ER1R2′.Then, sendingxy/R22ESKiE,1 to SSP(3)
SSP computesxy2ESKiE,1←xy/R22ESKiE⋅R2


### 5.3.7. Secure Inner Product

Given an encrypted vector ofFNi, xPKiO=x12t⋅EPKiO,t,x22t⋅EPKiO,t,…,xd2t⋅EPKiO,t and a plaintext vector of SSP, y=y12E,y22E,…,yd2E. SSP will obtain(9)xy2t+1EPKiO,t+1=∏j=1dxj2t⋅EPKiOyj2E,t+1.


### 5.3.8. Secure Summation

SSP wants to obtain the summation of∑j=1msi and si belongs to FNi. The process is described in Algorithm 2.(1)
FNi encrypts si′=si+αi with PKsO and sends si′PKsO to SSP(2)
SSP computess′PKsO=∏i=1msi′PKsO and decrypts s′PKsO. Then, SSP can obtain s=s′−αAlgorithm 2: Secure summation.
Input:si,i=1,2,…,mOutput:sFNs:fori=1⟶m:(1)FNi computes si′=si+αi(2) Encryptsi′ with PKsO and send si′PKsO to SSPendforSSP:(3) Computes′PKsO=∏i=1msi′PKsO(4) Decrypts′PKsO and compute s=s′−α


### 5.3.9. Secure Sigmoid Function:

To complete the nonlinear computation of the sigmoid function, we propose an algorithm called Secure Sigmoid Function. The process is described in Algorithm3.(1)
FNi has ex=ex1,ex2,…,exd. FNi encrypts exj2E with SKiE and sends it to SSP(2)
SSP hasw=w1,w2,…,wd. SSP computes(10)ewjxj2E2wj2EESKiE=aik)wj2Emodpie,yikexj2E)wj2Emodpie2wj2EE=2Ewj2E,ewx⋅2E2ws2EESKiE=∏j=1dewjxj2E2wj2EESKiE∏j=1d2wj2EE,wherews=w1+w2+⋯+wd. Then, SSP chooses two random integers R1,R2, R1,R2′=R22E∈0,2l/4 and sends ewx⋅2E2ws2EER1SKiE, 2ws2EER1R2′ to FNi(3)
FNi decrypts and computes(11)ewxR2=2Eewx⋅2E2ws2EER12ws2EER1R2′.Then, sendingewx/R22EPKiO to SSP(4)
SSP chooses two random integersR3,R4∈0,2l/4 computes as follows:(12)ewx2ER3PKiO=ewxR22EPKiOR2R3,ewx+12ER3R4PKiO=ewx2EPKiO⋅2EPKiOR3R4.Then, SSP sendsewx2ER3PKiO and ewx+12ER3R4PKiO to FNi(5)
FNi decrypts ewx2ER3PKiO, ewx+12ER3R4PKiO and obtains(13)xjewxewx+1R4=xjewxR3ewx+1R3R4.Then, SSP sendsxjewx/ewx+1R42EPKiO to SSP(6)
SSP computes(14)xjewxewx+12EPKiO,1←xjewxewx+1R42EPKiOR4.Algorithm 3: Secure sigmoid function.
Input:ex=ex1,ex2,…,exd,w=w1,w2,…,wdOutput:ewxFNi:forj=1⟶d:(1)
Sendexj2ESKiE to SSPendforSSP:(2)
ewjxj2E2wj2EESKiE=aik)wj2Emodpie,yikexj2E)wj2Emodpie(3)
2wj2EE=2Ewj2E(4)
ewx⋅2E2ws2EESKiE=∏j=1dewjxj2E2wj2EESKiE(5)
2ws2EE=∏j=1d2wj2EE(6)
Choose two random integersR1,R2, R1,R2′=R22E∈0,2l/4(7)
Sendewx⋅2E2ws2EER1SKiE, 2ws2EER1R2′ to FNiFNi:(8)
Decrypt and computeewx/R2=2Eewx⋅2E2ws2EER1/2ws2EER1R2′(9)
Sendewx/R22EPKiO to SSPSSP:(10)
Choose two random integersR3,R4∈0,2l/4(11)
ewx2ER3PKiO=ewx/R22EPKiOR2R3(12)
ewx+12ER3R4PKiO=ewx2EPKiO⋅2EPKiOR3R4(13)
Sendewx2ER3PKiO and ewx+12ER3R4PKiO to FNiFNi:(14)
Decryptewx2ER3PKiO, ewx+12ER3R4PKiO(15)
xjewx/ewx+1R4=xjewxR3/ewx+1R3R4(16)
Sendxjewx/ewx+1R42EPKiO to SSPSSP:(17)
xjewx/ewx+12EPKiO,1←xjewx/ewx+1R42EPKiOR4


### 5.3.10. Secure Sign Computation

Given encrypted data that is computed by SSP,x2t⋅EPKiO,tx2t⋅ESKiE,t. FNi needs to know if x2t⋅E<0 but cannot know the value of x2t⋅E.SSP flips a coins and chooses a random integer R∈0,2l/2−t⋅E. If s=1, SSP computes tmpPKiO=x2t⋅ERPKiOtmpSKiE=x2t⋅ERSKiE. If s=0, SSP computes tmpPKiO=−x2t⋅ERPKiOtmpSKiO=−x2t⋅ERSKiE. Then, SSP sends the computed data to FNi. FNi decrypts and obtains tmp. Let judge=1 if tmp<0 else judge=0. FNi sends judgePKsO. If s=1 and judge=1, SSP will know x2t⋅E<0. If s=1 and judge=0, x2t⋅E≥0. If s=0 and judge=1, x2t⋅E≥0. If s=0 and judge=0, x2t⋅E<0.Converting Cloud-ElGamal to OUFor this building block, we cite building block 7 of [5].Converting OU:For this building block, we cite building block 8 of [5].


## 5.3.1. Precision Control

In the machine learning training process, many data are floating-point numbers. Therefore, it is necessary to convert floating-point numbers to integers before encrypting. Generally, the conversion method is to multiply the floating-point number by2E or 10E [26], E is the precision. This method will significantly expand the original data. However, the plaintext space of the encryption algorithm is limited. Using the expanded data for homomorphic operation will lead to the problem of plaintext overflow and the pression of the data will change. For example, x2E⋅y2E⟶xy22E, xy22E+z2E⟶xy2E+z2E≠xy+z2E.To solve this problem, we propose a method called precision control. We express the encrypted data asc,t. The c is the ciphertext data and the t is the precision of the data. For example, xy⋅22E is expressed as xy⋅22E,2. Through the method, we can know the current precision of the data. When E⋅t+1≥l, we need to reduce the precision bits of encrypted data to avoid plaintext overflow. In order to better control the precision, we require that any participant must set t=1 before encrypting the data. The detailed method is as follows and is described in Algorithm 1.(1)
SSP chooses a random integerR,lR<E. For x2t⋅EPKiO,t, SSP sends x2t⋅E+R2t⋅E,t to FNi. For x2t⋅ESKiE,t, SSP sends x2t⋅ERSKiE,t to FNi.(2)
Forx2t⋅E+R2t⋅EPKiO,t,FNi computes x2E+R2E=x2t⋅E+R2t⋅E/2t−1E. For x2t⋅ERSKiE,t, FNi computes x2ER=x2t⋅ER/2t−1E. Then, FNi encrypts x2E+R2E (or x2ER) and sends x2E+R2EPKiE,1 (or x2ERSKiE,1) to SSP.(3)
Forx2E+R2EPKiE,1, SSP computes x2EPKiE,1←x2E+R2EPKiE⋅Ni−R2EPKiE. For x2ERSKiE,1, SSP computes the inverse R−1 of R modulo pie and obtains x2ESKiE,1←x2ERSKiE⋅R−1.Algorithm 1: Precision control.
Input:x2t⋅EPKiO,t or x2t⋅ESKiE,t,i∈1,mOutput:x2EPKiO,1 or x2ESKiE,1SSP:(1)
Sendx2t⋅E+R2t⋅EPKiO,torx2t⋅ERSKiE,t to FNiFNi:(2)
Decryptx2t⋅E+R2t⋅EPKiO,torx2t⋅ERSKiE,t(3)
Computex2E+R2E=x2t⋅E+R2t⋅E/2t−1Eorx2ER=x2t⋅ER/2t−1E and encrypt x2E(4)
Sendx2E+R2EPKiO,1 or x2ERSKiE,1 to SSPSSP:(5)
Computex2EPKiO,1←x2E+R2EPKiO⋅Ni−R2EPKiEorx2ESKiE,1←x2ERSKiE⋅R−1


## 5.3.2. Secure Addition

Given two ciphertext data encrypted withPKiO, x2t⋅EPKiO,t and y2t⋅EPKiO,t. SSP will obtain x+y2t⋅EPKiO,t=x2t⋅EPKiO⋅y2t⋅EPKiO,t.


## 5.3.3. Secure Subtraction

Given two ciphertext data encrypted withPKiO, x2t⋅EPKiO,t and y2t⋅EPKiO,t. SSP will obtain x−y2t⋅EPKiO,t=x2t⋅EPKiO⋅y2t⋅EPKiONi−1,t.


## 5.3.4. Secure Multiplication (OU)

Given a ciphertext data encrypted withPKiO, x2t⋅EPKiO,t and a plaintext data of SSP, y⋅2E. SSP will obtain xy2t+1EPKiO,t+1=x2t⋅EPKiOy⋅2E,t+1.


## 5.3.5. Secure Multiplication (Cloud-ElGamal):

Given two ciphertext data encrypted withSKiE, x2t1⋅ESKiE,t1 and y2t2⋅ESKiE,t2. SSP will obtain xy2t1+t2ESKiE,t1+t2=x2t1⋅ESKiE⋅y2t2⋅ESKiE,t1+t2.


## 5.3.6. Secure Division:

The algorithm is inspired by [27]. Given two ciphertext data encrypted with PKiO, x2t1⋅EPKiO,t1 and y2t2⋅EPKiO,t2. SSP will obtain x/y22EPKiO,2. The computation process is as follows:(1)
SSP chooses two random integersR1,R2∈0,2l/2 and computes(6)x2t1⋅ER1PKiO=x2t1⋅EPKiOR1,y2t2⋅ER2PKiO=y2t2⋅EPKiOR2.Then, SSP sends them toFNi(2)
FNi decrypts x2t1⋅ER1PKiO, y2t2⋅ER2PKiO and obtains xR1/yR2. Sending xR1/yR22EPKiO,1 to SSP(3)
SSP computesx/y22EPKiO,2←xR1/yR22EPKiOR2/R12ESecure power computationGiven a ciphertext data encrypted withSKiE, x2t⋅ESKiE,t and a plaintext data of SSP, y⋅2E. SSP will obtain xy2E,1. The computation process is as follows:(1)
SSP computes(7)xy2E2y2Et⋅ESKiE=aiky2Emodpie,yikx2t⋅E)y2Emodpie.Choosing a random integerR1∈0,2l/3, R2,R2′=R22E∈0,2l/3 and computes xy2E2y2Et⋅ER1SKiE=xy2E2y2Et⋅ESKiE⋅R1, 2y2Et⋅ER1R2′. Then, sending xy2E2y2Et⋅ER1SKiE and 2y2Et⋅ER1R2′SKiE to FNi.(2)
FNi decrypts and computes(8)xyR2=2Exy2E2y2Et⋅ER12y2Et⋅ER1R2′.Then, sendingxy/R22ESKiE,1 to SSP(3)
SSP computesxy2ESKiE,1←xy/R22ESKiE⋅R2


## 5.3.7. Secure Inner Product

Given an encrypted vector ofFNi, xPKiO=x12t⋅EPKiO,t,x22t⋅EPKiO,t,…,xd2t⋅EPKiO,t and a plaintext vector of SSP, y=y12E,y22E,…,yd2E. SSP will obtain(9)xy2t+1EPKiO,t+1=∏j=1dxj2t⋅EPKiOyj2E,t+1.


## 5.3.8. Secure Summation

SSP wants to obtain the summation of∑j=1msi and si belongs to FNi. The process is described in Algorithm 2.(1)
FNi encrypts si′=si+αi with PKsO and sends si′PKsO to SSP(2)
SSP computess′PKsO=∏i=1msi′PKsO and decrypts s′PKsO. Then, SSP can obtain s=s′−αAlgorithm 2: Secure summation.
Input:si,i=1,2,…,mOutput:sFNs:fori=1⟶m:(1)FNi computes si′=si+αi(2) Encryptsi′ with PKsO and send si′PKsO to SSPendforSSP:(3) Computes′PKsO=∏i=1msi′PKsO(4) Decrypts′PKsO and compute s=s′−α


## 5.3.9. Secure Sigmoid Function:

To complete the nonlinear computation of the sigmoid function, we propose an algorithm called Secure Sigmoid Function. The process is described in Algorithm3.(1)
FNi has ex=ex1,ex2,…,exd. FNi encrypts exj2E with SKiE and sends it to SSP(2)
SSP hasw=w1,w2,…,wd. SSP computes(10)ewjxj2E2wj2EESKiE=aik)wj2Emodpie,yikexj2E)wj2Emodpie2wj2EE=2Ewj2E,ewx⋅2E2ws2EESKiE=∏j=1dewjxj2E2wj2EESKiE∏j=1d2wj2EE,wherews=w1+w2+⋯+wd. Then, SSP chooses two random integers R1,R2, R1,R2′=R22E∈0,2l/4 and sends ewx⋅2E2ws2EER1SKiE, 2ws2EER1R2′ to FNi(3)
FNi decrypts and computes(11)ewxR2=2Eewx⋅2E2ws2EER12ws2EER1R2′.Then, sendingewx/R22EPKiO to SSP(4)
SSP chooses two random integersR3,R4∈0,2l/4 computes as follows:(12)ewx2ER3PKiO=ewxR22EPKiOR2R3,ewx+12ER3R4PKiO=ewx2EPKiO⋅2EPKiOR3R4.Then, SSP sendsewx2ER3PKiO and ewx+12ER3R4PKiO to FNi(5)
FNi decrypts ewx2ER3PKiO, ewx+12ER3R4PKiO and obtains(13)xjewxewx+1R4=xjewxR3ewx+1R3R4.Then, SSP sendsxjewx/ewx+1R42EPKiO to SSP(6)
SSP computes(14)xjewxewx+12EPKiO,1←xjewxewx+1R42EPKiOR4.Algorithm 3: Secure sigmoid function.
Input:ex=ex1,ex2,…,exd,w=w1,w2,…,wdOutput:ewxFNi:forj=1⟶d:(1)
Sendexj2ESKiE to SSPendforSSP:(2)
ewjxj2E2wj2EESKiE=aik)wj2Emodpie,yikexj2E)wj2Emodpie(3)
2wj2EE=2Ewj2E(4)
ewx⋅2E2ws2EESKiE=∏j=1dewjxj2E2wj2EESKiE(5)
2ws2EE=∏j=1d2wj2EE(6)
Choose two random integersR1,R2, R1,R2′=R22E∈0,2l/4(7)
Sendewx⋅2E2ws2EER1SKiE, 2ws2EER1R2′ to FNiFNi:(8)
Decrypt and computeewx/R2=2Eewx⋅2E2ws2EER1/2ws2EER1R2′(9)
Sendewx/R22EPKiO to SSPSSP:(10)
Choose two random integersR3,R4∈0,2l/4(11)
ewx2ER3PKiO=ewx/R22EPKiOR2R3(12)
ewx+12ER3R4PKiO=ewx2EPKiO⋅2EPKiOR3R4(13)
Sendewx2ER3PKiO and ewx+12ER3R4PKiO to FNiFNi:(14)
Decryptewx2ER3PKiO, ewx+12ER3R4PKiO(15)
xjewx/ewx+1R4=xjewxR3/ewx+1R3R4(16)
Sendxjewx/ewx+1R42EPKiO to SSPSSP:(17)
xjewx/ewx+12EPKiO,1←xjewx/ewx+1R42EPKiOR4


## 5.3.10. Secure Sign Computation

Given encrypted data that is computed by SSP,x2t⋅EPKiO,tx2t⋅ESKiE,t. FNi needs to know if x2t⋅E<0 but cannot know the value of x2t⋅E.SSP flips a coins and chooses a random integer R∈0,2l/2−t⋅E. If s=1, SSP computes tmpPKiO=x2t⋅ERPKiOtmpSKiE=x2t⋅ERSKiE. If s=0, SSP computes tmpPKiO=−x2t⋅ERPKiOtmpSKiO=−x2t⋅ERSKiE. Then, SSP sends the computed data to FNi. FNi decrypts and obtains tmp. Let judge=1 if tmp<0 else judge=0. FNi sends judgePKsO. If s=1 and judge=1, SSP will know x2t⋅E<0. If s=1 and judge=0, x2t⋅E≥0. If s=0 and judge=1, x2t⋅E≥0. If s=0 and judge=0, x2t⋅E<0.Converting Cloud-ElGamal to OUFor this building block, we cite building block 7 of [5].Converting OU:For this building block, we cite building block 8 of [5].


## 5.4. Privacy Preserving Machine Learning Training

In this section, we achieve four training protocols based on our proposed building blocks, which are popular machine learning models. We assume that the training data has been normalized before training.


## 5.5. Secure Logistic Regression (LR) Training

We use the stochastic gradient descent (SGD) algorithm to train a Logistic Regression model. SSP randomly selectsT data of FNs (all data of FNs have been numbered). The process is described in Algorithm 4.Algorithm 4: Secure logistic regression training.
Input: the selected data of FNs, iterationsT, learning rate LOutput: model parametersW=w1,w2,…,wdFNs:fori=1⟶m:(1)
FNi encrypts the selected data with PKiOSuch asxjj∈1,d, send xjy2EPKiO,1 to SSPendforSSP:forit=1⟶T:(2)
xjewx/ewx+12EPKiO,1←SecureSigmoidFunctionforj=1⟶d:(3)
SSP performsSecureAddition and SecureMultiplicationOU(4)
wj22EPKiO,2←wj22EPKiOxjewx/ewx+12EPKiONi−L2E⋅xjy2EPKiOL2E(5)
Converting OU:wj22EPKsO←wj22EPKiO(6)
wj2E=wj22E/2Eendforendfor


## 5.6. Secure SVM Training

For training a SVM model, we also use the SGD algorithm. The process is described in Algorithm5.Algorithm 5: Secure SVM training.
Input: the selected data of FNs,iterationsT, learning rate L, regularization parameter zOutput: model parametersW=w1,w2,…,wdFNs:fori=1⟶m:(1)
FNi encrypts the selected data with PKiOSuch asxjj∈1,d,sendxj2EPKiO,1, xjy2EPKiO,1 and y to SSPendforSSP:forit=1⟶T:(2)
wx22EPKiO,2←SecureInnerProductxPKiO,w(3)
t1PKiO=wx22EPKiO⋅Ni−1PKiOy=1t2PKiO=wx22EPKiONi−1⋅Ni−1PKiOy=−1(4)
SSP andFNi perform SecureSignComputationfort1PKiO, t2PKiOFNi:(5)
In the process ofSecureSignComputation:ify=1: FNi will judge t1PKiOelse: FNi will judge t2PKiOSSP:(7)
To computewj22EPKiO,2, SSP performsSecureAddition and SecureMultiplicationOUifs=1,judge=1ors=0,judge=0:forj=1⟶d:wj22EPKiO,2←wj22EPKiO⋅wj2EPKiONi−L⋅z2E⋅xjy2EPKiOL⋅z2EConverting OU:wj22EPKsO←wj22EPKiOwj2E=wj2E/2Eendforelse:forj=1⟶d:wj22EPKiO,2⟵wj22EPKiO⋅wj2EPKiONi−L⋅z2EConverting OU:wj22EPKsO⟵wj22EPKiOwj2E=wj2E/2Eendforendfor


## 5.7. Secure Naive Bayes (NB) Training

To train a naive Bayes model, the user needs to aggregate the class prior probabilityPy and the conditional probability Px|y from FNs. We assume that it is a binary classification problem. The process is described in Algorithm 6.Algorithm 6: Secure naive Bayes training.
Input: the data of FNsOutput: model parametersPy=1,Py=0,Pxi=0|y=0,Pxi=0|y=1,Pxi=1|y=0,Pxi=0|y=1FNs:fori=1⟶m:(1)
FNi computes Sy=1i+αi=∑j=1niyj+αi,Sxk=0|y=0i+αi,Sxk=0|y=1i+αi,k=1,2,…,dEncrypt them withPKsO and send them to SSPendforSSP:(2)
ComputeSy=1+αPKsO=∏i=1mSy=1i+αiPKsO,Sxk=0|y=0+αSxk=0|y=0+αSxk=0|y=0+αPKsO=∏i=1mSxk=0|y=0i+αiPKsOSxk=0|y=1+αSxk=0|y=1+αSxk=0|y=1+αPKsO=∏i=1mSxk=0|y=1i+αiPKsO(3)
Decrypt them withSKsOfork=1⟶d:(4)
Pxk=0|y=0=Sxk=0|y=0/n−Sy=1Pxk=0|y=1=Sxk=0|y=1/Sy=1Pxk=1|y=0=1−Pxk=0|y=0Pxk=1|y=1=1−Pxk=0|y=1endfor(5)
Py=1=Sy=1/n,Py=0=1−Py=1


## 5.8. Secure Deep Neural Network Training

The training process of deep neural networks contains nonlinear computations such as ReLU. To compute a nonlinear activation function, we will make an approximation method. The approximation method is proposed in [28] and the nonlinear activation functions will be converted to polynomial functions. Then, the training process can be completed with our proposed building blocks. In addition, we use the training method proposed in [29] to train the deep neural network models.


## 5.9. Privacy Preserving Machine Learning Model Updating

With the continuous increase of fog node data in the system or the addition of new fog nodes in the system, the quantity and quality of the overall data will be significantly improved. Therefore, it is very important to update the SSP’s trained model. For different types of machine learning models, we propose different updating methods. Specifically, we divide the machine learning model into the model trained by the gradient-descent method and the model trained by a nongradient-descent method. In order to prevent a differential attack, the model trained by the nongradient-descent method can be updated only when the data of the original fog nodes increases to more than 50%.A model trained by the gradient-descent method:For the model trained based on the gradient-descent method (e.g., LR and SVM), the updating process of the model is as follows:(1)
TA determines the fog nodes and SSP participating in the updating process and redistributes the system parameters for them (in the way of 5.1).(2)
FNs and SSP train a new model on the added data. SSP will obtain the new modelw′.(3)
Based on thew′ and the original model w, SSP will compute the updated model wi=n/n+n′wi+n′/n+n′wi′,i=1,2,…,d (n is the number of original data and n′ is the number of added data).


## 5.10. Model Trained by Nongradient-Descent Method

For the model trained by the nongradient-descent method (for example, naive Bayes), the updating process of the model is as follows:(1)
TA redistributes system parameters for all fog nodes and SSP to update the model (in the way of 5.1)(2)
FNs and SSPs use all the data to retrain a model. SSP takes the model as the updated model


## 6. Security Analysis

In this section, we make a security analysis of our proposed framework with the real and ideal paradigm and composition Theorem, like [27]. We use a simulator in the ideal world to simulate the view of an adversary (honest-but-curious) in the real world. We consider the adversaries are FNs and SSP, A=AFN1,AFN2,…,AFNm,ASSP. We will prove the security of our proposed basic building blocks.Theorem 1.
The precision control is secure against honest-but-curious adversariesA.Proof.
We construct two simulators based onFNi and SSP, SimFNi,SimSSP. The SimSSP computes x2t⋅E+R2t⋅E,t (or x2t⋅ERSKiE,t) and ASSP cannot distinguish the real execution from the ideal simulation due to the semantic security of the OU cryptosystem (or Cloud-ElGamal cryptosystem). For SimFNi, the view of AFNi is x2t⋅E+R2t⋅E (or x2t⋅ER). Because the statistics are distinguishable, AFNi cannot distinguish the real execution from the ideal simulation.Theorem 2.
The secure addition, secure subtraction, secure multiplication (OU), secure multiplication (Cloud-ElGamal), and secure inner product are secure against honest-but-curious adversariesA.Proof.
It should be noted that these basic building blocks are similar, so we will make a security analysis for secure addition as an example.
For secure addition, the simulatorSimSSP is the same as Theorem 1. Based on the semantic security of the OU cryptosystem, ASSP cannot distinguish x2t⋅EPKiO, y2t⋅EPKiO and x+y2t⋅EPKiO. The view of ASSP is indistinguishable in the real and ideal world.Theorem 3.
The secure division is secure against honest-but-curious adversariesA.Proof.
BecauseR1, R2 are random integers, x2t1⋅ER1 and y2t2⋅ER2 are statistics distinguishable for AFNi. The simulator SimSSP can only obtain x/y22EPKiO.Theorem 4.
The secure power computation is secure against honest-but-curious adversariesA.Proof.
BecauseR1, R2, R2′ are random integers, xy2E2y2Et⋅ER1, 2y2Et⋅ER1R2′ and xy/R2 are statistics distinguishable for AFNi. The simulator SimSSP can only obtain xy2ESKiE.Theorem 5.
The secure summation is secure against honest-but-curious adversariesA.Proof.
Thesi′ of FNi is computed through si+αi, so the ASSP cannot obtain the value of si based on the statistics distinguishable. At the same time, si′ is encrypted with PKsO. For s1′PKsO,s2′PKsO,…,sm′PKsO, the adversaries cannot distinguish the real execution from the ideal simulation due to the semantic security of the OU cryptosystem.Theorem 6.
The secure sigmoid function is secure against honest-but-curious adversariesA.Proof.
TheSimSSP performs computation on exj2ESKiE and wjj=1,2,…,d, which means that ASSP cannot obtain the value of ewx2E2ws2EESKiE. Because the statistics are distinguishable, SimFNi cannot obtain the value of ewx with ewx/R2 and xjewx/ewx+1R4. The SimSSP can only obtain xjewx/ewx+12EPKiO.Theorem 7.
The secure sign computation is secure against honest-but-curious adversariesA.Proof.
TheSimFNi can obtain x2t⋅ER or −x2t⋅ER through the building blocks. Based on the statistics distinguishable, AFNi cannot obtain the value of x.Theorem 8.
The secure machine learning training protocols are secure against honest-but-curious adversariesA.Proof.
We construct the machine learning training protocol through our designed building blocks in a modular way. The simulators areSimFN1,SimFN2,…,SimFNm,SimSSP, which is the same as Theorems 1–7. We have proved the security of the proposed building blocks, so the view of adversaries cannot distinguish the real execution from the ideal simulation.


## 7. Performance Evaluation

In this section, we evaluate our proposed ePMLF and compare it with Zhu et al. [5]. Zhu et al. [5] proposed a privacy-preserving ML training framework for the aggregation scenario. However, there are still some shortcomings to be improved, including the function of building blocks is not comprehensive and high communication and computation overhead, which makes the framework impractical. Our proposed framework effectively solves these problems. Our experimental environment is shown in Table 2.Table 2
Experimental environment.
Operating systemWindows 10CPUIntel(R) Core(TM)i7-10510U, 1.80 GHz, 2.30 GHzMemory8 GProgram languagePythonTo train LR, SVM, and NB, we implement our framework on three datasets of the UCI machine learning library, as shown in Table3.Table 3
Experiments datasets.
DatasetRecordsAttributesClassificationsDermatology366346Heart disease303135Breast cancer69992For the deep neural network, we will use the MNIST dataset to train a LeNet model [30]. The MNIST dataset contains 60,000 training samples and 10,000 testing samples.


### 7.1. Performance of Our Proposed Framework

#### 7.1.1. Building Blocks Evaluation

To test our proposed building blocks, we test three main building blocks, secure inner product, secure summation, and secure power computation. We set the different key lengths (256 bits, 512 bits, 1024 bits, and 2048 bits). The computation time results are shown in Figure3. It can be seen that the increase in key length will make the computation time longer. To balance security and efficiency, the key length is usually set to 1024 bits or 2048 bits.Figure 3
Computation time under different key lengths.Then, we evaluate the impact ofd on computation time for secure inner product and secure sigmoid function. Based on the above results, the key length will be set to 1024 bits. The results are shown in Figure 4. It can be seen that with the increase in d, the computation time of secure inner product and secure sigmoid function will increase.Figure 4
Computation time under differentd.


#### 7.1.2. Secure Machine Learning Training Analysis

In order to test the quality of the model trained using our framework, we test the accuracy of the trained model (LR, SVM, NB, and LeNet). The results are shown in Table4. From Table 4, it can be seen that the accuracy of the training model is high.Table 4
The accuracy of machine learning models.
Machine learning modelDatasetAccuracy (%)LRDermatology94.44Heart disease93.33Breast cancer94.16SVMDermatology98.61Heart disease93.33Breast cancer94.89NBBreast cancer97.81LeNetMNIST97.69


### 7.2. Comparative Analysis

In this section, we analyze the computation and communication overhead of our proposed framework. Then, we make a comparison with Zhu et al. [5]. According to [31], we know that the computation cost of an exponentiation operation is equal to 1.5l multiplication operations, where l is the length of ciphertexts. We assume that lo, le, lp and lr denote the ciphertext length of OU cryptosystem, Cloud-ElGamal cryptosystem, Paillier cryptosystem, and Cloud-RSA cryptosystem, respectively.


#### 7.2.1. Computation Complexity Analysis

In our proposed framework, we focus on the exponentiation operation and multiplication operation. The computation complexity of secure addition islo. The secure subtraction costs 2.5lo. For secure multiplication (OU) and secure multiplication (Cloud-ElGamal), they cost 1.5lo and 2le respectively. For secure division, the computation complexity is 9lo. For secure power computation, the computation complexity is 6.5le. For a secure inner product, the computation complexity is 2.5d⋅lo. For secure summation, the computation complexity is m⋅lo. The secure sigmoid function costs 5.5d⋅le+1.5le+10.5lo. The secure sign computation costs 1.5lo (or le). The comparison results are shown in Table 5. In our framework, the OU cryptosystem and Cloud-ElGamal cryptosystem are used to encrypt data. In [5], the Paillier cryptosystem and Cloud-RSA cryptosystem are used to encrypt data. It should be noted that lo<lp, le=lr and bi≫d. From Table 5, it can be seen that our proposed scheme has lower computation costs.Table 5
Comparison of computation complexity.
Building blocksOurs,k=1024bit[5], k=1024bitSecure additionlolpSecure subtraction2.5lo2.5lpSecure division9loNot supportSecure inner product2.5d⋅lo2.5d⋅lpSecure summationm⋅lo2.5m⋅lpSecure sigmoid function5.5d⋅le+1.5le+10.5lo∑i=1dbi+dlr+2.5lr+9lp


#### 7.2.2. Computation Overhead Analysis

We test the computation time of our framework and [5]. The key length of the cryptosystem is set to 1024 bits. The results are shown in Table 6. From Table 6, it can be seen that our framework is more efficient. Then, we compare the computation overhead of each participant with [5]. We assume that the SSP is the model owner and the FNs are the data owners. The comparison results are shown in Table 7 (m=2, d=2). From Table 7, the computation overhead is lower than [5] for SSP and FNs. We have obvious advantages to perform secure machine learning training.Table 6
Comparison of computation time.
Building blocksOurs,k=1024bits[5], k=1024bitsSecure addition0.019 ms0.08 msSecure subtraction42.6 ms80.5 msSecure division98.8 msNot supportSecure inner product10.1 ms79.6 msSecure summation181.8 ms1666.9 msSecure sigmoid function232.4 ms445.7 msTable 7
Comparison of computation overhead: model owner and data owners.
Building blocksModel owner (ms)Data owners (ms)Ours[5]Ours[5]Secure addition0.0190.0800Secure subtraction42.680.500Secure inner product10.179.600Secure summation11.931410.23133.64306.15Secure sigmoid function139.45231.6592.96190.67Exampleofsecuresigmoidfunction: The computation process of secure sigmoid function will cause the loss of accuracy. Therefore, it is very important to ensure accuracy. To prove the accuracy of the secure sigmoid function, we perform the building block by setting a case. We set ex=1.1,1.22 and w=1.31,2.42, which is the same as [5]. In the plaintext, the value ewx is 1.83. Our result is 1.81 and the result of [5] is 1.6588. It can be seen that our result is more accurate.


#### 7.2.3. Communication Analysis

We analyze the communication overhead and interactions of our building blocks and compare them with [5]. The secure addition, secure subtraction, secure multiplication (OU), secure multiplication (Cloud-ElGamal), and secure inner product are computed by the user, so the interactions of these building blocks are 0. For the secure division, the communication overhead is 3lo. For the secure summation, the communication overhead is m⋅lo and the interactions are m. For the secure sigmoid function, the communication overhead is d+1⋅le+4lo. The comparison results are shown in Table 8. It can be seen that the communication overhead of our proposed secure summation is much lower than [5].Table 8
Comparison of communication overhead.
Building blocksOurs[5]Secure addition00Secure subtraction00Secure division3loNot supportSecure inner product00Secure summationm⋅lo3m⋅lpSecure sigmoid functiond+1⋅le+4lod+1⋅lr+3lp


## 7.1. Performance of Our Proposed Framework

### 7.1.1. Building Blocks Evaluation

To test our proposed building blocks, we test three main building blocks, secure inner product, secure summation, and secure power computation. We set the different key lengths (256 bits, 512 bits, 1024 bits, and 2048 bits). The computation time results are shown in Figure3. It can be seen that the increase in key length will make the computation time longer. To balance security and efficiency, the key length is usually set to 1024 bits or 2048 bits.Figure 3
Computation time under different key lengths.Then, we evaluate the impact ofd on computation time for secure inner product and secure sigmoid function. Based on the above results, the key length will be set to 1024 bits. The results are shown in Figure 4. It can be seen that with the increase in d, the computation time of secure inner product and secure sigmoid function will increase.Figure 4
Computation time under differentd.


### 7.1.2. Secure Machine Learning Training Analysis

In order to test the quality of the model trained using our framework, we test the accuracy of the trained model (LR, SVM, NB, and LeNet). The results are shown in Table4. From Table 4, it can be seen that the accuracy of the training model is high.Table 4
The accuracy of machine learning models.
Machine learning modelDatasetAccuracy (%)LRDermatology94.44Heart disease93.33Breast cancer94.16SVMDermatology98.61Heart disease93.33Breast cancer94.89NBBreast cancer97.81LeNetMNIST97.69


## 7.1.1. Building Blocks Evaluation

To test our proposed building blocks, we test three main building blocks, secure inner product, secure summation, and secure power computation. We set the different key lengths (256 bits, 512 bits, 1024 bits, and 2048 bits). The computation time results are shown in Figure3. It can be seen that the increase in key length will make the computation time longer. To balance security and efficiency, the key length is usually set to 1024 bits or 2048 bits.Figure 3
Computation time under different key lengths.Then, we evaluate the impact ofd on computation time for secure inner product and secure sigmoid function. Based on the above results, the key length will be set to 1024 bits. The results are shown in Figure 4. It can be seen that with the increase in d, the computation time of secure inner product and secure sigmoid function will increase.Figure 4
Computation time under differentd.


## 7.1.2. Secure Machine Learning Training Analysis

In order to test the quality of the model trained using our framework, we test the accuracy of the trained model (LR, SVM, NB, and LeNet). The results are shown in Table4. From Table 4, it can be seen that the accuracy of the training model is high.Table 4
The accuracy of machine learning models.
Machine learning modelDatasetAccuracy (%)LRDermatology94.44Heart disease93.33Breast cancer94.16SVMDermatology98.61Heart disease93.33Breast cancer94.89NBBreast cancer97.81LeNetMNIST97.69


## 7.2. Comparative Analysis

In this section, we analyze the computation and communication overhead of our proposed framework. Then, we make a comparison with Zhu et al. [5]. According to [31], we know that the computation cost of an exponentiation operation is equal to 1.5l multiplication operations, where l is the length of ciphertexts. We assume that lo, le, lp and lr denote the ciphertext length of OU cryptosystem, Cloud-ElGamal cryptosystem, Paillier cryptosystem, and Cloud-RSA cryptosystem, respectively.


### 7.2.1. Computation Complexity Analysis

In our proposed framework, we focus on the exponentiation operation and multiplication operation. The computation complexity of secure addition islo. The secure subtraction costs 2.5lo. For secure multiplication (OU) and secure multiplication (Cloud-ElGamal), they cost 1.5lo and 2le respectively. For secure division, the computation complexity is 9lo. For secure power computation, the computation complexity is 6.5le. For a secure inner product, the computation complexity is 2.5d⋅lo. For secure summation, the computation complexity is m⋅lo. The secure sigmoid function costs 5.5d⋅le+1.5le+10.5lo. The secure sign computation costs 1.5lo (or le). The comparison results are shown in Table 5. In our framework, the OU cryptosystem and Cloud-ElGamal cryptosystem are used to encrypt data. In [5], the Paillier cryptosystem and Cloud-RSA cryptosystem are used to encrypt data. It should be noted that lo<lp, le=lr and bi≫d. From Table 5, it can be seen that our proposed scheme has lower computation costs.Table 5
Comparison of computation complexity.
Building blocksOurs,k=1024bit[5], k=1024bitSecure additionlolpSecure subtraction2.5lo2.5lpSecure division9loNot supportSecure inner product2.5d⋅lo2.5d⋅lpSecure summationm⋅lo2.5m⋅lpSecure sigmoid function5.5d⋅le+1.5le+10.5lo∑i=1dbi+dlr+2.5lr+9lp


### 7.2.2. Computation Overhead Analysis

We test the computation time of our framework and [5]. The key length of the cryptosystem is set to 1024 bits. The results are shown in Table 6. From Table 6, it can be seen that our framework is more efficient. Then, we compare the computation overhead of each participant with [5]. We assume that the SSP is the model owner and the FNs are the data owners. The comparison results are shown in Table 7 (m=2, d=2). From Table 7, the computation overhead is lower than [5] for SSP and FNs. We have obvious advantages to perform secure machine learning training.Table 6
Comparison of computation time.
Building blocksOurs,k=1024bits[5], k=1024bitsSecure addition0.019 ms0.08 msSecure subtraction42.6 ms80.5 msSecure division98.8 msNot supportSecure inner product10.1 ms79.6 msSecure summation181.8 ms1666.9 msSecure sigmoid function232.4 ms445.7 msTable 7
Comparison of computation overhead: model owner and data owners.
Building blocksModel owner (ms)Data owners (ms)Ours[5]Ours[5]Secure addition0.0190.0800Secure subtraction42.680.500Secure inner product10.179.600Secure summation11.931410.23133.64306.15Secure sigmoid function139.45231.6592.96190.67Exampleofsecuresigmoidfunction: The computation process of secure sigmoid function will cause the loss of accuracy. Therefore, it is very important to ensure accuracy. To prove the accuracy of the secure sigmoid function, we perform the building block by setting a case. We set ex=1.1,1.22 and w=1.31,2.42, which is the same as [5]. In the plaintext, the value ewx is 1.83. Our result is 1.81 and the result of [5] is 1.6588. It can be seen that our result is more accurate.


### 7.2.3. Communication Analysis

We analyze the communication overhead and interactions of our building blocks and compare them with [5]. The secure addition, secure subtraction, secure multiplication (OU), secure multiplication (Cloud-ElGamal), and secure inner product are computed by the user, so the interactions of these building blocks are 0. For the secure division, the communication overhead is 3lo. For the secure summation, the communication overhead is m⋅lo and the interactions are m. For the secure sigmoid function, the communication overhead is d+1⋅le+4lo. The comparison results are shown in Table 8. It can be seen that the communication overhead of our proposed secure summation is much lower than [5].Table 8
Comparison of communication overhead.
Building blocksOurs[5]Secure addition00Secure subtraction00Secure division3loNot supportSecure inner product00Secure summationm⋅lo3m⋅lpSecure sigmoid functiond+1⋅le+4lod+1⋅lr+3lp


## 7.2.1. Computation Complexity Analysis

In our proposed framework, we focus on the exponentiation operation and multiplication operation. The computation complexity of secure addition islo. The secure subtraction costs 2.5lo. For secure multiplication (OU) and secure multiplication (Cloud-ElGamal), they cost 1.5lo and 2le respectively. For secure division, the computation complexity is 9lo. For secure power computation, the computation complexity is 6.5le. For a secure inner product, the computation complexity is 2.5d⋅lo. For secure summation, the computation complexity is m⋅lo. The secure sigmoid function costs 5.5d⋅le+1.5le+10.5lo. The secure sign computation costs 1.5lo (or le). The comparison results are shown in Table 5. In our framework, the OU cryptosystem and Cloud-ElGamal cryptosystem are used to encrypt data. In [5], the Paillier cryptosystem and Cloud-RSA cryptosystem are used to encrypt data. It should be noted that lo<lp, le=lr and bi≫d. From Table 5, it can be seen that our proposed scheme has lower computation costs.Table 5
Comparison of computation complexity.
Building blocksOurs,k=1024bit[5], k=1024bitSecure additionlolpSecure subtraction2.5lo2.5lpSecure division9loNot supportSecure inner product2.5d⋅lo2.5d⋅lpSecure summationm⋅lo2.5m⋅lpSecure sigmoid function5.5d⋅le+1.5le+10.5lo∑i=1dbi+dlr+2.5lr+9lp


## 7.2.2. Computation Overhead Analysis

We test the computation time of our framework and [5]. The key length of the cryptosystem is set to 1024 bits. The results are shown in Table 6. From Table 6, it can be seen that our framework is more efficient. Then, we compare the computation overhead of each participant with [5]. We assume that the SSP is the model owner and the FNs are the data owners. The comparison results are shown in Table 7 (m=2, d=2). From Table 7, the computation overhead is lower than [5] for SSP and FNs. We have obvious advantages to perform secure machine learning training.Table 6
Comparison of computation time.
Building blocksOurs,k=1024bits[5], k=1024bitsSecure addition0.019 ms0.08 msSecure subtraction42.6 ms80.5 msSecure division98.8 msNot supportSecure inner product10.1 ms79.6 msSecure summation181.8 ms1666.9 msSecure sigmoid function232.4 ms445.7 msTable 7
Comparison of computation overhead: model owner and data owners.
Building blocksModel owner (ms)Data owners (ms)Ours[5]Ours[5]Secure addition0.0190.0800Secure subtraction42.680.500Secure inner product10.179.600Secure summation11.931410.23133.64306.15Secure sigmoid function139.45231.6592.96190.67Exampleofsecuresigmoidfunction: The computation process of secure sigmoid function will cause the loss of accuracy. Therefore, it is very important to ensure accuracy. To prove the accuracy of the secure sigmoid function, we perform the building block by setting a case. We set ex=1.1,1.22 and w=1.31,2.42, which is the same as [5]. In the plaintext, the value ewx is 1.83. Our result is 1.81 and the result of [5] is 1.6588. It can be seen that our result is more accurate.


## 7.2.3. Communication Analysis

We analyze the communication overhead and interactions of our building blocks and compare them with [5]. The secure addition, secure subtraction, secure multiplication (OU), secure multiplication (Cloud-ElGamal), and secure inner product are computed by the user, so the interactions of these building blocks are 0. For the secure division, the communication overhead is 3lo. For the secure summation, the communication overhead is m⋅lo and the interactions are m. For the secure sigmoid function, the communication overhead is d+1⋅le+4lo. The comparison results are shown in Table 8. It can be seen that the communication overhead of our proposed secure summation is much lower than [5].Table 8
Comparison of communication overhead.
Building blocksOurs[5]Secure addition00Secure subtraction00Secure division3loNot supportSecure inner product00Secure summationm⋅lo3m⋅lpSecure sigmoid functiond+1⋅le+4lod+1⋅lr+3lp


## 8. Conclusion

In this paper, we propose a privacy-preserving machine learning framework, including secure training data normalization, model training, and model updating. Based on our proposed framework, SSP can train different machine learning models. The trained models have high accuracy. Compared with the existing scheme, our proposed framework significantly reduces the computation and communication overhead. In future research, we will focus on the efficiency and robustness of the privacy-preserving machine learning training framework.



---

*Source: 8292559-2023-02-27.xml*