# SecureBP from Homomorphic Encryption

Authors: Qinju Liu; Xianhui Lu; Fucai Luo; Shuai Zhou; Jingnan He; Kunpeng Wang
Journal: Security and Communication Networks
 (2020)

---

## Abstract

We present a secure backpropagation neural network training model (SecureBP), which allows a neural network to be trained while retaining the confidentiality of the training data, based on the homomorphic encryption scheme. We make two contributions. The first one is to introduce a method to find a more accurate and numerically stable polynomial approximation of functions in a certain interval. The second one is to find a strategy of refreshing ciphertext during training, which keeps the order of magnitude of noise atO˜e33.


---

## Body

## 1. Introduction

Driven by massive amounts of data and the high scalability, versatility, and high efficiency of cloud computing, modern machine learning (ML) has been widely used in many fields, including health care, military, and finance [1–3]. These fields often contain a large amount of sensitive data, so how to protect the data privacy while using them becomes a very important problem. At present, there exist various approaches that can be used to protect data privacy. Differential privacy (DP), secure multiparty computation (MPC), and homomorphic encryption (HE) are the most widely used methods for this problem.DP allows one to control the amount of information leaked from an individual record in a dataset. By using DP, one can ensure privacy for any entity whose information is contained in the dataset as well as to create models that do not leak this information about the data they were trained on. Therefore, DP is mainly used in the training process. However, we are more concerned about how to use cryptographic methods to protect data privacy.Most MPC methods establish a communication protocol among the parties involved such that if the parties follow the protocol, then they will end with the desired results while protecting the security and privacy of their respective assets [4–7]. However, due to the large scale of data used in machine learning, the communication cost of MPC is very high.HE is also another major method to protect data privacy, which allows us to perform certain arithmetic operations on encrypted data without decryption. Fully homomorphic encryption (FHE) (it allows us to perform arbitrarily complex and efficiently computable evaluations over encrypted data without decrypting them) was originally introduced by Rivest et al. in 1978 [8]. But it had been an open problem until Gentry presented the first plausible candidate FHE construction based on ideal lattices in 2009 [9]. Since then, a series of works [10–15] have been proposed to improve the security assumptions and efficiency of FHE, following Gentry’s blueprint. Currently, some public libraries are available (Table 1), namely, HElib [16] and SEAL [17] based on BGV scheme [18] and FHEW [19] and TFHE [20, 21] based on GSW scheme [14] and HEAAN [22] based on CKKS scheme [23]. The BGV-based schemes can handle a lot of bits at the same time, so they can pack and batch many operations in the SIMD manner. However, the set of operations that are efficient with BGV depends on the section of the parameter set. The GSW-based schemes can use Boolean circuits to deal with nonlinear operations quickly, but their computational efficiency of arithmetic operations is relatively low. The CKKS-based schemes can perform efficient approximate arithmetic operations on encrypted data by introducing a novel encoding technique and a fast rescale operation, but they cannot deal with nonpolynomial operations. It is widely used in machine learning due to its high efficiency in arithmetic operations (which is why we chose the CKKS scheme for our SecureBP model).Table 1
Outstanding performance of HE schemes.
Based schemeLibraryPlaintextOperationBGVHElib, SEALFinite field packingAddition, multiplicationGSWFHEW, TFHEBinary stringLook-up tableCKKSHEAANReal/complex packing numbersFixed-point arithmeticTwo important use-cases for machine learning models are predictions-as-a-service (PaaS) setting and training-as-a-service (TaaS) setting. In the PaaS setting, a large organization (or the cloud) uses its proprietary data to train machine learning models. The organization now hopes to monetize the model by deploying services that allow users to upload their inputs and receive predictions for price. In the TaaS setting, the organization makes profits by deploying services that allow users to upload their encrypted inputs and receive the encrypted machine learning model. Moreover, in this setting, since the process of training an encrypted model is time- and resource-consuming, the techniques and proprietary tools for the training algorithm are often considered critical intellectual property by its owner, who is typically not willing to share them.BP [24] is one of the most classical and widely used neural network models. It is more powerful than linear regression and logistic regression models. Moreover, the BP network already has the basic module of deep neural network (DNN); in other words, the BP network is the cornerstone of DNN. Therefore, when we study the data privacy protection of machine learning, it is appropriate to take the BP network model as the breakthrough point.


### 1.1. Our Contributions

In this paper, we present a secure backpropagation neural network model (SecureBP) based on HE. In this model, in a setup phase, the data owner (user) encrypts his data and sends them to the cloud. In the computation phase, the cloud can train the model on the encrypted data without learning any information beyond the ciphertext of data. Technically, we have two main contributions: a more accurate polynomial approximation technique and a lightweight interactive scheme to refresh ciphertexts during training.We focus on the TaaS setting in this paper and we choose HE (i.e., CKKS scheme) as the method to protect user’s data. For clarity, let us review the technical challenges and difficulties of using HE for the BP network in the TaaS setting. Firstly, in the BP network, each node is activated before output by an activation function, which is usually selected by nonpolynomial functions, such as sigmoid, tanget-hyperbolic (tanh), or rectified linear unit (ReLU). However, most existing HE schemes is that they only support polynomial arithmetic operations. The evaluation of the activation function is an obstacle for the homomorphic implementation of the BP network since it cannot be expressed as a polynomial. In addition, in order to ensure security, HE introduces some noise in encryption, and the noise increases as the homomorphic computation proceeds. When the noise reaches a certain threshold, the decryption error will occur. Therefore, in view of the abovementioned technical difficulties, we make the following two contributions.The first contribution is that by using Chebyshev polynomials (in fact, several studies have suggested this approach, but none have examined it in detail), we introduce a more accurate polynomial approximationLnx of sigmoid function for a certain interval. Compared with Taylor polynomials, our method causes more similarities of derivatives with the sigmoid function (see Section 3.1).The second contribution is that we propose a lightweight interaction protocol, which is a novel strategy to refresh ciphertext during training. The trivial way to deal with the growing noise is bootstrapping. However, bootstrapping comes with high computational overhead. To avoid costly bootstrapping of HE, we present the lightweight interaction protocol during training. By this method, on the one hand, no technical information of the cloud training model is provided to the user. On the other hand, the noise of weight ciphertext grows linearly after it grows to a certain value.Now that the basic ingredients are in place, we construct our SecureBP network. To demonstrate the feasibility of our SecureBP, we estimate its performance on three datasets: Iris dataset, Diabetes dataset and Sonar dataset (see Table2), which are from the University of California at Irvine (UCI) dataset repository [25].Table 2
Performance of SecureBP in time and accuracy.
DatasetAccuracy (%)Time (ms)Iris79.67632.7Diabetes65.19962.1Sonar82.232.0993 × 108


### 1.2. Related Work

Before the current work, there have been some researches on privacy-preserving machine learning algorithm [26–29]. These papers propose solutions based on MPC and HE techniques (see Table 3), but they appear to incur some problems.Table 3
Research works in secure machine learning.
SettingPrior workProblemActivationTechniquePaaSDeepSecure [29]DNNReLU, ReLU, sigmoidMPCGazelle [27]CNNReLUMPC, HECryptoNets [30]CNNSquare functionSHEFHE-DiNN [26]DiNNSignFHEChameleon [28]DNNReLU, sigmoidMPC, HETaaSSecureML [27]LR, NNSigmoid, SoftmaxMPC[31]LRLeast squares approximationHEOursBPL3xHEPrivacy-preserving machine learning via MPC provides a promising solution by allowing different parties to train various models on their joint data without revealing any information beyond the outcome. They require interactivity between the party that holds the data and the party that performs the blind classification. Even though practical performances of MPC-based solutions have been impressive compared to FHE-based solutions, they incur other issues such as network latency and high bandwidth usage. Because of these downsides, HE-based solutions seem more scalable for real-life applications.Privacy-preserving machine learning based on HE is more challenging. As we mentioned before, the standard activation function is a challenge in applying HE to the machine learning algorithm. Faced with this challenge, Ran Gilad-Bachrach et al. [30] propose a solution (CryptoNets) where instead of standard activation function, they use a square function. Homomorphic computation depends on the total number of levels required to implement the network and results in a relatively high computational overhead which bounds CryptoNets practicability in resource-limited settings where the data owners have severe computational constraints. Moreover, the inherent limitation of most existing HE constructions is that they only support the arithmetic operations over modular spaces. Therefore, their approaches required the size of parameter for real number operations (i.e., no modular reduction over plaintext space) which is too large to be practically implemented.


### 1.3. Organization

Section2 briefly introduces some notations and reviews the framework of BP. Section 3 describes our SecureBP model. In section 4, we estimate our model and discuss the estimation and implementation results.


## 1.1. Our Contributions

In this paper, we present a secure backpropagation neural network model (SecureBP) based on HE. In this model, in a setup phase, the data owner (user) encrypts his data and sends them to the cloud. In the computation phase, the cloud can train the model on the encrypted data without learning any information beyond the ciphertext of data. Technically, we have two main contributions: a more accurate polynomial approximation technique and a lightweight interactive scheme to refresh ciphertexts during training.We focus on the TaaS setting in this paper and we choose HE (i.e., CKKS scheme) as the method to protect user’s data. For clarity, let us review the technical challenges and difficulties of using HE for the BP network in the TaaS setting. Firstly, in the BP network, each node is activated before output by an activation function, which is usually selected by nonpolynomial functions, such as sigmoid, tanget-hyperbolic (tanh), or rectified linear unit (ReLU). However, most existing HE schemes is that they only support polynomial arithmetic operations. The evaluation of the activation function is an obstacle for the homomorphic implementation of the BP network since it cannot be expressed as a polynomial. In addition, in order to ensure security, HE introduces some noise in encryption, and the noise increases as the homomorphic computation proceeds. When the noise reaches a certain threshold, the decryption error will occur. Therefore, in view of the abovementioned technical difficulties, we make the following two contributions.The first contribution is that by using Chebyshev polynomials (in fact, several studies have suggested this approach, but none have examined it in detail), we introduce a more accurate polynomial approximationLnx of sigmoid function for a certain interval. Compared with Taylor polynomials, our method causes more similarities of derivatives with the sigmoid function (see Section 3.1).The second contribution is that we propose a lightweight interaction protocol, which is a novel strategy to refresh ciphertext during training. The trivial way to deal with the growing noise is bootstrapping. However, bootstrapping comes with high computational overhead. To avoid costly bootstrapping of HE, we present the lightweight interaction protocol during training. By this method, on the one hand, no technical information of the cloud training model is provided to the user. On the other hand, the noise of weight ciphertext grows linearly after it grows to a certain value.Now that the basic ingredients are in place, we construct our SecureBP network. To demonstrate the feasibility of our SecureBP, we estimate its performance on three datasets: Iris dataset, Diabetes dataset and Sonar dataset (see Table2), which are from the University of California at Irvine (UCI) dataset repository [25].Table 2
Performance of SecureBP in time and accuracy.
DatasetAccuracy (%)Time (ms)Iris79.67632.7Diabetes65.19962.1Sonar82.232.0993 × 108


## 1.2. Related Work

Before the current work, there have been some researches on privacy-preserving machine learning algorithm [26–29]. These papers propose solutions based on MPC and HE techniques (see Table 3), but they appear to incur some problems.Table 3
Research works in secure machine learning.
SettingPrior workProblemActivationTechniquePaaSDeepSecure [29]DNNReLU, ReLU, sigmoidMPCGazelle [27]CNNReLUMPC, HECryptoNets [30]CNNSquare functionSHEFHE-DiNN [26]DiNNSignFHEChameleon [28]DNNReLU, sigmoidMPC, HETaaSSecureML [27]LR, NNSigmoid, SoftmaxMPC[31]LRLeast squares approximationHEOursBPL3xHEPrivacy-preserving machine learning via MPC provides a promising solution by allowing different parties to train various models on their joint data without revealing any information beyond the outcome. They require interactivity between the party that holds the data and the party that performs the blind classification. Even though practical performances of MPC-based solutions have been impressive compared to FHE-based solutions, they incur other issues such as network latency and high bandwidth usage. Because of these downsides, HE-based solutions seem more scalable for real-life applications.Privacy-preserving machine learning based on HE is more challenging. As we mentioned before, the standard activation function is a challenge in applying HE to the machine learning algorithm. Faced with this challenge, Ran Gilad-Bachrach et al. [30] propose a solution (CryptoNets) where instead of standard activation function, they use a square function. Homomorphic computation depends on the total number of levels required to implement the network and results in a relatively high computational overhead which bounds CryptoNets practicability in resource-limited settings where the data owners have severe computational constraints. Moreover, the inherent limitation of most existing HE constructions is that they only support the arithmetic operations over modular spaces. Therefore, their approaches required the size of parameter for real number operations (i.e., no modular reduction over plaintext space) which is too large to be practically implemented.


## 1.3. Organization

Section2 briefly introduces some notations and reviews the framework of BP. Section 3 describes our SecureBP model. In section 4, we estimate our model and discuss the estimation and implementation results.


## 2. Preliminaries

### 2.1. Notations

All logarithms are base 2 unless otherwise indicated. During homomorphic operations, we use⊗ to denote the multiplication between ciphertexts; ⊕ denotes the addition between ciphertexts and ⊙ denotes the scalar multiplication between a constant and a ciphertext.Next, we introduce some signs used in the BP network:(i)
xii∈m, the input value.(ii)
hjj∈z, the output value of hidden layer.(iii)
Okk∈d, the output value of output layer.(iv)
Wj,ihj∈z,i∈m, the weight connecting the hidden-layer j-th node and the input-layer i-th node.(v)
Wk,jOk∈d,j∈z, the weight connecting the output-layer k-th node and the hidden-layer j-th node.(vi)
bjhj∈z, the bias of hidden-layer j-th node.(vii)
bkOk∈d, the bias of output-layer k-th node.(viii)
L, the learning rate.


### 2.2. The Framework of BP

In this subsection, we give a brief review of one version of the BP network. For ease of presentation, in this paper, we only consider a neural network of three layers (input layer, hidden layer, output layer). It is trivial to extend our work to the multilayers network. This configuration can be seen from Figure1.Figure 1
A conventional BP network with three layers.In the BP algorithm, there is one forward phase and one backward phase during each iteration. Then, the whole BP algorithm can be described in Algorithm1.Algorithm 1: The conventional BP training algorithm.
(1) Set the number of iterations, weight matrix W, and bias vector b to small random initial values.(2) Repeat(3) Forward phase: beginning with the input nodes, compute weighted sums, and activation function for all nodes;(4) Backward phase: compute gradients for all nodes starting from output nodes;(5) Adjust the weights and biases(6) until the number of iterations reaches a preset value.The forward phase starts from the input layer and approaches the output layer. During this phase, weighted sums and activations are computed for every node in each layer using the activation function, which is normally the sigmoid function. That is,(1)hj=σ∑i=1mWj,ih⋅xi+bjh,in hidden layer,Ok=σ∑j=1zWk,jO⋅hj+bkO,in output layer.The backward phase starts from the output layer and descends toward the bottom layer (i.e., the input layer) of the network to compute gradients. Finally, we need to update the weights (Wj,ihWk,jO) and biases (bjh, bkO) using the computed gradients. The rules for updating are as follows:(2)Wk,jO=Wk,jO+l⋅ErrkO⋅hj,bkO=bkO+l⋅ErrkO,Wj,ih=Wj,ih+l⋅Errjh⋅xi,bjh=bjh+l⋅Errjh,where ErrkO=Ok1−Oktk−Ok and Errjh=hj1−hj∑kErrkOWk,jO.


## 2.1. Notations

All logarithms are base 2 unless otherwise indicated. During homomorphic operations, we use⊗ to denote the multiplication between ciphertexts; ⊕ denotes the addition between ciphertexts and ⊙ denotes the scalar multiplication between a constant and a ciphertext.Next, we introduce some signs used in the BP network:(i)
xii∈m, the input value.(ii)
hjj∈z, the output value of hidden layer.(iii)
Okk∈d, the output value of output layer.(iv)
Wj,ihj∈z,i∈m, the weight connecting the hidden-layer j-th node and the input-layer i-th node.(v)
Wk,jOk∈d,j∈z, the weight connecting the output-layer k-th node and the hidden-layer j-th node.(vi)
bjhj∈z, the bias of hidden-layer j-th node.(vii)
bkOk∈d, the bias of output-layer k-th node.(viii)
L, the learning rate.


## 2.2. The Framework of BP

In this subsection, we give a brief review of one version of the BP network. For ease of presentation, in this paper, we only consider a neural network of three layers (input layer, hidden layer, output layer). It is trivial to extend our work to the multilayers network. This configuration can be seen from Figure1.Figure 1
A conventional BP network with three layers.In the BP algorithm, there is one forward phase and one backward phase during each iteration. Then, the whole BP algorithm can be described in Algorithm1.Algorithm 1: The conventional BP training algorithm.
(1) Set the number of iterations, weight matrix W, and bias vector b to small random initial values.(2) Repeat(3) Forward phase: beginning with the input nodes, compute weighted sums, and activation function for all nodes;(4) Backward phase: compute gradients for all nodes starting from output nodes;(5) Adjust the weights and biases(6) until the number of iterations reaches a preset value.The forward phase starts from the input layer and approaches the output layer. During this phase, weighted sums and activations are computed for every node in each layer using the activation function, which is normally the sigmoid function. That is,(1)hj=σ∑i=1mWj,ih⋅xi+bjh,in hidden layer,Ok=σ∑j=1zWk,jO⋅hj+bkO,in output layer.The backward phase starts from the output layer and descends toward the bottom layer (i.e., the input layer) of the network to compute gradients. Finally, we need to update the weights (Wj,ihWk,jO) and biases (bjh, bkO) using the computed gradients. The rules for updating are as follows:(2)Wk,jO=Wk,jO+l⋅ErrkO⋅hj,bkO=bkO+l⋅ErrkO,Wj,ih=Wj,ih+l⋅Errjh⋅xi,bjh=bjh+l⋅Errjh,where ErrkO=Ok1−Oktk−Ok and Errjh=hj1−hj∑kErrkOWk,jO.


## 3. SecureBP Based on CKKS Scheme

In this section, we explain how to securely train the BP network model using the CKKS scheme.


### 3.1. A Decent Polynomial Approximation

In the preceding update formula, except for activation function inside neurons (i.e., sigmoid functionσx=1/1+e−x), all other operations in BP network are addition and multiplication, so they can be implemented over encrypted data. One limitation of the existing HE schemes is that they only support polynomial arithmetic operations. The evaluation of the activation function is an obstacle for the implementation of the BP network since it cannot be expressed as a polynomial. Hence, in order to operate a complete BP neural network over encrypted data, we replace the sigmoid function with polynomial approximations that are compatible with practical HE schemes.Actually, the Taylor polynomialsTdx=∑k=0dfx0/k!xk have been commonly used for approximation of the sigmoid function [32–34]:(3)σx=12+14x−148x3+1480x5+Ox7.However, we observe that the size of error grows rapidly asx increases. Besides, in order to guarantee the accuracy of the BP network, we have to use a higher degree Taylor polynomial, but it requires too many homomorphic multiplications to be practically implemented. In summary, although Taylor expansions are more convenient and easier to compute, the accuracy of estimation is not always consistent because it is a local approximation near a certain point. Therefore, we introduce another good candidate for approximation with better approximation ability to replace the sigmoid function: optimal and uniform polynomial approximation of σx. Not exactly, we find a polynomial function Lnx that minimizes the absolute value of the error between σx and Lnx within a given interval.The Chebyshev polynomials are used to construct the optimal uniform approximation polynomialsLnx. The Chebyshev polynomials Cnx can be simply defined as for −1≤x≤1,(4)Cnx=cosn⋅arccosx,n=0,1,2,…From the abovementioned definition, we can get two important properties of Chebyshev polynomials. The first property is that we can get a recurrence of Chebyshev polynomialsC0x=1,C1x=x,Cn+1x=2xCnx−Cn−1x. The second is that the Chebyshev polynomial Cnx has n different zero points on the interval −1,1, i.e., xk=cos2k−1π/2n,k=1,2,…,n. Then, we can get the important theorem in polynomial approximation as follows.Theorem 1.
Letfx be a continuous differentiable function on interval −1,1, Lnx be the interpolation polynomial, and its interpolation nodes x0,⋯,xn are the zero points of Chebychev polynomial Cn+1x, then Lnx is the optimal and uniform polynomial approximation of fx on interval −1,1, and(5)max−1≤x≤1fx−Lnx≤12nn+1!fn+1x∞.Proof.
(6)max−1≤x≤1fx−Lnx≤1n+1!fn+1x∞x−x0x−x1…x−xn∞=1n+1!fn+1x∞12nCn+1∞≤12nn+1!fn+1x∞.
Therefore, it can be seen from the abovementioned theorem that to find the optimal uniform approximation polynomial offx on the interval −1,1, we only need to set the interpolation node of Lnx as the zero point of Chebychev polynomial Cn+1x. For the function fx on an interval a,b, we can take the transformation x=a+b/2+b−a/2t,−1≤t≤1, so that fx=fa+b/2+b−a/2t=gt. Then, we can apply Theorem 1 to gt. We note that compared with Taylor polynomials, this method of polynomial approximation causes more similarities of derivatives with the sigmoid function, which might help produce a better model (see Figure 2).
To justify our claims, we compare the accuracy of the produced BP neural network model using different activation functions with the Iris dataset (see Table4).Figure 2
Polynomial approximation for the sigmoid functionσx on the interval −7,7. L3x=0.500781+0.14670403x+0.001198x2−0.001006x3 generated by our method and T3x=1/2+1/4x−1/4x−1/48x3 generated by Taylor expansions.Table 4
Accuracy of different activation functions on the Iris dataset.
Iteration50 (%)100 (%)150 (%)180 (%)200 (%)220 (%)240 (%)260 (%)280 (%)300 (%)σx33.3333.3340.6774.6783.3384.0084.0084.6782.6785.33T3x33.3333.3340.6751.3381.3384.0084.0084.0082.6782.67L3x33.3333.3345.3354.6758.0088.0090.0090.0091.3091.30


### 3.2. Our SecureBP Network Model

In this section, we explain how to perform the lightweight interactive protocol to refresh ciphertexts during the training phase. To be precise, we explicitly describe a full pipeline of the evaluation of the SecureBP. We adopt the same assumptions as in the previous section so that the whole database can be encrypted inm ciphertexts.First of all, in the setup phase, the user encrypts the dataset and sends them to the public cloud. The cloud randomly initializes weights and biases (in the initialization phase, the weights and biases can be plaintexts). Next, we introduce the iterative computing phase carried out in the cloud. The goal of each iteration is to update the weights and biases. Note thatct.xi (ct.hj, ct.ok, ct.Wj,ih, ct.Wk,jo, ct.bjh, ct.bko) denotes the ciphertext of xi (hj, ok, Wj,ih, Wk,jo, bjh, bko, respectively). Each iteration consists of the following six steps:Step 1.
Cloud starts the iterative computation (here,ct.rj (including ct.rk in (8)) represents the encryption of a small random number, which has no effect on the correctness of the decryption):(7)ct.hj′=∑ict.Wj,ih⊗ct.xi⊕ct.bjh,ct.h˜j=L3ct.hj′⊕ct.rj.Step 2.
Cloud sendsct.h˜jj∈z to the user. After decrypting and reencrypting them, the user sends the refresh ciphertext ct.hjj∈z to the cloud for further computation.Step 3.
Cloud computes(8)ct.ok′=∑ict.Wk,jo⊗ct.hi⊕ct.bko,ct.o˜k=L3ct.ok′⊕ct.rk.Step 4.
Cloud sendsct.o˜kk∈d to the user. After decrypting and reencrypting them, the user sends the refresh ciphertext ct.okk∈d to the cloud for further computation.Step 5.
Cloud updatesct.Wk,jo and ct.bko:(9)Errko=ct.ok⊗1−ct.ok⊗tk−ct.ok,ct.Wk,jo=ct.Wk,jo⊕l⊙Errko⊗ct.hj,ct.bko=ct.bko⊕l⊙Errko.Step 6.
Cloud updatesct.Wj,ih, ct.bjh,(10)Errjh=ct.hj⊗1−ct.hj∑kErrkoct.Wk,jo,ct.Wj,ih=ct.Wj,ih⊕l⊙Errjh⊗ct.xi,ct.bjh=ct.bjh⊕l⊙Errjh.
In the abovementioned iteration, we choose the interaction between the cloud and the user to avoid high-cost bootstrapping. We will send the outputs of the hidden layer and the output layer to the user. After the user refreshes these ciphertexts, they will be sent to the cloud to continue the subsequent homomorphic operations. Because the outputs of the hidden layer and output layer are two ciphertext vectors, with a total ofz+d ciphertexts, the communication cost between the cloud and the user is not high. Through the analysis of noise in the later section, we can find that the advantage of this interactive protocol makes the noise of ciphertext in the process of homomorphic operations grow linearly after it reaches a certain value (i.e., e33).
In this process, it should also be noted that what the cloud sends to the user is not the true outputs of the hidden layer and output layer (i.e.,ct.hj′j∈z and ct.ok′k∈d), but the disturbed ct.h˜jj∈z and ct.o˜kk∈d. The idea is to prevent the user from snooping into the cloud to train the neural network.


## 3.1. A Decent Polynomial Approximation

In the preceding update formula, except for activation function inside neurons (i.e., sigmoid functionσx=1/1+e−x), all other operations in BP network are addition and multiplication, so they can be implemented over encrypted data. One limitation of the existing HE schemes is that they only support polynomial arithmetic operations. The evaluation of the activation function is an obstacle for the implementation of the BP network since it cannot be expressed as a polynomial. Hence, in order to operate a complete BP neural network over encrypted data, we replace the sigmoid function with polynomial approximations that are compatible with practical HE schemes.Actually, the Taylor polynomialsTdx=∑k=0dfx0/k!xk have been commonly used for approximation of the sigmoid function [32–34]:(3)σx=12+14x−148x3+1480x5+Ox7.However, we observe that the size of error grows rapidly asx increases. Besides, in order to guarantee the accuracy of the BP network, we have to use a higher degree Taylor polynomial, but it requires too many homomorphic multiplications to be practically implemented. In summary, although Taylor expansions are more convenient and easier to compute, the accuracy of estimation is not always consistent because it is a local approximation near a certain point. Therefore, we introduce another good candidate for approximation with better approximation ability to replace the sigmoid function: optimal and uniform polynomial approximation of σx. Not exactly, we find a polynomial function Lnx that minimizes the absolute value of the error between σx and Lnx within a given interval.The Chebyshev polynomials are used to construct the optimal uniform approximation polynomialsLnx. The Chebyshev polynomials Cnx can be simply defined as for −1≤x≤1,(4)Cnx=cosn⋅arccosx,n=0,1,2,…From the abovementioned definition, we can get two important properties of Chebyshev polynomials. The first property is that we can get a recurrence of Chebyshev polynomialsC0x=1,C1x=x,Cn+1x=2xCnx−Cn−1x. The second is that the Chebyshev polynomial Cnx has n different zero points on the interval −1,1, i.e., xk=cos2k−1π/2n,k=1,2,…,n. Then, we can get the important theorem in polynomial approximation as follows.Theorem 1.
Letfx be a continuous differentiable function on interval −1,1, Lnx be the interpolation polynomial, and its interpolation nodes x0,⋯,xn are the zero points of Chebychev polynomial Cn+1x, then Lnx is the optimal and uniform polynomial approximation of fx on interval −1,1, and(5)max−1≤x≤1fx−Lnx≤12nn+1!fn+1x∞.Proof.
(6)max−1≤x≤1fx−Lnx≤1n+1!fn+1x∞x−x0x−x1…x−xn∞=1n+1!fn+1x∞12nCn+1∞≤12nn+1!fn+1x∞.
Therefore, it can be seen from the abovementioned theorem that to find the optimal uniform approximation polynomial offx on the interval −1,1, we only need to set the interpolation node of Lnx as the zero point of Chebychev polynomial Cn+1x. For the function fx on an interval a,b, we can take the transformation x=a+b/2+b−a/2t,−1≤t≤1, so that fx=fa+b/2+b−a/2t=gt. Then, we can apply Theorem 1 to gt. We note that compared with Taylor polynomials, this method of polynomial approximation causes more similarities of derivatives with the sigmoid function, which might help produce a better model (see Figure 2).
To justify our claims, we compare the accuracy of the produced BP neural network model using different activation functions with the Iris dataset (see Table4).Figure 2
Polynomial approximation for the sigmoid functionσx on the interval −7,7. L3x=0.500781+0.14670403x+0.001198x2−0.001006x3 generated by our method and T3x=1/2+1/4x−1/4x−1/48x3 generated by Taylor expansions.Table 4
Accuracy of different activation functions on the Iris dataset.
Iteration50 (%)100 (%)150 (%)180 (%)200 (%)220 (%)240 (%)260 (%)280 (%)300 (%)σx33.3333.3340.6774.6783.3384.0084.0084.6782.6785.33T3x33.3333.3340.6751.3381.3384.0084.0084.0082.6782.67L3x33.3333.3345.3354.6758.0088.0090.0090.0091.3091.30


## 3.2. Our SecureBP Network Model

In this section, we explain how to perform the lightweight interactive protocol to refresh ciphertexts during the training phase. To be precise, we explicitly describe a full pipeline of the evaluation of the SecureBP. We adopt the same assumptions as in the previous section so that the whole database can be encrypted inm ciphertexts.First of all, in the setup phase, the user encrypts the dataset and sends them to the public cloud. The cloud randomly initializes weights and biases (in the initialization phase, the weights and biases can be plaintexts). Next, we introduce the iterative computing phase carried out in the cloud. The goal of each iteration is to update the weights and biases. Note thatct.xi (ct.hj, ct.ok, ct.Wj,ih, ct.Wk,jo, ct.bjh, ct.bko) denotes the ciphertext of xi (hj, ok, Wj,ih, Wk,jo, bjh, bko, respectively). Each iteration consists of the following six steps:Step 1.
Cloud starts the iterative computation (here,ct.rj (including ct.rk in (8)) represents the encryption of a small random number, which has no effect on the correctness of the decryption):(7)ct.hj′=∑ict.Wj,ih⊗ct.xi⊕ct.bjh,ct.h˜j=L3ct.hj′⊕ct.rj.Step 2.
Cloud sendsct.h˜jj∈z to the user. After decrypting and reencrypting them, the user sends the refresh ciphertext ct.hjj∈z to the cloud for further computation.Step 3.
Cloud computes(8)ct.ok′=∑ict.Wk,jo⊗ct.hi⊕ct.bko,ct.o˜k=L3ct.ok′⊕ct.rk.Step 4.
Cloud sendsct.o˜kk∈d to the user. After decrypting and reencrypting them, the user sends the refresh ciphertext ct.okk∈d to the cloud for further computation.Step 5.
Cloud updatesct.Wk,jo and ct.bko:(9)Errko=ct.ok⊗1−ct.ok⊗tk−ct.ok,ct.Wk,jo=ct.Wk,jo⊕l⊙Errko⊗ct.hj,ct.bko=ct.bko⊕l⊙Errko.Step 6.
Cloud updatesct.Wj,ih, ct.bjh,(10)Errjh=ct.hj⊗1−ct.hj∑kErrkoct.Wk,jo,ct.Wj,ih=ct.Wj,ih⊕l⊙Errjh⊗ct.xi,ct.bjh=ct.bjh⊕l⊙Errjh.
In the abovementioned iteration, we choose the interaction between the cloud and the user to avoid high-cost bootstrapping. We will send the outputs of the hidden layer and the output layer to the user. After the user refreshes these ciphertexts, they will be sent to the cloud to continue the subsequent homomorphic operations. Because the outputs of the hidden layer and output layer are two ciphertext vectors, with a total ofz+d ciphertexts, the communication cost between the cloud and the user is not high. Through the analysis of noise in the later section, we can find that the advantage of this interactive protocol makes the noise of ciphertext in the process of homomorphic operations grow linearly after it reaches a certain value (i.e., e33).
In this process, it should also be noted that what the cloud sends to the user is not the true outputs of the hidden layer and output layer (i.e.,ct.hj′j∈z and ct.ok′k∈d), but the disturbed ct.h˜jj∈z and ct.o˜kk∈d. The idea is to prevent the user from snooping into the cloud to train the neural network.


## 4. Estimation

In this section, we show the parameters setting for BP and the CKKS scheme and analyze the estimation and implementation results.


### 4.1. Parameters Setting and Estimation Results

#### 4.1.1. Parameters for the BP Algorithm

In the BP model, the numbers of input nodes and output nodes are determined, while the number of hidden nodes is uncertain. In fact, the number of hidden nodes has an impact on the performance of the neural network; an empirical formula can determine the number of hidden nodes as follows:(11)z=d+m+a,where z is the number of hidden nodes, d is the number of input nodes, m is the output nodes, and a is an adjustment constant between 0 and 10.Weights are initialized as uniformly random values in the range−0.1,0.1. Feature values in each dataset are normalized between 0 and 1. The architecture and training parameters used in our secure neural network model are shown in Table 5, and we choose Iris, Diabetes, and Sonar datasets, which are from the University of California at Irvine (UCI) dataset repository [25]. The conventional BP learning network has the same parameters as the SecureBP algorithm.Table 5
Datasets and parameters in BP.
DatasetNumber of samplesNumber of featuresNumber of hidden nodesLearning rateIris150440.6Diabetes768850.5Sonar2086090.4


#### 4.1.2. Parameters for the CKKS Scheme

In the CKKS scheme, the coefficients of error polynomials are sampled from the discrete Gaussian distribution of standard deviationσ=3.2 and a secret key is chosen randomly from the set of signed binary polynomials with the Hamming weight h=64. We used the estimator of Albrecht et al. [35] to guarantee that the proposed parameter sets achieve at least 80 bit security level against the known attacks against the LWE problem.We analyze the growth of noises in some ciphertexts, and Table6 provides theoretical upper bounds on the noise growth during homomorphic operations. Note that e denotes the noise of a fresh ciphertext.Table 6
Noise growth during the SecureBP process.
The ciphertextThe 1st iterationThei-th iteration i=2,3,4,…ct.hje6+Oe5ii−1/23⋅e33+Oe32ct.oie6+Oe5i−13⋅e15+Oe14ct.Wj,ioe4+Oe3i⋅e4+Oe3ct.Wk,jhe10+Oe9i⋅i+1/2⋅e10+Oe9As can be seen from Table6, the maximum size of growth noise during homomorphic operations is O˜e33 [36]; we choose parameters as follows: L = 10, N = 215, logq=55, and logQL=611.


### 4.2. Estimation and Implementation Results

By carefully analyzing our SecureBP protocol, we calculate the number of homomorphic operations required for each step in the course of an iteration (as shown in Table7).Table 7
Number of operations in SecureBP.
StepEncDecMultCMultAdd100mz+2z0mz+z2zz000300zd+2d0zd+d4dd0005003d4d5d600d+3z3zd+2zTotalz+dz+dmz+2zd+5d+z4d+3zmz+2zd+6d+3zdFrom Table7, we can see that the computation time required for an iteration is only related to the number of nodes in each layer. Combined with the time required for each homomorphic operation in [36], we give the estimation time (Table 8) of training SecureBP network homomorphically with Iris, Diabetes, and Sonar datasets, and Table 9 shows the accuracy comparison of encrypted and unencrypted BP networks in the case of 10 and 23 iterations, respectively.Table 8
Homomorphic training of SecureBP.
DatasetEnc (ms)Dec (ms)Mult (ms)CMult (ms)Add (ms)Total (ms)Iris522396068836167.77632.7Diabetes81246.878721012219.39962.1Sonar406078209921716580.52.0993 × 108Table 9
Comparison of encrypted/unencrypted BP algorithm.
DatasetNumber of iterationsError rate of SecureBP (%)Error rate of conventional BP (%)Iris1066.6766.672320.4016.67Diabetes1036.9734.712334.9033.89Sonar1021.4518.262317.7717.21


### 4.3. Efficiency and Accuracy Discussion

There are still some limitations in the application of our evaluation model to an arbitrary dataset. On the one hand, the HE system is a promising solution for the privacy issue, but its efficiency in real applications remains an open question. In other words, one constraint in our approach is that the efficiency of the SecureBP network is limited by the efficiency of homomorphic operations. On the other hand, we find that the accuracy of the network model is positively correlated with the degree of approximate polynomials. However, the higher degree of polynomial means the more homomorphic operations, the more time it takes to train the network model. Therefore, we need the tradeoff between the training efficiency and accuracy of the model.


## 4.1. Parameters Setting and Estimation Results

### 4.1.1. Parameters for the BP Algorithm

In the BP model, the numbers of input nodes and output nodes are determined, while the number of hidden nodes is uncertain. In fact, the number of hidden nodes has an impact on the performance of the neural network; an empirical formula can determine the number of hidden nodes as follows:(11)z=d+m+a,where z is the number of hidden nodes, d is the number of input nodes, m is the output nodes, and a is an adjustment constant between 0 and 10.Weights are initialized as uniformly random values in the range−0.1,0.1. Feature values in each dataset are normalized between 0 and 1. The architecture and training parameters used in our secure neural network model are shown in Table 5, and we choose Iris, Diabetes, and Sonar datasets, which are from the University of California at Irvine (UCI) dataset repository [25]. The conventional BP learning network has the same parameters as the SecureBP algorithm.Table 5
Datasets and parameters in BP.
DatasetNumber of samplesNumber of featuresNumber of hidden nodesLearning rateIris150440.6Diabetes768850.5Sonar2086090.4


### 4.1.2. Parameters for the CKKS Scheme

In the CKKS scheme, the coefficients of error polynomials are sampled from the discrete Gaussian distribution of standard deviationσ=3.2 and a secret key is chosen randomly from the set of signed binary polynomials with the Hamming weight h=64. We used the estimator of Albrecht et al. [35] to guarantee that the proposed parameter sets achieve at least 80 bit security level against the known attacks against the LWE problem.We analyze the growth of noises in some ciphertexts, and Table6 provides theoretical upper bounds on the noise growth during homomorphic operations. Note that e denotes the noise of a fresh ciphertext.Table 6
Noise growth during the SecureBP process.
The ciphertextThe 1st iterationThei-th iteration i=2,3,4,…ct.hje6+Oe5ii−1/23⋅e33+Oe32ct.oie6+Oe5i−13⋅e15+Oe14ct.Wj,ioe4+Oe3i⋅e4+Oe3ct.Wk,jhe10+Oe9i⋅i+1/2⋅e10+Oe9As can be seen from Table6, the maximum size of growth noise during homomorphic operations is O˜e33 [36]; we choose parameters as follows: L = 10, N = 215, logq=55, and logQL=611.


## 4.1.1. Parameters for the BP Algorithm

In the BP model, the numbers of input nodes and output nodes are determined, while the number of hidden nodes is uncertain. In fact, the number of hidden nodes has an impact on the performance of the neural network; an empirical formula can determine the number of hidden nodes as follows:(11)z=d+m+a,where z is the number of hidden nodes, d is the number of input nodes, m is the output nodes, and a is an adjustment constant between 0 and 10.Weights are initialized as uniformly random values in the range−0.1,0.1. Feature values in each dataset are normalized between 0 and 1. The architecture and training parameters used in our secure neural network model are shown in Table 5, and we choose Iris, Diabetes, and Sonar datasets, which are from the University of California at Irvine (UCI) dataset repository [25]. The conventional BP learning network has the same parameters as the SecureBP algorithm.Table 5
Datasets and parameters in BP.
DatasetNumber of samplesNumber of featuresNumber of hidden nodesLearning rateIris150440.6Diabetes768850.5Sonar2086090.4


## 4.1.2. Parameters for the CKKS Scheme

In the CKKS scheme, the coefficients of error polynomials are sampled from the discrete Gaussian distribution of standard deviationσ=3.2 and a secret key is chosen randomly from the set of signed binary polynomials with the Hamming weight h=64. We used the estimator of Albrecht et al. [35] to guarantee that the proposed parameter sets achieve at least 80 bit security level against the known attacks against the LWE problem.We analyze the growth of noises in some ciphertexts, and Table6 provides theoretical upper bounds on the noise growth during homomorphic operations. Note that e denotes the noise of a fresh ciphertext.Table 6
Noise growth during the SecureBP process.
The ciphertextThe 1st iterationThei-th iteration i=2,3,4,…ct.hje6+Oe5ii−1/23⋅e33+Oe32ct.oie6+Oe5i−13⋅e15+Oe14ct.Wj,ioe4+Oe3i⋅e4+Oe3ct.Wk,jhe10+Oe9i⋅i+1/2⋅e10+Oe9As can be seen from Table6, the maximum size of growth noise during homomorphic operations is O˜e33 [36]; we choose parameters as follows: L = 10, N = 215, logq=55, and logQL=611.


## 4.2. Estimation and Implementation Results

By carefully analyzing our SecureBP protocol, we calculate the number of homomorphic operations required for each step in the course of an iteration (as shown in Table7).Table 7
Number of operations in SecureBP.
StepEncDecMultCMultAdd100mz+2z0mz+z2zz000300zd+2d0zd+d4dd0005003d4d5d600d+3z3zd+2zTotalz+dz+dmz+2zd+5d+z4d+3zmz+2zd+6d+3zdFrom Table7, we can see that the computation time required for an iteration is only related to the number of nodes in each layer. Combined with the time required for each homomorphic operation in [36], we give the estimation time (Table 8) of training SecureBP network homomorphically with Iris, Diabetes, and Sonar datasets, and Table 9 shows the accuracy comparison of encrypted and unencrypted BP networks in the case of 10 and 23 iterations, respectively.Table 8
Homomorphic training of SecureBP.
DatasetEnc (ms)Dec (ms)Mult (ms)CMult (ms)Add (ms)Total (ms)Iris522396068836167.77632.7Diabetes81246.878721012219.39962.1Sonar406078209921716580.52.0993 × 108Table 9
Comparison of encrypted/unencrypted BP algorithm.
DatasetNumber of iterationsError rate of SecureBP (%)Error rate of conventional BP (%)Iris1066.6766.672320.4016.67Diabetes1036.9734.712334.9033.89Sonar1021.4518.262317.7717.21


## 4.3. Efficiency and Accuracy Discussion

There are still some limitations in the application of our evaluation model to an arbitrary dataset. On the one hand, the HE system is a promising solution for the privacy issue, but its efficiency in real applications remains an open question. In other words, one constraint in our approach is that the efficiency of the SecureBP network is limited by the efficiency of homomorphic operations. On the other hand, we find that the accuracy of the network model is positively correlated with the degree of approximate polynomials. However, the higher degree of polynomial means the more homomorphic operations, the more time it takes to train the network model. Therefore, we need the tradeoff between the training efficiency and accuracy of the model.


## 5. Conclusion and Future Work

In this paper, we present a SecureBP network model for homomorphic training. We introduce two methods, more accuracy polynomial approximation and lightweight interactive protocol, to solve the difficulties encountered when the CKKS scheme is used to protect the BP network, and our method has a good experimental performance on different datasets. For future work, we plan to explore how to train the deep neural network and the convolutional neural network effectively on encrypted data in a training-as-a-service setting.



---

*Source: 5328059-2020-06-12.xml*