Title: Privacy-Preserving CNN Training with Transfer Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: Privacy-Preserving

Homomorphic Encryption

Multiclass Logistic Regression

Quadratic Gradient

Squared Likelihood Error

Volley Revolver
Abstract: Privacy-preserving nerual network inference has been well studied while homomorphic CNN training still remains an open challenging task.
In this paper,  we present a practical solution to implement privacy-preserving CNN training based on mere Homomorphic Encryption (HE) technique. To our best knowledge, this is the first attempt successfully to crack this nut and no work ever before has achieved this goal. Several techniques combine to accomplish the task:: (1) with transfer learning, privacy-preserving CNN training can be reduced to homomorphic neural network training, or even multiclass logistic regression (MLR) training; (2) via a faster gradient variant called $\texttt{Quadratic Gradient}$, an enhanced gradient method for MLR with a state-of-the-art performance in convergence speed is applied in this work to achieve high performance; (3) we employ the thought of transformation in mathematics to transform approximating Softmax function in the encryption domain to the approximation of the Sigmoid function. A new type of loss function termed $\texttt{Squared Likelihood Error}$   has been developed alongside to align with this change.; and (4) we use a simple but flexible matrix-encoding method named $\texttt{Volley Revolver}$  to manage the data flow in the ciphertexts, which is the key factor to complete the whole homomorphic CNN training.  The complete, runnable C++ code to implement our  work can be found at: \href{https://anonymous.4open.science/r/HE-CNNtraining-B355/}{$\texttt{https://anonymous.4open.science/r/HE-CNNtraining-B355/}$}. 

We select $\texttt{REGNET\_X\_400MF}$ as our pre-trained model for transfer learning. We use the first 128 MNIST training images as training data and the whole MNIST testing dataset as the testing data. The client only needs to upload 6 ciphertexts to the cloud and it takes $\sim 21$ mins to perform 2 iterations on a cloud with 64 vCPUs, resulting in a precision of $21.49\%$.


Abstract

Privacy-preserving nerual network inference has been well studied while homomorphic CNN training still remains an open challenging task. In this paper, we present a practical solution to implement privacy-preserving CNN training based on mere Homomorphic Encryption (HE) technique. To our best knowledge, this is the first attempt successfully to crack this nut and no work ever before has achieved this goal. Several techniques combine to accomplish the task:: (1) with transfer learning, privacy-preserving CNN training can be reduced to homomorphic neural network training, or even multiclass logistic regression (MLR) training; (2) via a faster gradient variant called ◗✉❛❞/a114❛/a116✐❝ ●/a114❛❞✐❡♥/a116 , an enhanced gradient method for MLR with a state-of-the-art performance in convergence speed is applied in this work to achieve high performance; (3) we employ the thought of transformation in mathematics to transform approximating Softmax function in the encryption domain to the approximation of the Sigmoid function. A new type of loss function termed ❙/a113✉❛/a114❡❞ ▲✐❦❡❧✐❤♦♦❞ ❊/a114/a114♦/a114 has been developed alongside to align with this change.; and (4) we use a simple but flexible matrix-encoding method named ❱♦❧❧❡② ❘❡✈♦❧✈❡/a114 to manage the data flow in the ciphertexts, which is the key factor to complete the whole homomorphic CNN training. The complete, runnable C++ code to implement our work can be found at: ❤/a116/a116♣/a115✿✴✴❛♥♦♥②♠♦✉/a115✳✹♦♣❡♥✳/a115❝✐❡♥❝❡✴/a114✴❍❊✲❈◆◆/a116/a114❛✐♥✐♥❣✲❇✸✺✺✴ . mins to perform 2 iterations on a cloud with 64 vCPUs,

We select ❘❊●◆❊❚❴❳❴✹✵✵▼❋ as our pre-trained model for transfer learning. We use the first 128 MNIST training images as training data and the whole MNIST testing dataset as the testing data. The client only needs to upload 6 ciphertexts to the cloud and it takes ∼ 21 resulting in a precision of 21 . 49% .

1 Introduction 25

1.1 Background 26

Applying machine learning to problems involving sensitive data requires not only accurate predictions 27

but also careful attention to model training. Legal and ethical requirements might limit the use of 28

machine learning solutions based on a cloud service for such tasks. As a particular encryption scheme, 29

homomorphic encryption provides the ultimate security for these machine learning applications and 30

ensures that the data remains confidential since the cloud does not need private keys to decrypt it. 31

However, it is a big challenge to train the machine learning model, such as neural networks or even 32

convolution neural networks, in such encrypted domains. Nonetheless, we will demonstrate that 33

cloud services are capable of applying neural networks over the encrypted data to make encrypted 34

training, and also return them in encrypted form. 35

1.2 Related work 36

Several studies on machine learning solutions are based on homomorphic encryption in the cloud 37 environment. Since Gilad-Bachrach et al. [1] firstly considered privacy-preserving deep learning 38 prediction models and proposed the private evaluation protocol ❈/a114②♣/a116♦◆❡/a116/a115 for CNN, many other 39 approaches [2, 3, 4, 5] for privacy-preserving deep learning prediction based on HE or its combination 40 with other techniques have been developed. Also, there are several studies [6, 7, 8, 9] working on 41 logistic regression models based on homomorphic encryption. 42

However, to our best knowledge, no work ever before based on mere HE techique has presented an 43 solution to successfully perform homomorphic CNN training. 44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

1.3 Contributions

Our specific contributions in this paper are as follows:

with various techniques, we initiate to propose a practical solution for privacy-preserving CNN training, demonstrating the feasibility of homomorphic CNN training.

We suggest a new type of loss function, ❙/a113✉❛/a114❡❞ ▲✐❦❡❧✐❤♦♦❞ ❊/a114/a114♦/a114 (SLE), which is friendly to pervacy-perserving manner. As a result, we can use the Sigmoid function to replace the Softmax function which is too diffuclt to calculate in the encryption domain due to its uncertainty.

We develop a new algorithm with SLE loss function for MLR using /a113✉❛❞/a114❛/a116✐❝ ❣/a114❛❞✐❡♥/a116 . Experiments show that this HE-friendly algorithm has a state-of-the-art performance in convergence speed.

2 Preliminaries

We adopt ' ⊗ ' to denote the kronecker product and ' glyph[circledot] ' to denote the component-wise multiplication between matrices.

2.1 Fully Homomorphic Encryption

Homomorphic Encryption (HE) is one type of encryption scheme with a special characteristic called Homomorphic , which allows to compute on encrypted data without having access to the secret key. Fully HE means that the scheme is fully homomorphic, namely, homomorphic with regards to both addition and multiplication, and that it allows arbitrary computation on encrypted data. Since Gentry proposed the first fully HE scheme [10] in 2009, some technological progress on HE has been made. For example, Brakerski, Gentry and Vaikuntanathan [11] present a novel way of constructing leveled fully homomorphic encryption schemes (BGV) and Smart and Vercauteren [12] introduced one of the most important features of HE systems, a packing technique based on polynomial-CRT called Single Instruction Multiple Data (aka SIMD) to encrypt multiple values into a single ciphertext. Another great progress in terms of machine learning applications is the rescaling procedure [13], which can manage the magnitude of plaintext effectively.

Modern fully HE schemes, such as ❍❊❆❆◆ , usually support seveal common homomorphic operations: the encryption algorithm ❊♥❝ encrypting a vector, the decryption algorithm ❉❡❝ decrypting a ciphertext, the homomorphic addition ❆❞❞ and multiplication ▼✉❧/a116 between two ciphertexts, the multiplication ❝▼✉❧/a116 of a contant vector with a ciphertext, the rescaling operation ❘❡❙❝❛❧❡ to reduce the magnitude of a plaintext to an appropriate level, the rotation operation ❘♦/a116 generating a new ciphertext encrypting the shifted plaintext vector, and the bootstrapping operation ❜♦♦/a116/a115/a116/a114❛♣ to refresh a ciphertext usually with a small ciphertext modulus.

2.2 Database Encoding Method

For a given database Z , Kim et al. [6] first developed an efficient database encoding method, in order 79 to make full use of the HE computation and storage resources. They first expand the matrix database 80 to a vector form V in a row-by-row manner and then encrypt this vector V to obtain a ciphertext 81 Z = Enc ( V ) . Also, based on this database encoding, they mentioned two simple operations via 82

shifting the encrypted vector by two different positions, respectively: the complete row shifting 83 and the incomplete column shifting. These two operations performing on the matrix Z output the 84 matrices Z ′ and Z ′′ , as follows: 85

The complete column shifting to obtain the matrix Z ′′′ can also be achieved by two ❘♦/a116 , two ❝▼✉❧/a116 , 86 and an ❆❞❞ . 87

88

89

90

91

Other works [14, 4] using the same encoding method also developed some other procedures, such as ❙✉♠❘♦✇❱❡❝ and ❙✉♠❈♦❧❱❡❝ to calculate the summation of each row and column, respectively. Such basic common and simple operations consisting of a series of HE operations are significantly important for more complex calculations such as the homomorphic evaluation of gradient.

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

2.3 Convolutional Neural Network

Inspired by biological processes, Convolutional Neural Networks (CNN) are a type of artificial neural network most commonly used to analyze visual images. CNNs play a significant role in image recognition due to their powerful performance. It is also worth mentioning that the CNN model is one of a few deep learning models built with reference to the visual organization of the human brain.

2.3.1 Transfer Learning

Transfer learning in machine learning is a class of methods in which a pretrained model can be used as an optimization for a new model on a related task, allowing rapid progress in modeling the new task. In real-world applications, very few researchers train entire convolutional neural networks from scratch for image processing-related tasks. Instead, it is common to use a well-trained CNN as a fixed feature extractor for the task of interest. In our case, we freeze all the weights of the selected pre-trained CNN except that of the final fully-connected layer. We then replace the last fully-connected layer with a new layer with random weights (such as zeros) and only train this layer.

REGNET_X_400MF To use transfer learning in our privacy-preserving CNN training, we adopt a new network design paradigm called ❘❡❣◆❡/a116 , recently introduced by Facebook AI researchers, as our pre-trained model. RegNet is a low-dimensional design space consisting of simple, regular networks. In particular, we apply ❘❊●◆❊❚❴❳❴✹✵✵▼❋ as a fixed feature extractor and replaced the final fully connected layer with a new one of zero weights. CNN training in this case can be simplified to multiclass logistic regression training. Since ❘❊●◆❊❚❴❳❴✹✵✵▼❋ only receive color images of size 224 × 224 , the grayscale images will be stacked threefold and images of different sizes will be resized to the same size in advance. These two transformations can be done by using PyTorch.

2.3.2 Datasets

We adopt three common datasets in our experiments: MNIST, USPS, and CIFAR10. Table 1 describes the three datasets.

3 Technical details

3.1 Multiclass Logistic Regression

Multiclass Logistic Regression, or Multinomial Logistic Regression, can be seen as an extension of logistic regression for multi-class classification problems. Supposing that the matrix X ∈ R n × (1+ d ) ,

126

127

128

129

130

131

132

133

134

×

×

the column vector Y ∈ N n × 1 , the matrix ¯ Y ∈ R n × c , and the matrix W ∈ R c × (1+ d ) represent 120 the dataset, class labels, the one-hot encoding of the class labels, and the MLR model parameter, 121 respectively: 122

MLR aims to maxsize L or ln L :

The loss function ln L is a multivariate function of [(1 + c )(1 + d )] variables, which has its column123 vector gradient ∇ of size [(1 + c )(1 + d )] and Hessian square matrix ∇ 2 of order [(1 + c )(1 + d )] as 124 follows: 125

Nesterov's Accelerated Gradient With ∇ or ∇ 2 , first-order gradient algorithms or second-order Newton-Raphson method are commonly applied in MLE to maxmise ln L . In particular, Nesterov's Accelerated Gradient (NAG) is a practical solution for homomorphic MLR without frequent inversion operations. It seems plausible that the NAG method is probably the best choice for privacy-preserving model training.

3.2 Chiang's Quadratic Gradient

❈❤✐❛♥❣✬/a115 ◗✉❛❞/a114❛/a116✐❝ ●/a114❛❞✐❡♥/a116 (CQG) [15, 16, 9] is a faster, promising gradient variant that can combine the first-order gradient descent/ascent algorithms and the second-order Newton-Raphson method, accelerating the raw Newton-Raphson method with various gradient algorithms and probably

helpful to build super-quadratic algorithms. For a function F ( x ) with its gradient g and Hessian 135 matrix H , to build CQG, we first construct a diagonal matrix ¯ B from the Hessian H itself: 136

where ¯ h ji is the elements of the matrix H and ε is a small constant positive number.

CQG for the function F ( x ) , defined as G = ¯ B · g , has the same dimension as the raw gradient g . To apply CQG in practice, we can use it in the same way as the first-order gradient algorithms, except that we need to replace the naive gradient with the quadratic gradient and adopt a new learning rate (usually by increasing 1 to the original learning rate).

For efficiency in applying CQG, a good bound matrix should be attempted to obtain in order to replace the Hessian itself. Chiang has proposed the enhanced NAG method via CQG for MLR with a fixed Hessian [17, 7, 18] substitute built from 1 2 X ᵀ X .

3.3 Approximating Softmax Function

It might be impractical to perfectly approximate Softmax function in the privacy-preserving domain due to its uncertainty. To address this issue, we employ the thought of transformation from mathematics: transforming one tough problem into another easier one. That is, instead of trying to approximate the Softmax function, we attempt to approximate the Sigmoid function in the encryption domain, which has been well-studied by several works using the least-square method.

In line with standard practice of the log-likelihood loss function involving the Softmax function, we should try to maximize the new loss function

We can prove that ln L 1 is concave and deduce that 1 4 E ⊗ X ᵀ X can be used to build the CQG for 151 ln L 1 . However, the performance of this loss function ln L 1 is not ideal, probably because for the 152 individual example its gradient and Hessian contain no information about any other class weights not 153 related to this example. 154

❙/a113✉❛/a114❡❞ ▲✐❦❡❧✐❤♦♦❞ ❊/a114/a114♦/a114 After many attempts to finding a proper loss function, we develop a novel loss function that can have a competitive performance to the log-likelihood loss function, which we term ❙/a113✉❛/a114❡❞ ▲✐❦❡❧✐❤♦♦❞ ❊/a114/a114♦/a114 (SLE):

We can also prove that ln L 2 is concave and that 1 4 E ⊗ X ᵀ X can be used to build the CQG for ln L 2 . The loss function SLE might be related to Mean Squared Error (MSE): the MSE loss function sums all the squared errors while SLE calculates the cumulative product of all the squared likelihood errors.

155

156

157

158

159

Combining together all the techniques above, we now have the enhanced NAG method with the SLE loss function for MLR training, described in detail in Algorithm 1.

/a80❡/a114❢♦/a114♠❛♥❝❡ ❊✈❛❧✉❛/a116✐♦♥ We test the convergence speed of the raw NAG method with log160 likelihood loss function (denoted as RawNAG), the NAG method with SLE loss function (denoted 161 as SigmoidNAG), and the enhanced NAG method via CQG with SLE loss function (denoted as 162 SigmoidNAGQG) on the three datasets described above: USPS, MNIST, and CIFAR10. Since two 163 different types of loss functions are used in these three methods, the loss function directly measuring 164 the performance of various methods will not be selected as the indicator. Instead, we select precision 165 as the only indicator in the following Python experiments. Note that we use ❘❊●◆❊❚❴❳❴✹✵✵▼❋ to in 166

137

138

139

140

141

142

143

144

145

146

147

148

149

150

167

168

169

170

171

172

173

174

175

Input: training dataset X ∈ R n × (1+ d ) ; one-hot encoding training label Y ∈ R n × c ; and the number κ of iterations;

advance extract the features of USPS, MNIST, and CIFAR10, resulting in a new same-size dataset with 401 features of each example. Figure 1 shows that our enhanced methods all converge faster than other algorithms on the three datasets.

3.4 Double Volley Revolver

Unlike those efficient, complex encoding methods [3], ❱♦❧❧❡② ❘❡✈♦❧✈❡/a114 is a simple, flexible matrix-encoding method specialized for privacy-preserving machine-learning applications, whose basic idea in a simple version is to encrypt the transpose of the second matrix for two matrices to perform multiplication. Figure 2 describes a simple case for the algorithm adopted in this encoding method.

The encoding method actually plays a significant role in implementing privacy-preserving CNN 176 training. Just as Chiang mentioned in [4], we show that Volley Revolver can indeed be used to 177 implement homomorphic CNN training. This simple encoding method can help to control and 178 manage the data flow through ciphertexts. 179

0

100

200

300

0

.

5

1

Iteration Number

RawNAG

SigmoidNAG

SigmoidNAGQG

(a) USPS Training

0

100

200

300

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Iteration Number

RawNAG

SigmoidNAG

SigmoidNAGQG

(b) USPS Testing

0

100

200

300

0

.

2

0

.

4

0

.

6

0

.

8

1

Iteration Number

RawNAG

SigmoidNAG

SigmoidNAGQG

(c) MNIST Training

0

100

200

300

0

.

2

0

.

4

0

.

6

0

.

8

1

Iteration Number

RawNAG

SigmoidNAG

SigmoidNAGQG

(d) MNIST Testing

0

100

200

300

0

.

2

0

.

4

0

.

6

0

.

8

Iteration Number

RawNAG

SigmoidNAG

SigmoidNAGQG

(e) CIFAR10 Training

0

100

200

300

0

.

2

0

.

4

0

.

6

0

.

8

Iteration Number

RawNAG

SigmoidNAG

SigmoidNAGQG

(f) CIFAR10 Testing

·

a

0

a

1

b

0

b

2

a

2

a

3

b

1

b

3

a

4

a

5

b

0

b

2

a

6

a

7

b

1

b

3

×

a

0

a

1

a

2

a

3

b

0

b

1

a

4

a

5

b

2

b

3

a

6

a

7

0

0

0

0

0

0

0

0

·

a

0

a

1

b

0

b

2

a

2

a

3

b

1

b

3

a

4

a

5

b

0

b

2

a

6

a

7

b

1

b

3

c

0

=

a

0

·

b

0

+

a

1

·

b

2

c

3

=

a

2

·

b

1

+

a

3

·

b

3

c

0

c

0

c

3

c

3

c

4

c

4

c

7

c

7

c

4

=

a

4

·

b

0

+

a

5

·

b

2

c

7

=

a

6

·

b

1

+

a

7

·

b

3

c

0

0

0

c

3

c

4

0

0

c

7

·

a

0

a

1

b

1

b

3

a

2

a

3

b

0

b

2

a

4

a

5

b

1

b

3

a

6

a

7

b

0

b

2

c

1

=

a

0

·

b

1

+

a

1

·

b

3

c

2

=

a

2

·

b

0

+

a

3

·

b

2

c

1

c

1

c

2

c

2

c

5

c

5

c

6

c

6

c

5

=

a

4

b

1

+

a

5

b

3

c

6

=

a

6

b

0

+

a

7

b

2

0

c

1

c

2

0

0

c

5

c

6

0

Encrypt

Encoding

Rot

(0)

Rot

(2)

SumColVec

(

·

)

Clean up the

redundant values

SumColVec

(

·

)

Clean up the

redundant values

⊕

⊕

·

·

·

·

However, we don't need to stick to encrypting the transpose of the second matrix. Instead, either of 180 the two matrices is transposed would do the trick: we could also encrypt the transpose of the first 181 matrix, and the corresponding multiplication algorithm due to this change is similar to the Algorithm 182 2 from [4]. 183

184

185

186

187

188

Also, if each of the two matrices are too large to be encrypted into a single ciphertext, we could also encrypt the two matrices into two teams A and B of multiple ciphertexts. In this case, we can see this encoding method as ❉♦✉❜❧❡ ❱♦❧❧❡② ❘❡✈♦❧✈❡/a114 , which has two loops: the outside loop deals with the calculations between ciphertexts from two teams while the inside loop literally calculates two sub-matrices encrypted by two ciphertexts A [ i ] and B [ j ] using the raw algorithm of Volley Revolver.

189

190

4 Privacy-preserving CNN Training

4.1 Polynomial Approximation

Although Algorithm 1 enables us to avoid computing the Softmax function in the encryption domain, 191 we still need to calculate the Sigmoid function using HE technique. This problem has been well 192 studied by several works and we adopt a simple one [19], that is (1) we first use the least-square method 193 to perfectly approximate the sigmoid function over the range [ -8 , +8] , obtaining a polynomial Z 11 194 of degree 11; and (2) we use a polynomial Z 3 of degree 3 to approximate the Sigmoid by minimizing 195 the cost function F including the squared gradient difference: 196

where λ 0 and λ 1 are two positive float numbers to control the shape of the polynomial to approximate. 197 Setting λ 0 = 128 and λ 1 = 1 would result in the polynomial we used in our privacy-preserving CNN 198 training: Z 3 = 0 . 5 + 0 . 106795345032 · x -0 . 000385032598 · x 3 . 199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

4.2 Homomorphic Evaluation

Before the homomorphic CNN training starts, the client needs to encrypt the dataset X , the data labels ¯ Y , the matrix ¯ B and the weight W into ciphertexts Enc ( X ) , Enc ( ¯ Y ) , Enc ( ¯ B ) and Enc ( W ) , respectively, and upload them to the cloud. For simplicity in presentation, we can just regard the whole pipeline of homomorphic evaluation of Algorithm 1 as updating the weight ciphertext: W = W + ¯ B glyph[circledot] ( ¯ Y -Z 3 ( X × W ᵀ )) ᵀ × X , regardless of the subtle control of the enhanced NAG method with the SLE loss function.

Since ❱♦❧❧❡② ❘❡✈♦❧✈❡/a114 only needs one of the two matrices to be transposed ahead before encryption and ( ¯ Y -Z 3 ( X × W ᵀ )) ᵀ × X happened to suffice this situation between any matrix multiplication, we can complete the homomorphic evaluation of CQG for MLR.

5 Experiments

The C++ source code to implement the experiments in this section is openly available at: ❤/a116/a116♣/a115✿✴✴❛♥♦♥②♠♦✉/a115✳✹♦♣❡♥✳/a115❝✐❡♥❝❡✴/a114✴❍❊✲❈◆◆/a116/a114❛✐♥✐♥❣✲❇✸✺✺✴ .

Implementation We implement the enhanced NAG with the SLE loss function based on HE with the library ❍❊❆❆◆ . All the experiments on the ciphertexts were conducted on a public cloud with 64 vCPUs and 192 GB RAM.

We adopt the first 128 MNIST training images as the training data and the whole test dataset as the testing data. Both the training images and testing images have been processed in advance with the pre-trained model ❘❊●◆❊❚❴❳❴✹✵✵▼❋ , resulting in a new dataset with each example of size 401.

5.1 Parameters

The parameters of ❍❊❆❆◆ we selected are: logN = 16 , logQ = 990 , logp = 45 , slots = 32768 , which ensure the security level λ = 128 . Refer [6] for the details of these parameters. We didn't use bootstrapping to refresh the weight ciphertexts and thus it can only perform 2 iterations of our algorithm. Each iteration takes ∼ 11 mins. The maximum runtime memory in this case is ∼ 18 GB. The 128 MNIST training images are encrypted into 2 ciphertexts. The client who own the private data has to upload these two ciphertexts, two ciphertexts encrypting the one-hot labels ¯ Y , one ciphertext encrypting the ¯ B and one ciphertext encrypting the weight W to the cloud. The inticial weight matrix W 0 we adopted is the zero matrix. The resulting MLR model after 2-iteration training has reached a pricision of 21 . 49% and obtain the loss of -147206 , which are consistent with the Python simulation experiment.

6 Conclusion

In this work, we initiated to implement privacy-persevering CNN training based on mere HE techniques by presenting a faster HE-friendly algorithm.

The HE operation bootstrapping could be adopted to refresh the weight ciphertexts. Python experiments imitating the privacy-preserving CNN training using Z 3 as Sigmoid substitution showed that using a large amount of data such as 8,192 images to train the MLE model for hundreds of iterations would finally reach 95% precision. The real experiments over ciphertexts conducted on a high-performance cloud with many vCPUs would take weeks to complete this test, if not months.

References

Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International conference on machine learning , pages 201-210. PMLR, 2016.

Hervé Chabanne, Amaury De Wargny, Jonathan Milgram, Constance Morel, and Emmanuel Prouff. Privacy-preserving classification on deep neural network. Cryptology ePrint Archive , 2017.

Xiaoqian Jiang, Miran Kim, Kristin Lauter, and Yongsoo Song. Secure outsourced matrix 245 computation and application to neural networks. In Proceedings of the 2018 ACM SIGSAC 246 Conference on Computer and Communications Security , pages 1209-1222, 2018. 247

John Chiang. A novel matrix-encoding method for privacy-preserving neural networks (infer248 ence). arXiv preprint arXiv:2201.12577 , 2022. 249

250

251

252

253

Florian Bourse, Michele Minelli, Matthias Minihold, and Pascal Paillier. Fast homomorphic evaluation of deep discretized neural networks. In Advances in Cryptology-CRYPTO 2018: 38th Annual International Cryptology Conference, Santa Barbara, CA, USA, August 19-23, 2018, Proceedings, Part III 38 , pages 483-512. Springer, 2018.

254

255

256

257

258

259

260

261

262

263

264

265

Andrey Kim, Yongsoo Song, Miran Kim, Keewoo Lee, and Jung Hee Cheon. Logistic regression model training based on the approximate homomorphic encryption. BMC medical genomics , 11(4):83, 2018.

Charlotte Bonte and Frederik Vercauteren. Privacy-preserving logistic regression training. BMC medical genomics , 11(4):86, 2018.

Miran Kim, Yongsoo Song, Shuang Wang, Yuhou Xia, and Xiaoqian Jiang. Secure logistic regression based on homomorphic encryption: Design and evaluation. JMIR medical informatics , 6(2):e19, 2018.

John Chiang. Privacy-preserving logistic regression training with a faster gradient variant. arXiv preprint arXiv:2201.10838 , 2022.

Craig Gentry. Fully homomorphic encryption using ideal lattices. In Proceedings of the forty-first annual ACM symposium on Theory of computing , pages 169-178, 2009.

Zvika Brakerski, Craig Gentry, and Vinod Vaikuntanathan. (leveled) fully homomorphic 266 encryption without bootstrapping. ACM Transactions on Computation Theory (TOCT) , 6(3):1267 36, 2014. 268

N.P. Smart and F. Vercauteren. Fully homomorphic simd operations. Cryptology ePrint Archive, 269 Report 2011/133, 2011. ❤/a116/a116♣/a115✿✴✴✐❛✳❝/a114✴✷✵✶✶✴✶✸✸ . 270

Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. Homomorphic encryption for 271 arithmetic of approximate numbers. In International Conference on the Theory and Application 272 of Cryptology and Information Security , pages 409-437. Springer, 2017. 273

Kyoohyung Han, Seungwan Hong, Jung Hee Cheon, and Daejun Park. Logistic regression on 274 homomorphic encrypted data at scale. In Proceedings of the AAAI Conference on Artificial 275 Intelligence , volume 33, pages 9466-9471, 2019. 276

John Chiang. Multinomial logistic regression algorithms via quadratic gradient, 2023. 277

John Chiang. Quadratic gradient: Uniting gradient algorithm and newton method as one. arXiv 278 preprint arXiv:2209.03282 , 2022. 279

Dankmar Böhning and Bruce G Lindsay. Monotonicity of quadratic-approximation algorithms. 280 Annals of the Institute of Statistical Mathematics , 40(4):641-663, 1988. 281

Dankmar Böhning. Multinomial logistic regression algorithm. Annals of the institute of 282 Statistical Mathematics , 44(1):197-200, 1992. 283

John Chiang. On polynomial approximation of activation function. arXiv preprint 284 arXiv:2202.00004 , 2022. 285

Peer reviews:
comments: SUMMARY:
This paper combines several existing techniques to achieve privacy-preserving CNN training. These techniques include transfer learning,  Quadratic Gradient,  mathematical transformation, and matrix-encoding method Volley Revolver. 

This writing is more of a technical document rather than a research paper with insights.

STRENGTHS:
1)For the first time, they apply homomorphic encryption to neural network training.
2)They demonstrate the feasibility of homomorphic CNN training.
3)They propose pervacy-perserving friendly Squared Likelihood Error (SLE) for CNN training.
4)Experimentally, their algorithm has a state-of-the-art performance in convergence speed.

WEAKNESSES:
1)The introduction of related works is pretty simple, which makes it difficult to evaluate the contributes of the paper.
2)The quality of writing/presentation is very weak and unreadable.

QUESTIONS:
1)Is the propose method suitable for other neural network, like RNN, Transformers? 
2)Do you re-evaluate your method by replacing the pretrained REGNET_X_400MF to another one?

LIMITATIONS:
See Questions and weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-29

comments: SUMMARY:
The paper presents a method for CNN transfer learning implemented in homomorphic encryption to protect privacy.

STRENGTHS:
I'm not aware of the method being implemented in HE before.

WEAKNESSES:
I cannot judge the machine learning aspects, but I don't see a strong novelty on the cryptographic side. The paper claims that some prior work is overly complex without going into details.

I find it concerning that the work relies relatively heavily on non-peer reviewed references by a single author (5 out of 19).

Line 150 says "well-studied by several works" without giving any reference.


Minor issues:
- l6: ::
- l15: .;
- l50: pervacy-persevering (privacy-preserving?)
- l51: diffuclt
- l71: seveal
- l154: After many attempts (unscholarly language)

QUESTIONS:
Line 108 mentions setting all the weight of an FC layer to zero. Wouldn't that erase all signals?

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed a CNN training technique on the homomorphic encryption domain based on transfer learning. A gradient variant called Quadratic Gradient on homomorphic encryption was proposed. And a sigmoid function-based Softmax approximation was proposed. In addition, a new loss function for squared likelihood error was proposed, and a matrix-encoding method called Volly Revolver was also proposed. Finally, they released the code they implemented.\n\nSTRENGTHS:\nProperly implementing functions for training on a homomorphic encryption domain is challenging. It is worth evaluating for implementing this and also disclosing their source code.\n\nWEAKNESSES:\nThis paper performed a simulation on the MNIST dataset for performance evaluation. This seems too simple a dataset, even considering the homomorphic encryption environment. Although they claim that it is to be the first implementation of transfer learning-based CNN training on the homomorphic encryption domain, a similar study was recently published first. Of course, this paper takes a different approach.\n\n[*] https://openreview.net/forum?id=jJXuL3hQvt\n\nThis paper is considered incomplete in several respects. The main reason is that the proposed scheme's threat model needs to be clarified. At first, \"the proposed architecture\" is not clear. There needs to be a description of the proposed architecture. They should explain the exact part where homomorphic encryption was actually carried out in the transfer learning process and what benefits can be gained from doing so.\n\nQUESTIONS:\nWhat is the security target to achieve in this paper? It is necessary to explain the threat model that the client or server faces and what benefits the client or server can get from the proposed homomorphic scheme. For example, if the client data is encrypted, the client's privacy can be protected when the server processes the data, but from the server's point of view, it is necessary to clearly explain which portion of transfer learning should be carried out on the encrypted domain. This paper lacks a discussion on this, and it is difficult to know where data processing takes place from the paper alone.\n\nIn particular, this paper said that Bootstrapping was not used. It is a very bold and interesting claim. However, they do not provide information on the entire architecture of the proposed scheme. The requirement of bootstrapping is closely related to the information on the homomorphic encryption parameters, such as the ciphertext size, the number of available slots, the number of multiplicative levels, and so on. In fact, the claim that bootstrapping is not used means that homomorphic encryption was applied only in a very small part of the entire operation in their (unknown) architecture, which raises the question of whether the method proposed in this paper is applicable in a practical application. However, judging this part is impossible because it is not explained in detail in the paper.\n\nFinally, [-8,8] is used as the average pooling range. However, it seems too small according to recent work [*].\n\nLIMITATIONS:\nNot exactly.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper employs a few heuristic methods to accelerate logistic regression training over encrypted data. The heuristics considered include: a new loss function called squared likelihood error (SLE) along with a polynomial approximation of sigmoid function, a faster gradient-descent method based on quadratic gradient, and a matrix encoding method called volley revolver.\n\nSTRENGTHS:\nThe only strength of the paper is that it attempts to solve a really challenging problem of learning over encrypted data and it provides the code apriori through an anonymous GitHub link.\n\nWEAKNESSES:\n1) First and foremost, the title of the paper is misleading. The paper never deals with CNN training even in the limited context of transfer learning. It is true that most practical ML applications start with a pre-trained model and finetunes the parameters of this model. However, transfer learning implies that the whole model is finetuned apart from learning the application-specific last fully connected (FC) layer. What this paper attempts to do is just learn the last FC layer, which is nothing but multiclass logistic regression (MLR) training. Therefore, the title of the paper should not claim anything about CNN training.\n\n2) Numerous attempts have been made over the last five years attempting to achieve MLR training on encrypted data, which have not been acknowledged in this paper and compared against. For example, see the works starting from:\n\n[A] Crawford et al., \"Doing Real Work with FHE: The Case of Logistic Regression\", 2018\n[B] Han et al., \"Logistic regression on homomorphic encrypted data at scale\", AAAI 2019\n[C] Bergamaschi et al., \"Homomorphic Training of 30,000 Logistic Regression Models\", 2019\n\n3) This current paper appears to be very similar to the rejected NeurIPS 2022 submission entitled \"Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant\". While the NeurIPS 2022 submission focused on only the quadratic gradient component, the current paper also introduces the SLE loss. However, it is not clear how this SLE loss function is better. Moreover, what is the expression for the gradient of $ln L_2$ and where is it used in Algorithm 1?\n\n4) The so-called volley revolver does not constitute any novel \"matrix-encoding\" method. Such packing tricks are regularly used in the context of efficient SIMD operations in FHE.\n\n5) Overall, none of the three claimed contributions (namely, quadratic gradient, SLE loss, and volley revolver) appear to be original or significant enough to make an overall impact.\n\n6) Finally, though the paper claims that the goal is to make logistic regression training practical, not a single experimental result has been shown to prove this point. Running 2 iterations with 128 MNIST images takes approximately 21 minutes and the last line claims that real experiments would take \"weeks, if not months\". There are other reported works in the literature, which showed more realistic results.\n\n[D] Nandakumar et al., \"Towards Deep Neural Network Training on Encrypted Data\", CVPR-W 2019\n[E] Lou et al., \"Glyph: Fast and Accurately Training Deep Neural Networks on Encrypted Data\", NeurIPS 2020\n\nQUESTIONS:\nPlease see weaknesses of the paper.\n\nLIMITATIONS:\nAll the limitations have not been presented and addressed. There appears to be no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-10"}