Title: Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant
Venue: NeurIPS 2023
Accepted: no
Keywords: Privacy-Preserving

Logistic Regression

Quadratic Gradient

Simplified Fixed Hessian

Numerical Optimization
Abstract: Logistic regression training over encrypted data has been an attractive idea to security concerns for years. In this paper, we propose a faster gradient variant called $\texttt{quadratic gradient}$ for privacy-preserving logistic regression training. The core of $\texttt{quadratic gradient}$ can be seen as an extension of the simplified fixed Hessian.  

We enhance  Nesterov's accelerated gradient (NAG) and Adaptive Gradient Algorithm (Adagrad) respectively with $\texttt{quadratic gradient}$ and evaluate the enhanced algorithms on several datasets. 
Experiments show that the enhanced methods have a state-of-the-art performance in convergence speed compared to the raw first-order gradient methods. We then  adopt the enhanced NAG method to implement homomorphic logistic regression training, obtaining a comparable result  by only $3$ iterations.

There is a promising chance that $\texttt{quadratic gradient}$ could be used to enhance other first-order gradient methods for general numerical optimization problems.

1

Abstract

15

16

17

18

19

20

21

22

23

24

25

1 Introduction

Given a person's healthcare data related to a certain disease, we can train a logistic regression (LR) model capable of telling whether or not this person is likely to develop this disease. However, such personal health information is highly private to individuals. The privacy concern, therefore, becomes a major obstacle for individuals to share their biomedical data. The most secure solution is to encrypt the data into ciphertexts first by Homomorphic Encryption (HE) and then securely outsource the ciphertexts to the cloud, without allowing the cloud to access the data directly. iDASH is an annual competition that aims to call for implementing interesting cryptographic schemes in a biological context. Since 2014, iDASH has included the theme of genomics and biomedical privacy. The third track of the 2017 iDASH competition and the second track of the 2018 iDASH competition were both to develop homomorphic-encryption-based solutions for building an LR model over encrypted data.

Several studies on logistic regression models are based on homomorphic encryption. Kim et al. [14] 26

discussed the problem of performing LR training in an encrypted environment. They used the full 27

batch gradient descent in the training process and the least-squares method to get the approximation 28

of the sigmoid function. In the iDASH 2017 competition, Bonte and Vercauteren [4], Kim et al. [12], 29

Chen et al. [5], and Crawford et al. [8] all investigated the same problem that Kim et al. [14] studied. 30

In the iDASH competition of 2018, Kim et al. [13] and Blatt et al. [2] further worked on it for an 31

efficient packing and semi-parallel algorithm. The papers most relevant to this work are [4] and [12]. 32

Bonte and Vercauteren [4] developed a practical algorithm called the simplified fixed Hessian (SFH) 33

method. Our study complements their work and adopts the ciphertext packing technique proposed by 34

Kim et al. [12] for efficient homomorphic computation. 35

Our specific contributions in this paper are as follows: 36

37

38

We propose a new gradient variant, /a113✉❛❞/a114❛/a116✐❝ ❣/a114❛❞✐❡♥/a116 , which can combine the firstorder gradient methods and the second-order Newton-Raphson method as one.

39

40

41

42

43

44

We develop two enhanced gradient methods by equipping the original methods with /a113✉❛❞/a114❛/a116✐❝ ❣/a114❛❞✐❡♥/a116 . The resulting methods show a state-of-the-art performance in the convergence speed.

We adopt the enhanced NAG method to implement privacy-preserving logistical regression training, to our best knowledge, which seems to be the best candidate without compromising much on computation and storage.

2 Preliminaries 45

We adopt the square brackets ' [ ] ' to denote the index of a vector or matrix element in what follows. 46 For example, for a vector v ∈ R ( n ) and a matrix M ∈ R m × n , v [ i ] or v [ i ] means the i -th element of 47 vector v and M [ i ][ j ] or M [ i ][ j ] the j -th element in the i -th row of M . 48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

2.1 Fully Homomorphic Encryption

Fully Homomorphic Encryption (FHE) is a type of cryptographic scheme that can be used to compute an arbitrary number of additions and multiplications directly on the encrypted data. It was not until 2009 that Gentry constructed the first FHE scheme via a bootstrapping operation [9]. FHE schemes themselves are computationally time-consuming; the choice of dataset encoding matters likewise to the efficiency. In addition to these two limits, how to manage the magnitude of plaintext [11] also contributes to the slowdown. Cheon et al. [6] proposed a method to construct an HE scheme with a /a114❡/a115❝❛❧✐♥❣ procedure which could eliminate this technical bottleneck effectively. We adopt their open-source implementation ❍❊❆❆◆ while implementing our homomorphic LR algorithms. It is inevitable to pack a vector of multiple plaintexts into a single ciphertext for yielding a better amortized time of homomorphic computation. ❍❊❆❆◆ supports a parallel technique (aka ❙■▼❉ ) to pack multiple numbers in a single polynomial by virtue of the Chinese Remainder Theorem and provides rotation operation on plaintext slots. The underlying HE scheme in ❍❊❆❆◆ is well described in [12, 14, 10].

2.2 Database Encoding Method

Kim et al. [12] proposed an efficient and promising database-encoding method by using ❙■▼❉ technique, which could make full use of the computation and storage resources. Suppose that a database has a training dataset consisting of n samples with (1 + d ) covariates, they packed the training dataset Z into a single ciphertext in a row-by-row manner.

Using this encoding scheme, we can manipulate the data matrix Z by performing HE operations on the ciphertext Enc [ Z ] , with the help of only three HE operations - rotation, addition and multiplication.

Han et al. [10] introduced several operations to manipulate the ciphertexts, such as a procedure named ' ❙✉♠❈♦❧❱❡❝ ' to compute the summation of the columns of a matrix. By dint of these basic operations, more complex calculations such as computing the gradients in logistic regression models are achievable.

2.3 Logistic Regression

Logistic regression is widely used in binary classification tasks to infer whether a binary-valued 75 variable belongs to a certain class or not. LR can be generalized from linear regression [15] by 76 mapping the whole real line ( β T x ) to (0 , 1) via the sigmoid function σ ( z ) = 1 / (1+exp( -z )) , where 77 the vector β ∈ R (1+ d ) is the main parameter of LR and the vector x = (1 , x 1 , . . . , x d ) ∈ R (1+ d ) the 78 input covariate. Thus logistic regression can be formulated with the class label y ∈ {± 1 } as follows: 79

LR sets a threshold (usually 0 . 5 ) and compares its output with it to decide the resulting class label. 80

The logistic regression problem can be transformed into an optimization problem that seeks a param81 eter β to maximize L ( β ) = ∏ n i =1 Pr( y i | x i , β ) or its log-likelihood function l ( β ) for convenience in 82 the calculation: 83

where n is the number of examples in the training dataset. LR does not have a closed form of 84 maximizing l ( β ) and two main methods are adopted to estimate the parameters of an LR model: 85 (a) gradient descent method via the gradient; and (b) Newton's method by the Hessian matrix. The 86 gradient and Hessian of the log-likelihood function l ( β ) are given by, respectively: 87

where S is a diagonal matrix with entries S ii = ( σ ( y i β T x i ) -1) σ ( y i β T x i ) and X the dataset. 88

The log-likelihood function l ( β ) of LR has at most a unique global maximum [1], where its gradient 89 is zero. Newton's method is a second-order technique to numerically find the roots of a real-valued 90 differentiable function, and thus can be used to solve the β in ∇ β l ( β ) = 0 for LR. 91

3 Technical Details 92

It is quite time-consuming to compute the Hessian matrix and its inverse in Newton's method for each iteration. One way to limit this downside is to replace the varying Hessian with a fixed matrix ¯ H . This novel technique is called the fixed Hessian Newton's method. Böhning and Lindsay [3] have shown that the convergence of Newton's method is guaranteed as long as ¯ H ≤ ∇ 2 β l ( β ) , where ¯ H is a symmetric negative-definite matrix independent of β and ' ≤ ' denotes the Loewner ordering in the sense that the difference ∇ 2 β l ( β ) -¯ H is non-negative definite. With such a fixed Hessian matrix ¯ H , the iteration for Newton's method can be simplified to:

Böhning and Lindsay also suggest the fixed matrix ¯ H = -1 4 X T X is a good lower bound for the 93 Hessian of the log-likelihood function l ( β ) in LR. 94

3.1 the Simplified Fixed Hessian method 95

Bonte and Vercauteren [4] simplify this lower bound ¯ H further due to the need for inverting the fixed Hessian in the encrypted domain. They replace the matrix ¯ H with a diagonal matrix B whose diagonal elements are simply the sums of each row in ¯ H . They also suggest a specific order of calculation to get B more efficiently. Their new approximation B of the fixed Hessian is:

where ¯ h ki is the element of ¯ H . This diagonal matrix B is in a very simple form and can be obtained 96 from ¯ H without much difficulty. The inverse of B can be approximated in the encrypted form by 97

means of computing the inverse of every diagonal element of B via the iterative of Newton's method 98 with an appropriate start value. Their simplified fixed Hessian method can be formulated as follows: 99

100

where b ii is the reciprocal of ∑ d i =0 ¯ h 0 i and ∇ i is the element of ∇ β l ( β ) .

Consider a special situation: if b 00 , . . . , b dd are all the same value -η with η > 0, the iterative formula 101 of the SFH method can be given as: 102

which is the same as the formula of the naive gradient ascent method. Such coincident is just what the idea behind this work comes from: there is some relation between the Hessian matrix and the learning rate of the gradient (descent) method. We consider b ii · ∇ i as a new enhanced gradient variant's element and assign a new learning rate to it. As long as we ensure that this new learning rate decreases from a positive floating-point number greater than 1 (such as 2) to 1 in a bounded number of iteration steps, the fixed Hessian Newton's method guarantees the algorithm will converge eventually.

The SFH method proposed by Bonte and Vercauteren [4] has two limitations: (a) in the construction of the simplified fixed Hessian matrix, all entries in the symmetric matrix ¯ H need to be non-positive. For machine learning applications the datasets will be in advance normalized into the range [0,1], meeting the convergence condition of the SFH method. However, for other cases such as numerical optimization, it doesn't always hold; and (b) the simplified fixed Hessian matrix B that Bonte and Vercauteren [4] constructed, as well as the fixed Hessian matrix ¯ H = -1 4 X T X , can still be singular, especially when the dataset is a high-dimensional sparse matrix, such as the MNIST datasets. We extend their work by removing these limitations so as to generalize this simplified fixed Hessian to be invertible in any case and propose a faster gradient variant, which we term /a113✉❛❞/a114❛/a116✐❝ ❣/a114❛❞✐❡♥/a116 .

3.2 Quadratic Gradient

Suppose that a differentiable scalar-valued function F ( x ) has its gradient g and Hessian matrix H , with any matrix ¯ H ≤ H in the Loewner ordering for a maximization problem as follows:

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

where ∇ 2 ij = ∇ 2 ji = ∂ 2 F ∂x i ∂x j . We construct a new Hessian matrix ˜ B as follows: 122

where ε is a small positive constant to avoid division by zero (usually set to 1 e -8 ). 123

As long as ˜ B satisfies the convergence condition of the above fixed Hessian method, ˜ B ≤ H , we 124 can use this approximation ˜ B of the Hessian matrix as a lower bound. Since we already assume that 125 ¯ H ≤ H , it will suffice to show that ˜ B ≤ ¯ H . We prove ˜ B ≤ ¯ H in a similar way that [4] did. 126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

Lemma 1. Let A ∈ R n × n be a symmetric matrix, and let B be the diagonal matrix whose diagonal entries B kk = -ε -∑ i | A ki | for k = 1 , . . . , n , then B ≤ A .

glyph[negationslash]

Proof. By definition of the Loewner ordering, we have to prove the difference matrix C = A -B is non-negative definite, which means that all the eigenvalues of C need to be non-negative. By construction of C we have that C ij = A ij + ε + ∑ n k =1 | A ik | for i = j and C ij = A ij for i = j . By means of Gerschgorin's circle theorem, we can bound every eigenvalue λ of C in the sense that | λ -C ii | ≤ ∑ i = j | C ij | for some index i ∈ { 1 , 2 , . . . , n } . We conclude that λ ≥ A ii + ε + | A ii | ≥ ε > 0 for all eigenvalues λ and thus that B ≤ A .

glyph[negationslash]

Definition 3.1 ( ◗✉❛❞/a114❛/a116✐❝ ●/a114❛❞✐❡♥/a116 ) . Given such a ˜ B above, we define the quadratic gradient as G = ¯ B · g with a new learning rate η , where ¯ B is a diagonal matrix with diagonal entries ¯ B kk = 1 / | ˜ B kk | , and η should be always no less than 1 and decrease to 1 in a limited number of iteration steps. Note that G is still a column vector of the same size as the gradient g . To maximize the function F ( x ) , we can use the iterative formulas: x k +1 = x k + η · G , just like the naive gradient. To minimize the function F ( x ) is the same as to just maximize the function -F ( x ) , in which case we need to construct the ˜ B by any good lower bound ¯ H of the Hessian -H of -F ( x ) or any good upper bound ¯ H of the Hessian H of F ( x ) . We point out here that ¯ H could be the Hessian matrix H itself.

In our experiments, we use ¯ H = -1 4 X T X to construct our ˜ B .

3.3 Two Enhanced Methods

◗✉❛❞/a114❛/a116✐❝ ●/a114❛❞✐❡♥/a116 can be used to enhance NAG and Adagrad.

NAG is a different variant of the momentum method to give the momentum term much more prescience. The iterative formulas of the gradient ascent method for NAG are as follows:

where V t +1 is the intermediate variable used for updating the final weight β t +1 and γ t ∈ (0 , 1) is a smoothing parameter of moving average to evaluate the gradient at an approximate future position [12]. The enhanced NAG is to replace (3) with V t +1 = β t + (1 + α t ) · G . Our enhanced NAG method is described in Algorithm 1 .

Adagrad is a gradient-based algorithm suitable for dealing with sparse data. The updated operations 152 of Adagrad and its quadratic-gradient version, for every parameter β [ i ] at each iteration step t , are as 153 follows, respectively: 154

Performance Evaluation We evaluate the performance of various algorithms in the clear using the 155 Python programming language on the same desktop computer with an Intel Core CPU G640 at 156 1.60 GHz and 7.3 GB RAM. Since our focus is on how fast the algorithms converge in the training 157 phase, the loss function, maximum likelihood estimation (MLE), is selected as the only indicator. We 158 evaluate four algorithms, NAG, Adagrad, and their quadratic-gradient versions (denoted as Enhanced 159 NAG and Enhanced Adagrad, respectively) on the datasets that Kim et al. [12] adopted: the iDASH 160 genomic dataset (iDASH), the Myocardial Infarction dataset from Edinburgh (Edinburgh), Low Birth 161 weight Study (lbw), Nhanes III (nhanes3), Prostate Cancer study (pcs), and Umaru Impact Study 162 datasets (uis). The genomic dataset is provided by the third task in the iDASH competition of 2017, 163 which consists of 1579 records. Each record has 103 binary genotypes and a binary phenotype 164 indicating if the patient has cancer. The other five datasets all have a single binary dependent variable. 165 Figures 1 and 2 show that except for the enhanced Adagrad method on the iDASH genomic dataset 166 our enhanced methods all converge faster than their original ones in other cases. In all the Python 167

Algorithm 1 The Enhanced Nesterov's Accelerated Gradient method

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

0

20

40

-

2

-

1

0

·

10

4

Iteration Number

Maximum Likelihood Estimation

Adagrad

Enhanced Adagrad

(a) The iDASH dataset

0

20

40

-

800

-

600

-

400

-

200

Iteration Number

Adagrad

Enhanced Adagrad

0

20

40

-

140

-

120

-

100

Iteration Number

Adagrad

Enhanced Adagrad

(c) The lbw dataset

0

20

40

-

2

-

1

.

5

-

1

-

0

.

5

·

10

4

Iteration Number

Maximum Likelihood Estimation

Adagrad

Enhanced Adagrad

0

20

40

-

400

-

300

-

200

Iteration Number

Adagrad

Enhanced Adagrad

(e) The pcs dataset

0

20

40

-

500

-

400

-

300

Iteration Number

Adagrad

Enhanced Adagrad

(f) The uis dataset

experiments, the time to calculate the ¯ B in quadratic gradient G before running the iterations and the time to run each iteration for various algorithms are negligible (few seconds).

Results Analysis In Figure 1a, the enhanced Adagrad algorithm failed to outperform the original Adagrad algorithm. The possible reason for that might be related to the limitations of the raw Adagrad method. Without a doubt, Adagrad is a novel algorithm initiated to accelerate each element of the gradient with different learning rates. However, Adagrad tends to converge to a suboptimal solution due to its aggressive, monotonically decreasing learning rates. This would lead to its main limitation that in the later training phase every learning rate for different components of the gradient is too close to zero due to keeping adding positive additional terms to the denominator, stopping the algorithm from learning anything.

On the other hand, the original Adagrad method has another little-noticed limitation: the learning rate in the first few iterations tends to be large. While this limitation does not affect the performance of the original Adagrad method to some extent, the enhanced Adagrad method exacerbates this phenomenon

the enhanced Adagrad [7] cannot be applied to general optimization problems such as Rosenbrock's function. The exploding learning rate would be too large for the algorithm to survive the first several iterations, finally leading the optimization function to some point where its output cannot be represented by the computer system. This might explain why the performance of this algorithm in all cases, not just on the iDASH genome dataset, seems to be meaningless, numerically unstable, and fluctuates in the first few iterations.

Several improved algorithms upon the Adagrad method, such as RMSProp, have been proposed in 188 order to address these issues existed, via using an exponential moving average of historical gradients 189 rather than just the sum of all squared gradients from the beginning of training. We might be able 190 to overcome the problems existing in the enhanced Adagrad method by adopting the enhanced 191 Adagrad-like variants, like the enhanced Adadelta method and the enhanced RMSProp method. One 192 research work that could confirm this hypothesis is the enhanced Adam method [7]. 193

0

20

40

-

8

,

000

-

6

,

000

-

4

,

000

-

2

,

000

Iteration Number

Maximum Likelihood Estimation

NAG

Enhanced NAG

(a) The iDASH dataset

0

20

40

-

800

-

600

-

400

-

200

Iteration Number

NAG

Enhanced NAG

0

20

40

-

130

-

120

-

110

-

100

Iteration Number

NAG

Enhanced NAG

(c) The lbw dataset

0

20

40

-

1

.

2

-

1

-

0

.

8

-

0

.

6

·

10

4

Iteration Number

Maximum Likelihood Estimation

NAG

Enhanced NAG

0

20

40

-

260

-

240

-

220

-

200

Iteration Number

NAG

Enhanced NAG

(e) The pcs dataset

0

20

40

-

400

-

380

-

360

-

340

-

320

Iteration Number

NAG

Enhanced NAG

(f) The uis dataset

4 Privacy-preserving LR Training 194

Adagrad method is not a practical solution for homomorphic LR due to its frequent inversion 195 operations. It seems plausible that the enhanced NAG is probably the best choice for privacy196 preserving LR training. We adopt the enhanced NAG method to implement privacy-preserving 197 logistic regression training. The difficulty in applying the quadratic gradient is to invert the diagonal 198 matrix ˜ B in order to obtain ¯ B . We leave the computation of matrix ¯ B to data owner and let the 199 data owner upload the ciphertext encrypting the ¯ B to the cloud. Since data owner has to prepare the 200 dataset and normalize it, it would also be practicable for the data owner to calculate the ¯ B owing to 201 no leaking of sensitive data information. 202

203

204

205

206

207

208

209

210

Privacy-preserving logistic regression training based on HE techniques faces a difficult dilemma that no homomorphic schemes are capable of directly calculating the sigmoid function in the LR model. A common solution is to replace the sigmoid function with a polynomial approximation by using the widely adopted least-squares method. We can call a function named ' ♣♦❧②❢✐/a116✭ · ✮ ' in the Python package Numpy to fit the polynomial in a least-square sense. We adopt the degree 5 polynomial approximation g ( x ) by which Kim et al. [12] used the least square approach to approximate the sigmoid function over the domain [ -8 , 8] : g ( x ) = 0 . 5+0 . 19131 · x -0 . 0045963 · x 3 +0 . 0000412332 · x 5 .

211

212

213

214

215

Given the training dataset X ∈ R n × (1+ d ) and training label Y ∈ R n × 1 , we adopt the same method that Kim et al. [12] used to encrypt the data matrix consisting of the training data combined with training-label information into a single ciphertext ct Z . The weight vector β (0) consisting of zeros and the diagnoal elements of ¯ B are copied n times to form two matrices. The data owner then encrypt the two matrices into two ciphertexts ct (0) β and ct ¯ B , respectively.

The pulbic cloud takes the three ciphertexts ct Z , ct (0) β and ct ¯ B and evaluates the enhanced NAG 216 algorithm to find a decent weight vector by updating the vector ct (0) β . Refer to [12] for a detailed 217 description about how to calculate the gradient by HE programming. 218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

5 Experiments

Implementation We implement the enhanced NAG based on HE with the library ❍❊❆❆◆ . The C++ source code is publicly available at ❤/a116/a116♣/a115✿✴✴❛♥♦♥②♠♦✉/a115✳✹♦♣❡♥✳/a115❝✐❡♥❝❡✴/a114✴■❉❆❙❍✷✵✶✼✲✷✹✺❇ . All the experiments on the ciphertexts were conducted on a public cloud with 32 vCPUs and 64 GB RAM.

For a fair comparison with [12], we utilized the same 10-fold cross-validation (CV) technique on the same iDASH dataset consisting of 1579 samples with 18 features and the same 5-fold CV technique on the other five datasets. Like [12], We consider the average accuracy and the Area Under the Curve (AUC) as the main indicators. Tables 1 and 2 show the two experiment results, respectively. The two tables also provide the average evaluation running time for each iteration and the storage (encrypted dataset for the baseline work and encrypted dataset and ¯ B for our method). We adopt the same packing method that Kim et al. [12] proposed and hence our solution has similar storage of ciphertexts to [12] with some extra ciphertexts to encrypt the ¯ B .

The parameters of ❍❊❆❆◆ we set are same to [12]: logN = 16 , logQ = 1200 , logp = 30 , slots = 32768 , which ensure the security level λ = 80 . Refer [12] for the details of these parameters. Since our enhanced NAG method need to consume more modulus to preserve the precision of ¯ B , we use logp = 60 to encrypt the matrix ¯ B and thus only can perform 3 iterations of the enhanced NAG method. Yet despite only 3 iterations, our enhanced NAG method still produces a comparable result.

6 Conclusion

In this paper, we proposed a faster gradient variant called /a113✉❛❞/a114❛/a116✐❝ ❣/a114❛❞✐❡♥/a116 , and implemented the quadratic-gradient version of NAG in the encrypted domain to train the logistic regression model.

The quadratic gradient presented in this work can be constructed from the Hessian matrix directly, and thus somehow combines the second-order Newton's method and the first-order gradient (descent) method together. There is a good chance that quadratic gradient could accelerate other gradient methods such as Adagrad, Adadelta, RMSprop, Adam [8], AdaMax and Nadam, which is an open future work.

Also, /a113✉❛❞/a114❛/a116✐❝ ❣/a114❛❞✐❡♥/a116 might substitute and supersede the line-search method, for example when using enhanced Adagrad-like methods, and could use gradient descent methods to accelerate Newton's method, resulting in super-quadratic algorithms.

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

References

Allison, P. D. (2008). Convergence failures in logistic regression.

Blatt, M., Gusev, A., Polyakov, Y., Rohloff, K., and Vaikuntanathan, V. (2019). Optimized homomorphic encryption solution for secure genome-wide association studies. IACR Cryptology ePrint Archive , 2019:223.

Böhning, D. and Lindsay, B. G. (1988). Monotonicity of quadratic-approximation algorithms. Annals of the Institute of Statistical Mathematics , 40(4):641-663.

Bonte, C. and Vercauteren, F. (2018). Privacy-preserving logistic regression training. BMC medical genomics , 11(4):86.

Chen, H., Gilad-Bachrach, R., Han, K., Huang, Z., Jalali, A., Laine, K., and Lauter, K. (2018). Logistic regression over encrypted data from fully homomorphic encryption. BMC medical genomics , 11(4):3-12.

Cheon, J. H., Kim, A., Kim, M., and Song, Y. (2017). Homomorphic encryption for arithmetic of approximate numbers. In International Conference on the Theory and Application of Cryptology and Information Security , pages 409-437. Springer.

Chiang, J. (2022). Quadratic gradient: Uniting gradient algorithm and newton method as one. arXiv preprint arXiv:2209.03282 .

Crawford, J. L., Gentry, C., Halevi, S., Platt, D., and Shoup, V . (2018). Doing real work with fhe: the case of logistic regression. In Proceedings of the 6th Workshop on Encrypted Computing & Applied Homomorphic Cryptography , pages 1-12.

Gentry, C. (2009). Fully homomorphic encryption using ideal lattices. In Proceedings of the forty-first annual ACM symposium on Theory of computing , pages 169-178.

Han, K., Hong, S., Cheon, J. H., and Park, D. (2019). Logistic regression on homomorphic encrypted data at scale. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 9466-9471.

Jäschke, A. and Armknecht, F. (2016). Accelerating homomorphic computations on rational numbers. In International Conference on Applied Cryptography and Network Security , pages 405-423. Springer.

Kim, A., Song, Y., Kim, M., Lee, K., and Cheon, J. H. (2018a). Logistic regression model training based on the approximate homomorphic encryption. BMC medical genomics , 11(4):83.

Kim, M., Song, Y., Li, B., and Micciancio, D. (2019). Semi-parallel logistic regression for gwas on encrypted data. IACR Cryptology ePrint Archive , 2019:294.

Kim, M., Song, Y., Wang, S., Xia, Y., and Jiang, X. (2018b). Secure logistic regression based on homomorphic encryption: Design and evaluation. JMIR medical informatics , 6(2):e19.

Murphy, K. P. (2012). Machine learning: a probabilistic perspective . The MIT Press, Cambridge, MA.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a second-order version of Nesterov's accelerated gradient (NAG) descent and Adagrad for logistic regression by incorporating an approximation to the Hessian. The authors call this the \"quadratic gradient\". Specifically, a diagonal approximation to the Hessian for logistic regression is proposed. Some empirical results are shown to illustrate the benefit of the proposed method over vanilla NAG and Adagrad.\n\nSTRENGTHS:\nThe proposed approximation of the diagonal Hessian may be interesting for NAG.\n\nWEAKNESSES:\nUnfortunately, this paper has several weaknesses.\n\n**1.** **Limited novelty**: The proposed approximation to the Hessian in Section 3.2 seems like a trivial and incremental extension of the idea of reference [4] discussed in Section 3.1. \n\n**2.** **Lack of clarity**: The proposed methods are unclear to me and the presentation needs to be heavily improved.\n\n* The enhanced NAG method described in line 150 is unclear to me -- what is $G$ here and is $\\alpha_t$ the step-size here? Moreover, Algorithm 1 seems different from the discussions in Section 3.3. What are $\\alpha_0$ and $\\alpha_1$? They don't look like the quantity $\\alpha_t$ introduced in Section 3.3. Why is $\\alpha_1$ chosen to be $0.5(1 + \\sqrt{1 + 4 \\alpha_0^2})$? I don’t understand lines 31 and 37 in Algorithm 1 and what are $\\gamma$ and $\\eta$ here? What is the role of $W$ in lines 34 and 35 – it is not being used at all. In summary, the enhanced NAG method/Algorithm 1 has been presented poorly and I don't understand the method at all. \n\n* What is the enhanced Adagrad algorithm? The two equations after line 154 which are supposed to explain the enhancement are not very clear. What is the difference between $G^{(t)}$ and $g^{(t)}$ in these equations? Is $G = \\tilde{B}^{-1} g$ here? There is no algorithm summarizing it like Algorithm 1 for enhanced NAG. Also, suddenly for Adagrad, the authors have a negative sign in front of the gradient which corresponds to minimizing the function whereas for NAG and the previous discussions in the paper, maximizing the function has been considered. Please stick to either minimization or maximization for consistency.\n\n**3.** **Premise of enhanced Adagrad:** One way to interpret Adagrad is that it tries to maintain a diagonal approximation of the Hessian inverse and applies it to the gradients (a.k.a. preconditioning). So I'm not sure why applying a second approximation of the Hessian inverse on the *already preconditioned* gradients makes sense intuitively. Additionally, the authors themselves point out that enhanced Adagrad cannot be applied to general optimization problems (line 182) due to \"learning-rate explosion\". Then why introduce this method at all?\n\n**4.** **Setup and experiments**: The datasets on which experiments are performed are not standard benchmarking datasets in the ML community and appear to have very few features (looking at Table 2). There are no test set statistics provided. I'd be more convinced if the authors showed empirical results in a *standard logistic regression setup without any kind of encryption* (which frankly seems irrelevant to me in this paper) on benchmarking ML datasets.\n\n----\n\n*Some general comments*: The introduction on logistic regression can be compressed. It is standard to consider the *negative* log-likelihood objective and apply gradient *descent* to minimize it. A couple of small typos -- in line 96, I guess it should be \"$\\bar{h}_{k i}$ is the $k^\\text{th}$ element in the $i^\\text{th}$ row of the Hessian\" and in line 216, it should be \"public\".\n\nQUESTIONS:\nPlease see the Weaknesses.\n\nLIMITATIONS:\nNot in too much detail but as I mentioned in Weaknesses, the authors point out that enhanced Adagrad cannot be applied to general optimization problems. No foreseeable negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new approach to improve the gradient used by first-order optimization methods in logistic regression by utilizing a constant bound to the Hessian matrix. The authors demonstrate how to use their method under a fully Homomorphic-Encryption scenario. They test their method on many real-world datasets under non-private settings and Homomorphic-Encryption settings.\n\nSTRENGTHS:\n1. The paper is written clearly.\n2. The experiments are all using real-world datasets which have strong practical implications.\n\nWEAKNESSES:\n1. The `quadratic gradient` method is not very new. As the authors have mentioned in Section 3.1 (Line 95), most parts of the method were proposed by Bonte and Vercauteren. I understand that the missing non-negative restriction is important for using the convergence results by Böhning and Lindsay (Line 92). However, using the absolute value is rather an straightforward solution. \n* A more interesting and critical question remaining to be answered is why this proposed `quadratic gradient` method is faster as the authors claimed in the conclusion (Line 238). \n* Another problem with this method is its usage being restricted in logistic regression: The authors provided a choice of $\\bar{H}$ for logistic regression, while it may be very hard to generalize it to other problems, especially neural network training. \n2. The experiments have not shown much advantage of using the `quadratic gradient` method. In Table 1 and Table 2, the accuracy and AUC of the proposed method are almost always lower than the compared baseline method [12]. I understand that the learning time is reduced, but it was not a main problem in [12] as shown in the tables, and we are not sure if there is a tradeoff between the learning time and the accuracy.\n* The description of the experiment details in Section 5 is very short. The authors suggest the readers refer to [12]. I think it would be better to have the details in supplementary and a discussion of the weaknesses of [12] in these experiment settings, along with why the proposed method solves those weaknesses.\n\nMinor weaknesses:\n1. The maximum likelihood estimation (MLE) is commonly referring to the estimate for $\\beta$. The value of the loss function is the negative log-likelihood. That said, the y-axis of the figures could be corrected. Also, the objective function is usually the mean of the loss for each data point, not the sum.\n2. Typo: Line 216, pulbic -> public.\n---\nI have read the rebuttal which answered my questions but did not fully address my concerns on the weakness.\n\nQUESTIONS:\n1. The cited paper [7] also used `quadratic gradient`. Is it a following up paper to this paper? If yes, it is better to have a dedicated discussion of the relationship between this paper and [7], e.g. about the difference and the novelty.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes the quadratic gradient for privacy-preserving logistic regression. Such gradient is used together with Nesterov’s accelerated gradient (NAG) and Adagrad on Homomorphic Encryption techniques.

STRENGTHS:
This paper tackles the privacy-preserving (in the sense of encryption) logistic regression. The paper is clear with introduction and motivation. The algorithm is easy-to-follow and well-supported by experiments.

WEAKNESSES:
Overall, it is hard to understand the privacy concerns in this work as there are many privacy-preserving techniques available. Without an example or experiment of privacy attack, the motivation of using HE in the first place is not backed up. In addition, only [12] is compared in the empirical results. While the new method seems promising, the lack of other baselines makes it hard to understand the limitations and benefits of the new algorithm.

QUESTIONS:
See weakness.

LIMITATIONS:
NA.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new gradient method that can be efficiently used under homomorphic encryption. The proposed method replaces the gradient $g$ by an approximation of $H^{-1}g$, where $H$ is the Hessian. This approximation is done using a specific diagonal matrix, that speeds up convergence while being possible to use under homomorphic encryption.\n\nSTRENGTHS:\n1. The paper proposes a new method for privacy-preserving logistic regression under FHE, that achieves reasonable results with less computation than existing methods.\n2. The proposed method is quite versatile as it can be applied to a variety of optimization algorithms.\n\nWEAKNESSES:\n1. Experimental results are not convincing. Contrary to the paper's claims, the proposed method often performs way worse than existing ones (especially on iDASH, Edunburgh and pcs). Datasets are also very small, and therefore do not account for how the method scales with the dimension. Given this is an empirical paper, it seems a bit insufficient.\n2. The fixed-Hessian method seems to be closer to preconditioning (see e.g. [1]), where gradients are linearly transformed before being used, than to a proper second order method.\n3. The proposed method does not seem to be too novel. In particular, the paper refers to [2] (which itself refers to a paper with the same title as this manuscript), which proposes a very similar method.\n\n[1] Preconditioned Stochastic Gradient Descent, Xi-Lin Li 2015.\n[2] Quadratic Gradient: Uniting gradient algorithm and newton method as one, by Chiang, 2022.\n\nQUESTIONS:\n1. Are there settings where the difference in runtime between the proposed method and existing ones is more significant? What would cause this difference to get larger?\n2. When normalizing the data, this has an impact on the Hessian matrix (and therefore on $\\bar B$). In particular, one could imagine that after normalization, $X^T X$ is close to the identity. Could you comment on the effect of normalization on $\\bar B$?\n3. Line 108, it is claimed that with proper learning rate, the fixed Hessian Netwon method converges: can you give evidence/proof to support this claim?\n\nLIMITATIONS:\nLimitations are not discussed in the paper. In particular, experiments only consider a specific setting, with a choice of parameters that seems arbitrary (e.g., 3 vs. 7 iterations), and some claims are not supported with evidence.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-21"}