Title: value: Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant
Authors: value: Li-Yue Sun
Venue: NeurIPS 2023
Keywords: value: Privacy-Preserving
Logistic Regression
Quadratic Gradient
Simplified Fixed Hessian
Numerical Optimization

Abstract: value: Logistic regression training over encrypted data has been an attractive idea to security concerns for years. In this paper, we propose a faster gradient variant called $\texttt{quadratic gradient}$ for privacy-preserving logistic regression training. The core of $\texttt{quadratic gradient}$ can be seen as an extension of the simplified fixed Hessian.  

We enhance  Nesterov's accelerated gradient (NAG) and Adaptive Gradient Algorithm (Adagrad) respectively with $\texttt{quadratic gradient}$ and evaluate the enhanced algorithms on several datasets. 
Experiments show that the enhanced methods have a state-of-the-art performance in convergence speed compared to the raw first-order gradient methods. We then  adopt the enhanced NAG method to implement homomorphic logistic regression training, obtaining a comparable result  by only $3$ iterations.

There is a promising chance that $\texttt{quadratic gradient}$ could be used to enhance other first-order gradient methods for general numerical optimization problems.


Review discussion:
The proposed approximation of the diagonal Hessian may be interesting for NAG.
Unfortunately, this paper has several weaknesses.
**1.** **Limited novelty**: The proposed approximation to the Hessian in Section 3.2 seems like a trivial and incremental extension of the idea of reference [4] discussed in Section 3.1.
**2.** **Lack of clarity**: The proposed methods are unclear to me and the presentation needs to be heavily improved.
* The enhanced NAG method described in line 150 is unclear to me -- what is $G$ here and is $\alpha_t$ the step-size here?
Moreover, Algorithm 1 seems different from the discussions in Section 3.3. What are $\alpha_0$ and $\alpha_1$? They don't look like the quantity $\alpha_t$ introduced in Section 3.3. Why is $\alpha_1$ chosen to be $0.5(1 + \sqrt{1 + 4 \alpha_0^2})$? I don’t understand lines 31 and 37 in Algorithm 1 and what are $\gamma$ and $\eta$ here? What is the role of $W$ in lines 34 and 35 – it is not being used at all.
In summary, the enhanced NAG method/Algorithm 1 has been presented poorly and I don't understand the method at all.
* What is the enhanced Adagrad algorithm? The two equations after line 154 which are supposed to explain the enhancement are not very clear. What is the difference between $G^{(t)}$ and $g^{(t)}$ in these equations? Is $G = \tilde{B}^{-1} g$ here? There is no algorithm summarizing it like Algorithm 1 for enhanced NAG.
Also, suddenly for Adagrad, the authors have a negative sign in front of the gradient which corresponds to minimizing the function whereas for NAG and the previous discussions in the paper, maximizing the function has been considered. Please stick to either minimization or maximization for consistency.
**3.** **Premise of enhanced Adagrad:** One way to interpret Adagrad is that it tries to maintain a diagonal approximation of the Hessian inverse and applies it to the gradients (a.k.a. preconditioning).
So I'm not sure why applying a second approximation of the Hessian inverse on the *already preconditioned* gradients makes sense intuitively.
Additionally, the authors themselves point out that enhanced Adagrad cannot be applied to general optimization problems (line 182) due to "learning-rate explosion".
Then why introduce this method at all?
**4.** **Setup and experiments**: The datasets on which experiments are performed are not standard benchmarking datasets in the ML community and appear to have very few features (looking at Table 2). There are no test set statistics provided. I'd be more convinced if the authors showed empirical results in a *standard logistic regression setup without any kind of encryption* (which frankly seems irrelevant to me in this paper) on benchmarking ML datasets.
*Some general comments*: The introduction on logistic regression can be compressed. It is standard to consider the *negative* log-likelihood objective and apply gradient *descent* to minimize it. A couple of small typos -- in line 96, I guess it should be "$\bar{h}_{k i}$ is the $k^\text{th}$ element in the $i^\text{th}$ row of the Hessian" and in line 216, it should be "public".
Please see the Weaknesses.
Not in too much detail but as I mentioned in Weaknesses, the authors point out that enhanced Adagrad cannot be applied to general optimization problems. No foreseeable negative societal impact.
No ethics review needed.
$\textbf{Response}$
We would like to thank the reviewers for their input. Their comments have been thoroughly considered, and altering the manuscript in accordance with these comments will significantly improve the quality of our paper in the next submission.
**C1: The enhanced NAG method described in line 150 is unclear to me -- what is  $G$ here and is  $\alpha_t$ the step-size here?
Moreover, Algorithm 1 seems different from the discussions in Section 3.3. What are $\alpha_0$ and $\alpha_1$? They don't look like the quantity $\alpha_t$  introduced in Section 3.3. Why is $\alpha_1$ chosen to be $0.5(1 + \sqrt{1 + 4\alpha_0^2})$
? I don’t understand lines 31 and 37 in Algorithm 1 and what are  $\gamma$ and $\eta$ here? What is the role of $W$ in lines 34 and 35 – it is not being used at all.
A1: We apologize for not clearly presenting the proposed method. $G$ here refers to the quadratic gradient, namely $\bar B g$, and $\alpha_t$ here is the step size (the learning rate). We misused the symbol in the Algorithm 1 and the discussion in Section 3.3. It seems that NAG has many variants and we just adopt the one chosen by the baseline work. In this variant,  $\alpha_0$ and $\alpha_1$ are its parameters, and $\alpha_1$ is set to be $0.5(1 + \sqrt{1 + 4\alpha_0^2})$.   $\gamma$ is also the parameter of this NAG variant.  $\alpha_0$, $\alpha_1$ and $\gamma$  are commonly set in this way.  $\eta$ is the step size (the learning rate) in Algorihtm1.
NAG computed the expected weight of the current iteration by using the previous weight.
Hence, we need an intermediate to store the last-step weight, which is the role of $W$ in lines 34 and 35. It has been used to update the current weight in line 34.
**C2: What is the enhanced Adagrad algorithm? The two equations after line 154 which are supposed to explain the enhancement are not very clear. What is the difference between $G^{(t)}$  and $g^{(t)}$ in these equations? Is $G = \tilde B^{-1} g$
here?
A2: The enhanced Adagrad algorithm is similar to the raw Adagrad algorithm. $G^{(t)}$ refer to the quadratic gradient at step $t$ while $g^{(t)}$  means the raw gradient at the $t$-$th$ step. Yes, $G$ here is  $\tilde B^{-1} g$.
**C3: One way to interpret Adagrad is that it tries to maintain a diagonal approximation of the Hessian inverse and applies it to the gradients (a.k.a. preconditioning).
So I'm not sure why applying a second approximation of the Hessian inverse on the already preconditioned gradients makes sense intuitively.
Additionally, the authors themselves point out that enhanced Adagrad cannot be applied to general optimization problems (line 182) due to "learning-rate explosion".
Then why introduce this method at all?
A3: We have little knowledge of Adagrad but we do realize the similar performance shared by the raw gradient and the quadratic gradient. We want to apply quadratic gradients to other gradient methods instead of just NAG,  and Adagrad is the starting point of Adagrad-like methods. The enhanced Adagrad method did outperform the raw one in some cases but we didn't realize the fundamental limitations of the enhanced Adagrad method until we try to apply it to the general optimization problems.
**C4: The datasets on which experiments are performed are not standard benchmarking datasets in the ML community and appear to have very few features (looking at Table 2). There are no test set statistics provided.
A4: We have tested the proposed methods on larger datasets (MNIST) with a testing set. The result is also positive.
But we would like to leave it to further work, just like what the baseline work did.
**C5: It is standard to consider the negative log-likelihood objective and apply gradient descent to minimize it.
A5: We apologize for deliberately not using the conventional method of minimizing the loss function. There are three reasons for this: (a) the direct goal of logistic regression is to maximize the log-likelihood objective. Turning this problem into minimizing the negative log-likelihood function might introduce complexity; (b) it is exactly maximizing the log-likelihood objective that helps to develop the idea of
quadratic gradient. We wish to emphasize this; and (c) we would like to mention the versatile aspect of quadratic gradient, whose basic form is just the (simplified) fixed Hessian method.
While applying (fixed Hessian) Newton's method, the updated formula is always to subtract something ($W = W - H^{-1}g$) even for the maximization problems. The proposed gradient variant helps to use Newton's method in a convenient way. Most attention has been paid to gradient descent method but gradient ascent method can also be used with the gradient.
Thanks for the rebuttal. I have read it and would like to keep my score. My question about limited novelty has not been addressed though.
Thank you for your feedback.
We admit that the primary idea behind this work may initially appear as a minor and incremental extension of the concept presented in reference [4].
However, this extension, despite its seemingly small change, was developed after a thorough examination of the Simplified Fixed Hessian (SFH) method [4]. The original SFH method, along with Fixed Hessian Newton's method, introduces a compelling idea but still carries certain limitations.
For instance, it is not applicable to datasets like MNIST and cannot be employed for numerical optimization problems.
Whether or not our work is lacking in novelty, we guess that its practical implications are noteworthy. The proposed gradient variant has the potential to combine the strengths of first-order gradient methods and second-order Newton's methods, thereby enabling the creation of a variety of faster algorithms.
1. The paper is written clearly.
2. The experiments are all using real-world datasets which have strong practical implications.
1. The `quadratic gradient` method is not very new. As the authors have mentioned in Section 3.1 (Line 95), most parts of the method were proposed by Bonte and Vercauteren. I understand that the missing non-negative restriction is important for using the convergence results by Böhning and Lindsay (Line 92).
However, using the absolute value is rather an straightforward solution.
* A more interesting and critical question remaining to be answered is why this proposed `quadratic gradient` method is faster as the authors claimed in the conclusion (Line 238).
* Another problem with this method is its usage being restricted in logistic regression: The authors provided a choice of $\bar{H}$ for logistic regression, while it may be very hard to generalize it to other problems, especially neural network training.
2. The experiments have not shown much advantage of using the `quadratic gradient` method. In Table 1 and Table 2, the accuracy and AUC of the proposed method are almost always lower than the compared baseline method [12]. I understand that the learning time is reduced, but it was not a main problem in [12] as shown in the tables, and we are not sure if there is a tradeoff between the learning time and the accuracy.
* The description of the experiment details in Section 5 is very short. The authors suggest the readers refer to [12]. I think it would be better to have the details in supplementary and a discussion of the weaknesses of [12] in these experiment settings, along with why the proposed method solves those weaknesses.
Minor weaknesses:
1. The maximum likelihood estimation (MLE) is commonly referring to the estimate for $\beta$. The value of the loss function is the negative log-likelihood.
That said, the y-axis of the figures could be corrected.
Also, the objective function is usually the mean of the loss for each data point, not the sum.
2. Typo: Line 216, pulbic -> public.
I have read the rebuttal which answered my questions but did not fully address my concerns on the weakness.
1. The cited paper [7] also used `quadratic gradient`. Is it a following up paper to this paper? If yes, it is better to have a dedicated discussion of the relationship between this paper and [7], e.g.
about the difference and the novelty.
N.A.
No ethics review needed.
$\textbf{Response}$
We would like to thank the reviewers for their input. Their comments have been thoroughly considered, and altering the manuscript in accordance with these comments will significantly improve the quality of our paper in the next submission.
**C1 : The cited paper [7] also used quadratic gradient. Is it a following up paper to this paper? If yes, it is better to have a dedicated discussion of the relationship between this paper and [7], e.g.
about the difference and the novelty.
A1 : Yes, the cited paper [7] is a follow-up paper to this work. The underlying theory of our gradient variant presented in this manuscript suggests its application in a variety of optimization algorithms but this paper should not include such exploration. The cited paper [7] also discusses other related issues.
**C2 : A more interesting and critical question remaining to be answered is why this proposed quadratic gradient method is faster as the authors claimed in the conclusion (Line 238).
A2 : This manuscript shows the proposed gradient variant can be used for the momentum-based gradient method (NAG) and the Adagrad method. The cited paper shows it can also be applied to the Adam method, which is a hybrid of the first two. It might be able to enhance other gradient methods.
**C3 : Another problem with this method is its usage being restricted in logistic regression: The authors provided a choice of for logistic regression, while it may be very hard to generalize it to other problems, especially neural network training.
A3 : The choice of constant $H$ for logistic regression has a similar performance to that derived directly from the varying Hessain itself. It should be very hard to find such a constant choice for other problems.
However, we can build the proposed gradient variant $\bar B g$ directly from the Hessian. Basically, the raw quadratic gradient method without using any first-order gradient methods is the (Simplified) Fixed Hessian method, except that the fixed Hessian in this case is not fixed.
**C4: The experiments have not shown much advantage of using the quadratic gradient method. In Table 1 and Table 2, the accuracy and AUC of the proposed method are almost always lower than the compared baseline method [12]. I understand that the learning time is reduced, but it was not a main problem in [12] as shown in the tables, and we are not sure if there is a tradeoff between the learning time and the accuracy.
A4:   The compared baseline method [12] has better accuracy and AUC than the proposed method. This is because the baseline method performs 7 iterations while the proposed method only performs 3 iterations.
That is where the reduction of learning time for the proposed method comes from. Maybe we should adopt the bootstrapping operation in order to perform the same iterations as the baseline and that would surely increase the learning time.
A1: Got it.
A2: If I understand your statement correctly, this 'faster' claim is from the intuition that NAG, Adagrad, or Adam should be faster than the vanilla SGD. I think some theoretical analysis would help support this claim.
A3: Yes.
Therefore, I wanted to say that this paper as an extension of the simplified fixed Hessian is restricted to the logistic regression setting.
A4: Yes. Usually, the baseline is for the comparison between the best possible performance of the previous method, and the best of your method. You are free to choose the hyperparameter to demonstrate various aspects of advantage in your method.
Based on my comments above, I will keep my score at 3.
A2: This 'faster' claim here (Line 238) refers to the empirical experimental results that the proposed enhenced gradient methods outperform the corresponding raw gradient counterparts.
For instance, the enhanced NAG using quadratic gradient outperfroms the raw NAG. An exception is the enhanced Adagrad that sometimes may not surpass the performance of the raw Adagrad method due to the inherant nature of Adagrad.
A4: OK, I see.
Thank you for your feedback .
I have no further questions.
This paper tackles the privacy-preserving (in the sense of encryption) logistic regression. The paper is clear with introduction and motivation. The algorithm is easy-to-follow and well-supported by experiments.
Overall, it is hard to understand the privacy concerns in this work as there are many privacy-preserving techniques available. Without an example or experiment of privacy attack, the motivation of using HE in the first place is not backed up.
In addition, only [12] is compared in the empirical results.
While the new method seems promising, the lack of other baselines makes it hard to understand the limitations and benefits of the new algorithm.
See weakness.
NA.
No ethics review needed.
$\textbf{Response}$
We would like to thank the reviewers for their input. Their comments have been thoroughly considered, and altering the manuscript in accordance with these comments will significantly improve the quality of our paper in the next submission.
**C1 : Overall, it is hard to understand the privacy concerns in this work as there are many privacy-preserving techniques available. Without an example or experiment of privacy attack, the motivation of using HE in the first place is not backed up.
In addition, only [12] is compared in the empirical results.
While the new method seems promising, the lack of other baselines makes it hard to understand the limitations and benefits of the new algorithm.
** A1 : There are other techniques available for privacy-preserving concerns with their own advantages. Using HE might not be the most efficient and HE is very time-consuming. It may be inefficient to use  [12] as the only baseline.
1. The paper proposes a new method for privacy-preserving logistic regression under FHE, that achieves reasonable results with less computation than existing methods.
2. The proposed method is quite versatile as it can be applied to a variety of optimization algorithms.
1. Experimental results are not convincing.
Contrary to the paper's claims, the proposed method often performs way worse than existing ones (especially on iDASH, Edunburgh and pcs). Datasets are also very small, and therefore do not account for how the method scales with the dimension. Given this is an empirical paper, it seems a bit insufficient.
2. The fixed-Hessian method seems to be closer to preconditioning (see e.g. [1]), where gradients are linearly transformed before being used, than to a proper second order method.
3. The proposed method does not seem to be too novel.
In particular, the paper refers to [2] (which itself refers to a paper with the same title as this manuscript), which proposes a very similar method.
[1] Preconditioned Stochastic Gradient Descent, Xi-Lin Li 2015.
[2] Quadratic Gradient: Uniting gradient algorithm and newton method as one, by Chiang, 2022.
1. Are there settings where the difference in runtime between the proposed method and existing ones is more significant? What would cause this difference to get larger?
2.
When normalizing the data, this has an impact on the Hessian matrix (and therefore on $\bar B$).
In particular, one could imagine that after normalization, $X^T X$ is close to the identity. Could you comment on the effect of normalization on $\bar B$?
3. Line 108, it is claimed that with proper learning rate, the fixed Hessian Netwon method converges: can you give evidence/proof to support this claim?
Limitations are not discussed in the paper.
In particular, experiments only consider a specific setting, with a choice of parameters that seems arbitrary (e.g., 3 vs. 7 iterations), and some claims are not supported with evidence.
No ethics review needed.
$\textbf{Response}$
We would like to thank the reviewers for their input. Their comments have been thoroughly considered, and altering the manuscript in accordance with these comments will significantly improve the quality of our paper in the next submission.
**C1 : Are there settings where the difference in runtime between the proposed method and existing ones is more significant? What would cause this difference to get larger?
A1 : We didn't test various settings on the proposed method. We used to think that the baseline work adopted the learning rate $1/(t+1)$ and we just used $1 + 1/(t+1)$ as the learning rate for the proposed method.
Later we found that they adopted the learning rate $10/(t+1)$ and that for the proposed method the setting $1 + 1/(t+1)$  slightly outperforms that of $1 + 10/(t+1)$ .
Indeed, this paper only shows empirical results. We suspect that there existed more optimal parameters for the enhanced NAG method. To further explore the optimal parameters of the proposed method is currently beyond our capability.
So, our focus in this work is to find a faster algorithm than the baseline work.
**C2 : When normalizing the data, this has an impact on the Hessian matrix (and therefore on  $\bar B$).
In particular, one could imagine that after normalization,  $X^TX$ is close to the identity. Could you comment on the effect of normalization on $\bar B$?
A2 : After normalizing the data, we guess the approximate diagonal Hessian matrix has smaller diagonal elements and therefore $\bar B$ has larger diagonal elements, which seems helpful to the training process.
It is suggested in the paper [2] that the eigenvalues of the Hessian matrix, in this case, that is related to the diagonal elements of $\bar B$, can be helpful to find a safe learning rate for the raw gradient descent method.
As a result, we might be able to safely use a larger learning rate after normalizing the data.
However, it might be difficult to tell the overall effect of normalization on the training process. In the iteration formulas, the updated item $H^{-1}  g$ for the normal Newton's method and the proposed method in this work is not only determined by $H$. And we might not know the effect of the normalization on the gradient $g$.
Even if not normalizing the data has a detrimental effect on $\bar B$, it might also have a similar role in Newton's method or the raw gradient descent method.
**C3 : Line 108, it is claimed that with proper learning rate, the fixed Hessian Netwon method converges: can you give evidence/proof to support this claim?
A3 : This is the annoying problem always existed that puzzles us for a long time.
That is why we assumed that the new learning rate for our gradient variant descent to 1 in a bound number of steps so that the later stage is just the raw Simplified Fixed Hessian Newton's method.
It is too difficult for us to prove the convergence of the proposed methods including the enhanced NAG method and the enhanced Adagrad method. And proofing their convergences might help to find the optimal parameters,  if, of course, they converge. The proof of the convergence for the raw first-order gradient method might help.
$\textbf{References}$
[2] Quadratic Gradient: Uniting gradient algorithm and newton method as one, by Chiang, 2022.
Thank you for your answer. I am not sure I understand what you mean about the step size schedule: what are the final results?
The results of [2] seem highly influenced by the current draft, which is a problem. Claims that are not supported by evidence should also be removed from the draft.
[2] Quadratic Gradient: Uniting gradient algorithm and newton method as one, by Chiang, 2022.
Glad to be of help, and I apologize for any misunderstandings.
We intended to compare the performance of the raw NAG method with the enhanced version,  with as little modification as possible at the algorithmic level.
Therefore, after substituting the raw gradient with the quadratic gradient,  we simply add 1 to the step size schedule used by the baseline work for the enhanced NAG method. We conducted a comparison using two sets of configurations: $1 + 1/(t+1)$ for the enhanced NAG method and $1/(t+1)$ for the raw NAG method; $1 + 10/(t+1)$ for the enhanced NAG method and $10/(t+1)$ for the raw NAG method. The step size $1 + 1/(t+1)$ for the enhanced NAG method showed slightly better performance than $1 + 10/(t+1)$ for the same method, and both outperformed their corresponding raw NAG counterparts.
Ultimately, we opted for $1 + 10/(t+1)$ since the baseline utilized $10/(t+1)$. We are uncertain if there are other step size schedules that offer better performance.
Sure,  untested claims will be removed from this manuscript. Thank you for your valuable feedback.

Meta-review:
role: PC
data: comment: value: This paper studies the problem of private logistic regression. The authors develop the quadratic gradient method to accelerate the private training of the logistic regression.

The reviewers have the following concerns about the paper that have not been addressed:
1. The proposed method has a limited novelty
2. The claim of the proposed method is faster is not well supported
3. The lack of other baselines
4. The experimental comparisons are not fair

Decision: Reject