Parameter-free HE-friendly Logistic Regression
Year: 2021

Abstract:
Privacy in machine learning has been widely recognized as an essential ethical and legal issue, because the data used for machine learning may contain sensitive information. Homomorphic encryption has recently attracted attention as a key solution to preserve privacy in machine learning applications. However, current approaches on the training of encrypted machine learning have relied heavily on hyperparameter selection, which should be avoided owing to the extreme difficulty of conducting validation on encrypted data. In this study, we propose an effective privacy-preserving logistic regression method that is free from the approximation of the sigmoid function and hyperparameter selection. In our framework, a logistic regression model can be transformed into the corresponding ridge regression for the logit function. We provide a theoretical background for our framework by suggesting a new generalization error bound on the encrypted data. Experiments on various real-world data show that our framework achieves better classification results while reducing latency by ∼ 68%, compared to the previous models.

heading: 1 Introduction
text: Machine learning on encrypted data (MLED) is an effective method to ensure privacy for both machine learning models and data used for the model training. Unlike other privacy-preserving machine learning methods, MLED does not require decryption of intermediate products of the algorithm or data transfer between participants, enabling a complete outsourcing of machine learning. As data have become increasingly valuable, MLED models have been actively studied in an attempt to combine data and computing power that are separated from each other. With the development of efficient homomorphic encryption (HE) schemes that enable MLED, recent studies [4, 9, 38, 30, 29, 31] cover most machine learning models, from simple linear regression to deep neural networks (DNN).
However, current research on MLEDs remains lacking owing to the limitations of the proposed HE schemes. Operations on ciphertexts are very inefficient in terms of computation time and memory consumption, compared to operations on plaintexts. In addition, after performing a certain number of operations, a costly procedure called bootstrapping is required [19, 5]. Therefore, the majority of studies are limited to the inference phase, assuming a situation that provides machine learning inference as a service. The training phase should be studied more deeply top extract the maximum value from the data.
One of the MLED models, whose training steps have been commonly studied, is logistic regression (LR). LR is a simple classification model that linearly divides the data space, which is used effectively in various fields, including medicine, marketing, and geology [8, 10, 28]. [23] trained an encrypted LR model with gradient descent and approximated the sigmoid function with the most similar polynomials within a certain range. Despite performing well on several benchmark datasets, their ∗Equal contribution. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
methodology raises some concerns because the training parameters, such as learning rate, number of iterations, and range of sigmoid function approximation should be predetermined. Indeed, this problem is common for all proposed MLED training models.
In this study, we propose an effective privacy-preserving LR method that is free from most hyperparameter selections. First, we train an unencrypted classification model to extract the prediction probability for each label. Then, we solve for a ridge regression that predicts the logit result of the estimated probability. Our theoretical results provide a new generalization error bound, which can be optimized by properly estimating the logit and our proposed mean matching, which aims to reduce the gap between the distributions of encrypted and unencrypted data. To benefit from the trade-off between security and efficiency, we define private variables, whose privacy is more important than the privacy of others, which has been widely recognized in other fields but has rarely been introduced into MLED. The experimental results show that, compared to the prior encrypted LR models, our method achieves better classification results and lower training latency.

heading: 2 Preliminary Information
text: Throughout this paper, vectors are written in bold lowercase and matrices in bold uppercase. A ciphertext is indicated in the form of the corresponding message enclosed in brackets (e.g. [m]). Operations between ciphertexts or between ciphertext and plaintext are both represented same as those between plaintexts.
Fully Homomorphic Encryption A fully HE (FHE) scheme consists of four procedures : KGen,Enc,Dec and Eval. KGen generates secret key sk and public key pk [18]. A message m is encrypted to a ciphertext [m] = Enc(m, pk), and a ciphertext [m′] is decrypted back to a plaintext m′ = Dec([m′], sk). Eval evaluates a function f with a vector of ciphertexts ([m1], . . . , [mk]) as an input; thus, it holds that Dec(Eval(([m1], . . . , [mk]), f, pk), sk) = f(m1, . . . ,mk). An FHE scheme naturally enables addition and multiplication; therefore, non-polynomial operations must be approximated by polynomial operations. Noise is introduced when a message is first encrypted, which grows as the operation between ciphertexts proceeds. If the noise exceeds a certain level, the ciphertext cannot be decrypted correctly. FHE allows an unlimited number of operations through bootstrapping which resets the noise, but bootstrapping incurs a large computational cost. Therefore, leveled HE (LHE) which avoids bootstrapping is preferred in many studies, with the number of operations required and corresponding scheme parameters determined in advance. In the case of using gradient descent in MLED training, LHE is not suitable because the number of iterations is very limited; whereas our method is suitable for LHE because the number of required operations is not too large and can be determined before implementation.
Logistic Regression with HE The loss function of LR is the negative log-likelihood, which is given by
J(θ) = n∑ i=1 log(1 + exp(−yiθTxi)) (1)
where θ is the weight vector, xi is i-th input and yi ∈ {+1,−1} is the label of i-th input. [22] minimized (1) with Nesterov’s accelerated gradient, which is a slight modification of the gradient descent algorithm. The gradient descent algorithm can be written as follows:
θt+1 = θt − ηgt, gt = ∂J
∂θ = − n∑ i=1 sig(−yiθTt xi) · yixi (2)
where η is the learning rate and sig(x) = 1/(1 + exp(−x)) is the sigmoid function. When implementing this model with HE, there are two major limitations. The first is the selection of the learning rate and number of iterations for the gradient descent. In Figure 1(a), we plotted how the loss changes according to the number of iterations for different learning rates. Observe that, the performance can vary drastically depending on the learning rate. In addition, the optimal learning rate is different depending on the number of iterations (compare the orange line (η = 0.1) and pink line (η = 0.01)). When dealing with plaintext, the optimal parameters can be found through cross validation, but this is impossible in MLED because the the weight vector cannot be decrypted during training. On the other hand, our method can obtain a closed-form solution without using gradient descent.
Another limitation is that the sigmoid function cannot be evaluated efficiently using HE. Because the low-order Taylor expansion results in a good approximation only in a localized domain, [22] approximated the sigmoid function with a polynomial showing the lowest mean squared error in the domain [−8, 8]. Figure 1(b) shows the different approximation results according to the domain range when approximated by a 3rd order polynomial. The maximum error between the polynomial and the sigmoid function is 0.11 when the range is [−8, 8] (blue line), which increases to 0.26 when the range is [−20, 20] (red line). Therefore, it is essential to obtain a tight bound for the input value of the sigmoid function, which is not possible for encrypted data. Conversely, our method solves ridge regression on the ciphertext; thus, there is no need to evaluate the sigmoid function.
Partial Encryption of Private Variables It has been widely accepted that privacy for some variables is more important than security for other variables. In the field of privacy-preserving data mining, k-anonymity [40] transforms the published variables to prevent exposure of personal identity. Subsequent concepts, such as l-diversity [32, 41] and t-closeness [27] were proposed to protect other private variables, including personal income and medical records. In fair machine learning studies [25, 42], which have gained popularity recently, variables that may cause discrimination are defined as sensitive variables, which are required to be not disclosed [20, 21]. In cryptography research, [39, 13] presented security criteria and variables were selectively encrypted accordingly.
When partially encrypting some private variables, the main purpose is to achieve a trade-off between security and computation & storage efficiency. In Chapter 3, we propose an efficient ridge regression method with a partially encrypted dataset that cannot be directly applied to LR. We separate operations on encrypted and non-encrypted variables, whereas in LR, the input of the sigmoid function is the weighted sum of all variables. In addition, simply applying the existing LR algorithm to partially encrypted data does not result in sufficient efficiency nor does it solve the problems of parameter selection. We solve this problem by replacing the LR with ridge regression for the pre-calculated logit.

heading: 3 Framework
text: In this study, we propose a new framework that can effectively handle classification tasks, while preserving the privacy of private variables. We set two different datasets for the same variables, D1 = {(xi1, yi1)|i = 1, . . . , n1} and D2 = {(x j 2, y j 2)|j = 1, . . . , n2}, where x j 2 = (xj21, . . . , x j 2p, [x j 2(p+1)], . . . , [x j 2(p+`)]) contains ` encrypted private variables.
Threat Model The participants of our framework consist of data owners O, a modeler M and a crypto-service provider C. We assume that the participants are honest-but-curious and do not collude, which is widely accepted in MLED studies [37, 34, 1, 35]. Our security goals are as follows:
• Neither M nor C should obtain information on private variables.
• Neither O nor C should obtain information on the learned model.
Protocol The details of our protocol to achieve the security goal are as follows:
• (Teacher modeling) M trains a teacher model fs with an unencrypted dataset D1, where D1 is owned by M or publicly available.
• (Encryption) C generates keys (pk, sk) and sends pk to O and M. O encrypts the private variables of their dataset D2 and sends D2 to M.
• (Training on encrypted data) M infers encrypted logit Enc(l2) = fs(D2) and evaluates Enc(l̃2) = Enc(l2) +Enc(β) through mean matching. M trains the privacy-preserving ridge regression on D2 and Enc(l̃2) and obtains Enc(ω).
• (Decryption) M generates a random polynomial r and sends Enc(ω + r) to C. C decrypts w+ r, adds a random discrete Gaussian noise e, and sends w+ r+ e back to M. M subtracts r, and the final weight is obtained as w + e.
Similar to [12, 16], the security of our protocol follows directly from the semantic security (against passive adversaries) of the underlying HE scheme. In our protocol, e is added to defend against attack proposed by [26], which, according to [6], with a high probability causes only <1 bits of precision loss.
Our framework consists of three steps, excluding encryption and decryption. The first step is teacher modeling, which is the first of our protocol, and the second and third steps are mean matching and ridge regression, which corresponds with the third protocol.
Teacher modeling In the first step, M trains a classification model with D1, which mimics the first phase of the knowledge distillation in the first model; then, we extract the soft probability from the target label. However, our method has several major differences from knowledge distillation. First, knowledge distillation aims to train a rather simple model that performs comparably to a complex teacher model, in which extracting the probability is a means to achieve the goal. On the other hand, in our framework the probability plays a more important role. We transform the classification task into a regression problem through the probability. Another difference is that in our method, the teacher model does not need to be a complex model. There are several advantages to using a simple model as a teacher model, which we will demonstrate later.
Meanwhile, we should assume that M should possess an unencrypted D1 that can be used in step 1. Considering that M is depicted as a company seeking to profit from their model in many studies [30, 29, 31], there may be individuals who provide their private information to M in return. To be more realistic, we assumed there are relatively few people who publish their private information. Moreover, the underlying distributions of D1 and D2 are heterogeneous because they have different features than those who do not want to provide private information. To fit the former setting, we use a simple model that generalizes sufficiently with a small amount of data as the teacher model.
Mean matching In the second step, using the teacher model M infers the logit of D2. In the case where the teacher model uses LR, the inference is simply an inner product between the model parameter θ and xjs, which can be efficiently computed with HE using slot-wise rotation (See the Allsum algorithm in [23]). Denoting the teacher model as fs and logits of D1 and D2 as l1 and l2, respectively, we can obtain l2 from D2 using the teacher model fs. However, because D1 and D2 have different distributions, also l1 and l2 might. Therefore, rather than directly using the logit distribution l2, we suggest adding a regularization term β to l2 so that it can consider the difference between two distributions. The β should satisfy:∑
i fs(x i 1)∑
i y i 1
=
∑ i(fs(x
i 2) + β)∑ i y i 2
(3)
Now applying the regularization term β, we can obtain the shifted logit l̃2, where l̃i2 = fs(x i 2) + β.
Note that (3) can be seen as a simplified version of kernel mean matching reported in [17], which is widely used in domain adaptation studies [24]. Kernel mean matching attempts to match the means of the distributions of xs in a kernel space. When directly applying mean matching to logits, the distributions of logits are forced to become similar, which has a negative effect on moving the
prediction away from the true label. Therefore, we multiplied the logit by a weight, which reflects the distribution of the true label.
Ridge regression Finally, we train a ridge regression on D2, where the target variable is the shifted logit l̃2, not y2. Intuitively, using a well-estimated probability enables a training at least as accurate as that achieved when using a binary target; our theoretical results reported in Chapter 4 support this claim. By encrypting a small private portion of the entire information, we can train an efficient ridge regression without having to explore the learning parameters. LetX = ((x12)
T · · · (xn22 )T )T ∈ Rn2×(p+`), [Hs] = ([h1] · · · [h`]) be the encrypted ` columns ofX andX(−s) the other p columns. Then the ridge estimate is
f̂ = X(XTX+λIp+`) −1XT [l̃2] = XX T (XXT+λIn2) −1[l̃2] = [l̃2]−λ(XXT+λIn2)−1[l̃2].
(4)
BecauseXXT = X(−s)XT(−s) +[Hs][H T s ], by applying the Sherman-Woodbury inversion formula we have
(XXT + λIn2) −1 = A−1 −A−1[Hs](I` + [HTs ]A−1[Hs])−1[HTs ]A−1
where A = X(−s)XT(−s) + λIn2 . Using the singular value decomposition (SVD) of X(−s) = UΣV T where U ∈ Rn2×n2 and V ∈ Rp×p are orthogonal matrices, and Σ ∈ Rn2×p is a diagonal matrix with diagonal entries σ1 ≥ · · · ≥ σp ≥ σp+1 = · · · = σn2 = 0, we have
A−1 = (UΣΣTUT + λIn2) −1 = U(ΣΣT + λIn2) −1UT .
Then the ridge estimate can be rewritten as
f̂ = [l̃2]− λA−1[l̃2] + λ  n2∑ j=1 uju T j [Hs] σ2j + λ  (I` + [Ξ])−1  n2∑ j=1 1 σ2j + λ [HTs ]uju T j [l̃2]  (5) where uj are the columns of U and [Ξ] = ∑n2 j=1
1 σ2j+λ [HTs ]uju T j [Hs]. The result implies that,
by separating the operations that involve encrypted data from those that do not, we can reduce the dimension of the matrix inversion from (p+ `)× (p+ `) to `× `. Using the Newton-Schulz iterative algorithm [3] for matrix inversion, multiplications between (p + `) × (p + `) matrices reduce to multiplications between ` × ` matrices, which results in a quadratic reduction in the computation cost. Moreover, it is known that Newton-Schulz type methods can fail with large matrices [15], thus reducing the size of the matrix greatly improves the stability. It is notable that when ` = 1 or 2, the matrix inverse reduces to a scalar inverse, and can be more easily calculated with Goldschmidt’s division algorithm [14].
Based on having the freedom to choose up+1, . . . ,un2 because σp+1 = · · · = σn2 = 0, [Ξ] and f̂ can be further simplified as
Algorithm 1 Training Ridge Regression with Encrypted Private Variable Input: unencypted dataX(−s), encrypted private variable [hs], encrypted target variable [l] Output: two ciphertexts [ω(−s)], [ωs] such that the weight vector of Ridge regression is ωT = (ωT(−s) ωs) Pre-computations [-] Calculate matrixA−1 = (X(−s)XT(−s) + λIn2)
−1 using the collected user data X(−s) [-] Obtain orthogonal matrices U = (u1 · · ·up),V = (v1 . . .vp) and diagonal entries σ = (σ1, · · · , σp) using the SVD ofX(−s) Computations on Encrypted data [-] [c1]← MatVecProd(V , σσ2+λ ,U , [l]); [c2]← MatVecProd(V , σ σ2+λ ,U , [hs]); [-] [c3]← MatVecProd(U , σ 2
σ2+λ ,U , [l]); [c4]← MatVecProd(U , σ
2
σ2+λ ,U , [hs]);
[-] [c5]← ([hs]− [c4])/λ; [c6]← InnerProduct([c5], [l]); [c7]← Inv(InnerProduct([c5], [hs]) + 1) [-] Output [ω(−s)]← [c1]− [c2] · [c6] · [c7] and [ωs]← [c6] · [c7]
[Ξ] = p+∑̀ j=1 1 σ2j + λ [HTs ]uju T j [Hs],
f̂ = [l̃2]− λA−1[l̃2] + λ p+∑̀ j=1 uju T j [Hs] σ2j + λ  (Is + [Ξ])−1 p+∑̀ j=1 1 σ2j + λ [HTs ]uju T j [l̃2]  . (6)
by choosing
ûp+k = hk − p+k−1∑ i=1 ui(u T i hk), up+k = ûp+k/‖ûp+k‖, k = 1, ..., `.
Algorithm Algorithm 1 describes the procedure of training ridge regression, assuming that there is one private variable. For convenience, our method was described as obtaining the ridge estimate, but to infer the test data, the algorithm is slightly modified to output the weight vector. We assume that all n2 samples of private variables are encoded into a single ciphertext to enable SIMD operations. If the number of samples is larger than the maximum number of samples which can be packed into a ciphertext, we can naturally split the samples into multiple ciphertexts. Excluding addition and multiplication operations for the ciphertext, the algorithm consists of three functions: InnerProduct, Inv, and MatVecProduct. InnerProduct returns a ciphertext in which the values of all plaintext slots are the same as the inner product of the two vectors. Inv evaluates Goldschmidt’s division algorithm stated in Algorithm 1; however, in the case where s ≥ 3, it evaluates an iterative method for matrix inversion. MatVecProduct evaluates efficient matrix-vector multiplication using SVD of the input matrix. The detailed algorithms of these functions are provided in the Appendix.
Extension to nonlinear models A limitation of our framework is that it can only model the linear relationship between the input variables and target variable. However, when partially encrypting private variables, our method has the capacity to model the nonlinear relationship between variables using kernel methods. The formulation opf the extension, and some experimental results are provided in the Appendix.

heading: 4 Theoretical Framework
text: This section provides the core theory behind our proposed methodology. We provide a generalization error bound for the proposed method.
Consider a classification task with an input space X , an output space Y = {0, 1}, and hypothesis spaceZ = X × [0, 1]. Let S = {zi = (xi, yi) ∈ Z : i = 1, ..., Ns} represent the given open labelled data of Ns samples. In our classification settings, we have two distinct distributions: (x, y) ∼ DS
according to a sample distribution DS over Z with y ∈ Y and (x, p) ∼ DT according to a target distribution DT over Z with p ∈ [0, 1]. Utilizing these concepts and following the notations in [33], a hypothesis h : Z → < is a scoring function such that we assign to each point x the class label of the maximum score h(x, y), that is, arg maxy∈Y h(x, y). The margin ξh(z) at a sample example z = (x, y) ∼ DS is then defined by
ξh(z) = h(x, y)−max y′ 6=y
h(x, y′)
Thus, h misclassifies z iff ξh(z) ≤ 0. The generalization error (or risk) of a hypothesis h ∈ H := {h : Z → <} in a sample distribution DS and a true target distribution DT are defined, respectively, by RDS (h) = Ez∼DS [1ξh(z)≤0], RDT (h) = Ez∼DT [1ξh(z)≤0] We next define theH-divergence (short version ofH∆H-divergence) as in [2] which measures the discrepancy between two different distributions as in by
DH(DS ,DT ) = sup h,h′∈H |RDS (h ′)−RDT (h)|
We are now ready to derive the following generalization error risk in our classification setting. Lemma 1. Let DS and DT be the sample and the true target distributions, respectively. Then for any hypothesis h ∈ H, the following inequality holds:
RDT (h) ≤ min h′∈H RDS (h ′) +DH(DS ,DT ) (7)
As a result, the upper bound of empirical expected target error can be decomposed into two parts: The first term is the empirical source error. In our framework, we try to minimize this term by training a teacher classification model with D1 in the first step. Then the second term is the H-divergence between the DS and DT which implies that we should reduce the gap between two distributions to achieve better performance. The mean matching step in our framework corresponds to this part, which is a simplified version of kernel mean matching [17].
We next present the following margin bound for classification in the probably approximately correct (PAC) learning framework. Theorem 2. Let DS and DT be the sample and the true target distributions, respectively. Then, for any δ > 0, with probability at least 1− δ, the following classification generalization bound holds for all hypothesis h ∈ Hρ = {(x, y)→ β · (yx) : ‖β‖2 ≤ 1/ρ, ‖x‖2 ≤ r}:
RDT (h) ≤ 1
N N∑ i=1 loge0 ( 1 + e−2yiβ·xi ) +DHρ(DS ,DT )
+ 16r
ρ √ N +
√ log log2 4r ρ
N + √ log 2δ 2N
(8)
where e0 = log(1 + 1/e).

heading: 5 Experiments
text: In this section, we evaluate our method using various real-world datasets. Through experiments, we argue that our method achieves better classification results compared to the existing methods with a shorter computation time. In addition, we verify that even when the distributions of published and encrypted data are very different, our method can properly correct the difference and maintain the classification performance.
Datasets We used five widely used classification datasets from the UCI data repository: The adult income dataset (Adult), bank marketing dataset (Bank), Wisconsin Breast Cancer dataset (Cancer), Pima Indians Diabetes dataset (Diabetes), and Australian Credit Approval (Credit) dataset [11]. Adult and Bank datasets are commonly used in fair machine learning studies, thus, we treat variables that can induce social bias, such as gender and age, like private variables. For the other datasets, the variable that had the greatest impact on the classification performance was selected as a private variable. In practical applications, data owners can flexibly set private variables according to their security standards. The explanation for each dataset is provided in the Appendix.
Experimental Setup All the experiments were performed on a machine equipped with 40 threads of an Intel Xeon E-2660 v3 @2.60GHz CPU processor. We implemented step 1 of our framework with Python 3.6.3, using the LR module in the scikit-learn library. Other steps were implemented with C++, using HEAAN v1.1 [7] for HE. HEAAN is an implementation of the CKKS scheme; a detailed description of the scheme and its parameters are provided in the Appendix. We used privacy-preserving LR depicted in [23] (LRHE) as a baseline comparison method because it is the only MLED training model that works within a practical computation time. We do not compare our method to the HE-friendly SVM model reported in [36], because their model assumes that the data owners pre-compute the kernel matrix, which can leak information about the model. Comparing our model to DNN models reported in [29] is not of our interest because their model has a latency that is too high. Note that we trained LRHE with all variables encrypted because LRHE hardly benefits from partial encryption, as mentioned in Chapter 2. Indeed, encrypting only private variables for LRHE resulted in time savings of less than 1.2%.
For each dataset, we randomly sampled 20% as test samples, and 20% of the other 80% were treated as plaintext data, which were used for the training of step 1. We encrypted the private variables of the remaining 60% and used them for steps 2 and 3. For CKKS parameters, we usedN = 216, qL = 21200, and P = 240. The sigmoid function approximation degree for LRHE was set to 3 because increasing the degree results in a larger multiplicative depth and less possible number of gradient descents with LHE. In addition, we observed that increasing the degree up to 7 did not significantly affect the performance of the model. The learning rate for LRHE was chosen in {0.001, 0.0001, 0.00001} to achieve the best classification performance.
Results We compared the methods in terms of their classification accuracy and computation time. The results of the experiments are summarized in Table 1. For all datasets, Ours achieved higher accuracy and lower latency compared to LRHE. In particular, regarding the computation time, Ours can reduce latency by 66-68% through efficient computation using private variables. Although the performance of LR using plaintext is not inferior to ours, the performance of LRHE degrades owing to the sigmoid approximation and limited iterations of gradient descent. On the other hand, because Ours is free from parameter selection, its performance does not decrease compared to the same operation using plaintext.
As an ablation study, we additionally trained step 3 of Ours using gradient descent, with all variables encrypted (denoted as Ours-grad in Table 1). Because the multiplicative depth per iteration of Ours-grad is lower than that of LRHE owing to not using the sigmoid approximation, we trained Ours-grad for eight iterations. The learning rate was set in {0.001, 0.0001, 0.00001}.The procedure that requires the most computation time in Ours-grad is the matrix-matrix multiplication, which causes Ours-grad to have a greater latency compared to LRHE. However, the computation time per iteration of LRHE is higher than that of Ours-grad, and the result of the matrix-matrix multiplication can be reused through the iterations after calculating once. Therefore, it is implied that as the number of iterations increases, Ours-grad will be more efficient than LRHE. In addition, the classification performance of Ours-grad is similar to or better than that of Ours, which means that Ours-grad converges better by allowing more iterations than LRHE. Therefore, even in situations where the partial encryption of private variables is limited, our method has an advantage over the existing methods.
Effectiveness of mean matching In this section, we verify that the mean matching in step 2 of our method is a simple but effective way to mitigate performance degradation due to the difference between distributions of unencrypted and encrypted data. Here, we do not randomly divide the dataset,
instead divide it referring to the value of a certain variable. Diabetes dataset, for example, was divided according to whether the value of ’Glucose’ variable was greater than (or less than) its median. Figure 3(a) plots the dimension reduction result for Diabetes dataset using t-stochastic neighborhood embedding and confirms that the two subsets (X1 and X2) have very separate distributions compared to each other. We sampled 20% of the entire dataset from X1 and trained step 1 with the samples. 80% of X2 was used for steps 2 and 3, with the remaining 20% of X2 used as the test samples.
Table 2 summarizes the experimental results for three datasets: Adult, Diabetes and Credit. For the other two, there were no variables that could properly divide them. The test accuracy of the LR model trained with X1 was lower than that of the LRHE trained with X2 because the domains of X1 and X2 were different, and the test set was sampled from X2. The performance of Ours without mean matching was not different from that of the LR with X1. However, with mean matching the classification accuracy of Ours was raised to the same level as that of LRHE. Even when the accuracy of LR with X1 was 50% or lower than that of LRHE, applying mean matching completely recovered the classification performance of Ours.
To verify why the mean matching of our method works well, in Figure 3(b), we plotted the histogram of logits inferred by LR models. In the figure, the red and blue histograms represent the distributions of logit of X1 for LR learned with X1 (=l1) and logit of X2 for LR learned with X2, respectively. Thus, the blue histogram can be seen as the distribution of the logit of X2 when it is classified "correctly". However, because the model is biased toward the distribution of X1 when trained with X1, the distribution of the logit of X2 (=l2) is also biased toward the distribution of the logit of X1, as shown in the green histogram. Therefore, mean matching plays the role of shifting the biased distribution (orange histogram) to fit the correct distribution.

heading: 6 Conclusion
text: We proposed an efficient HE-friendly classification method, that protects both the users’ private information and secrecy of the model. We trained a ridge regression model for the logit instead of logistic regression, which has a closed-form solution and is free from parameter search. To extract
logit values from a binary label, we trained a teacher model on unencrypted data that can output logit, similar to that of knowledge distillation. Owing to encrypting only private variables that require a high level of security instead of encrypting all information, our method can achieve higher efficiency and training stability. Our algorithm is HE scheme-free; it can bring efficiency when implemented with any widely used HE schemes.
Although our study aims to extend the existing machine learning method in the privacy-preserving direction, our model can be corrupted by malicious participants because our security model is limited to semi-honest, non-colluding participants. Designing a secure system for weaker security assumptions should be conducted in future studies.
Acknowledgments and Disclosure of Funding
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and Technology Information and Communication (2019R1A2C2002358)

Review summaries:
This paper introduces an interesting new approach for learning a classifier on homomorphically encrypted data. A few important assumptions are made: only very few features (one to three maybe) are privacy-protected and the modeler is able to build a regression model on a part of the dataset. Next, further training data including private features are used to convert the initial model into a Ridge regression model. The key innovation is that the encrypted computation step can be isolated to make it very simple when few features are encrypted.  

This study considers a situation that a part of unencrypted samples are available while the other samples are provided with encrypted by FHE (or LHE). First, the modeler trains a logistic regression model with unencrypted samples. By using this model, given encrypted samples,  the modeler obtains the encrypted logits. To absorb the difference of sample generating distributions of non-private samples and private samples, moment matching is applied on encrypted logits (I could not understand why this works).  Then, ridge regression is solved with the encrypted logits with labels, in that the solution can be obtained in an analytic form instead of iterative methods as logistic regression regularly requires. **Due to the lack of a detailed description of the security model and global information flow of the protocol, this summary might be wrong.