Parameter-free HE-friendly Logistic Regression

1 Introduction . Machine learning on encrypted data ( MLED ) is an effective method to ensure privacy for both machine learning models and data used for the model training . Unlike other privacy-preserving machine learning methods , MLED does not require decryption of intermediate products of the algorithm or data transfer between participants , enabling a complete outsourcing of machine learning . As data have become increasingly valuable , MLED models have been actively studied in an attempt to combine data and computing power that are separated from each other . With the development of efficient homomorphic encryption ( HE ) schemes that enable MLED , recent studies [ 4 , 9 , 38 , 30 , 29 , 31 ] cover most machine learning models , from simple linear regression to deep neural networks ( DNN ) . However , current research on MLEDs remains lacking owing to the limitations of the proposed HE schemes . Operations on ciphertexts are very inefficient in terms of computation time and memory consumption , compared to operations on plaintexts . In addition , after performing a certain number of operations , a costly procedure called bootstrapping is required [ 19 , 5 ] . Therefore , the majority of studies are limited to the inference phase , assuming a situation that provides machine learning inference as a service . The training phase should be studied more deeply top extract the maximum value from the data . One of the MLED models , whose training steps have been commonly studied , is logistic regression ( LR ) . LR is a simple classification model that linearly divides the data space , which is used effectively in various fields , including medicine , marketing , and geology [ 8 , 10 , 28 ] . [ 23 ] trained an encrypted LR model with gradient descent and approximated the sigmoid function with the most similar polynomials within a certain range . Despite performing well on several benchmark datasets , their ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . methodology raises some concerns because the training parameters , such as learning rate , number of iterations , and range of sigmoid function approximation should be predetermined . Indeed , this problem is common for all proposed MLED training models . In this study , we propose an effective privacy-preserving LR method that is free from most hyperparameter selections . First , we train an unencrypted classification model to extract the prediction probability for each label . Then , we solve for a ridge regression that predicts the logit result of the estimated probability . Our theoretical results provide a new generalization error bound , which can be optimized by properly estimating the logit and our proposed mean matching , which aims to reduce the gap between the distributions of encrypted and unencrypted data . To benefit from the trade-off between security and efficiency , we define private variables , whose privacy is more important than the privacy of others , which has been widely recognized in other fields but has rarely been introduced into MLED . The experimental results show that , compared to the prior encrypted LR models , our method achieves better classification results and lower training latency . 2 Preliminary Information . Throughout this paper , vectors are written in bold lowercase and matrices in bold uppercase . A ciphertext is indicated in the form of the corresponding message enclosed in brackets ( e.g . [ m ] ) . Operations between ciphertexts or between ciphertext and plaintext are both represented same as those between plaintexts . Fully Homomorphic Encryption A fully HE ( FHE ) scheme consists of four procedures : KGen , Enc , Dec and Eval . KGen generates secret key sk and public key pk [ 18 ] . A message m is encrypted to a ciphertext [ m ] = Enc ( m , pk ) , and a ciphertext [ m′ ] is decrypted back to a plaintext m′ = Dec ( [ m′ ] , sk ) . Eval evaluates a function f with a vector of ciphertexts ( [ m1 ] , . . . , [ mk ] ) as an input ; thus , it holds that Dec ( Eval ( ( [ m1 ] , . . . , [ mk ] ) , f , pk ) , sk ) = f ( m1 , . . . , mk ) . An FHE scheme naturally enables addition and multiplication ; therefore , non-polynomial operations must be approximated by polynomial operations . Noise is introduced when a message is first encrypted , which grows as the operation between ciphertexts proceeds . If the noise exceeds a certain level , the ciphertext can not be decrypted correctly . FHE allows an unlimited number of operations through bootstrapping which resets the noise , but bootstrapping incurs a large computational cost . Therefore , leveled HE ( LHE ) which avoids bootstrapping is preferred in many studies , with the number of operations required and corresponding scheme parameters determined in advance . In the case of using gradient descent in MLED training , LHE is not suitable because the number of iterations is very limited ; whereas our method is suitable for LHE because the number of required operations is not too large and can be determined before implementation . Logistic Regression with HE The loss function of LR is the negative log-likelihood , which is given by J ( θ ) = n∑ i=1 log ( 1 + exp ( −yiθTxi ) ) ( 1 ) where θ is the weight vector , xi is i-th input and yi ∈ { +1 , −1 } is the label of i-th input . [ 22 ] minimized ( 1 ) with Nesterov ’ s accelerated gradient , which is a slight modification of the gradient descent algorithm . The gradient descent algorithm can be written as follows : θt+1 = θt − ηgt , gt = ∂J ∂θ = − n∑ i=1 sig ( −yiθTt xi ) · yixi ( 2 ) where η is the learning rate and sig ( x ) = 1/ ( 1 + exp ( −x ) ) is the sigmoid function . When implementing this model with HE , there are two major limitations . The first is the selection of the learning rate and number of iterations for the gradient descent . In Figure 1 ( a ) , we plotted how the loss changes according to the number of iterations for different learning rates . Observe that , the performance can vary drastically depending on the learning rate . In addition , the optimal learning rate is different depending on the number of iterations ( compare the orange line ( η = 0.1 ) and pink line ( η = 0.01 ) ) . When dealing with plaintext , the optimal parameters can be found through cross validation , but this is impossible in MLED because the the weight vector can not be decrypted during training . On the other hand , our method can obtain a closed-form solution without using gradient descent . Another limitation is that the sigmoid function can not be evaluated efficiently using HE . Because the low-order Taylor expansion results in a good approximation only in a localized domain , [ 22 ] approximated the sigmoid function with a polynomial showing the lowest mean squared error in the domain [ −8 , 8 ] . Figure 1 ( b ) shows the different approximation results according to the domain range when approximated by a 3rd order polynomial . The maximum error between the polynomial and the sigmoid function is 0.11 when the range is [ −8 , 8 ] ( blue line ) , which increases to 0.26 when the range is [ −20 , 20 ] ( red line ) . Therefore , it is essential to obtain a tight bound for the input value of the sigmoid function , which is not possible for encrypted data . Conversely , our method solves ridge regression on the ciphertext ; thus , there is no need to evaluate the sigmoid function . Partial Encryption of Private Variables It has been widely accepted that privacy for some variables is more important than security for other variables . In the field of privacy-preserving data mining , k-anonymity [ 40 ] transforms the published variables to prevent exposure of personal identity . Subsequent concepts , such as l-diversity [ 32 , 41 ] and t-closeness [ 27 ] were proposed to protect other private variables , including personal income and medical records . In fair machine learning studies [ 25 , 42 ] , which have gained popularity recently , variables that may cause discrimination are defined as sensitive variables , which are required to be not disclosed [ 20 , 21 ] . In cryptography research , [ 39 , 13 ] presented security criteria and variables were selectively encrypted accordingly . When partially encrypting some private variables , the main purpose is to achieve a trade-off between security and computation & storage efficiency . In Chapter 3 , we propose an efficient ridge regression method with a partially encrypted dataset that can not be directly applied to LR . We separate operations on encrypted and non-encrypted variables , whereas in LR , the input of the sigmoid function is the weighted sum of all variables . In addition , simply applying the existing LR algorithm to partially encrypted data does not result in sufficient efficiency nor does it solve the problems of parameter selection . We solve this problem by replacing the LR with ridge regression for the pre-calculated logit . 3 Framework . In this study , we propose a new framework that can effectively handle classification tasks , while preserving the privacy of private variables . We set two different datasets for the same variables , D1 = { ( xi1 , yi1 ) |i = 1 , . . . , n1 } and D2 = { ( x j 2 , y j 2 ) |j = 1 , . . . , n2 } , where x j 2 = ( xj21 , . . . , x j 2p , [ x j 2 ( p+1 ) ] , . . . , [ x j 2 ( p+ ` ) ] ) contains ` encrypted private variables . Threat Model The participants of our framework consist of data owners O , a modeler M and a crypto-service provider C. We assume that the participants are honest-but-curious and do not collude , which is widely accepted in MLED studies [ 37 , 34 , 1 , 35 ] . Our security goals are as follows : • Neither M nor C should obtain information on private variables . • Neither O nor C should obtain information on the learned model . Protocol The details of our protocol to achieve the security goal are as follows : • ( Teacher modeling ) M trains a teacher model fs with an unencrypted dataset D1 , where D1 is owned by M or publicly available . • ( Encryption ) C generates keys ( pk , sk ) and sends pk to O and M. O encrypts the private variables of their dataset D2 and sends D2 to M. • ( Training on encrypted data ) M infers encrypted logit Enc ( l2 ) = fs ( D2 ) and evaluates Enc ( l̃2 ) = Enc ( l2 ) +Enc ( β ) through mean matching . M trains the privacy-preserving ridge regression on D2 and Enc ( l̃2 ) and obtains Enc ( ω ) . • ( Decryption ) M generates a random polynomial r and sends Enc ( ω + r ) to C. C decrypts w+ r , adds a random discrete Gaussian noise e , and sends w+ r+ e back to M. M subtracts r , and the final weight is obtained as w + e. Similar to [ 12 , 16 ] , the security of our protocol follows directly from the semantic security ( against passive adversaries ) of the underlying HE scheme . In our protocol , e is added to defend against attack proposed by [ 26 ] , which , according to [ 6 ] , with a high probability causes only < 1 bits of precision loss . Our framework consists of three steps , excluding encryption and decryption . The first step is teacher modeling , which is the first of our protocol , and the second and third steps are mean matching and ridge regression , which corresponds with the third protocol . Teacher modeling In the first step , M trains a classification model with D1 , which mimics the first phase of the knowledge distillation in the first model ; then , we extract the soft probability from the target label . However , our method has several major differences from knowledge distillation . First , knowledge distillation aims to train a rather simple model that performs comparably to a complex teacher model , in which extracting the probability is a means to achieve the goal . On the other hand , in our framework the probability plays a more important role . We transform the classification task into a regression problem through the probability . Another difference is that in our method , the teacher model does not need to be a complex model . There are several advantages to using a simple model as a teacher model , which we will demonstrate later . Meanwhile , we should assume that M should possess an unencrypted D1 that can be used in step 1 . Considering that M is depicted as a company seeking to profit from their model in many studies [ 30 , 29 , 31 ] , there may be individuals who provide their private information to M in return . To be more realistic , we assumed there are relatively few people who publish their private information . Moreover , the underlying distributions of D1 and D2 are heterogeneous because they have different features than those who do not want to provide private information . To fit the former setting , we use a simple model that generalizes sufficiently with a small amount of data as the teacher model . Mean matching In the second step , using the teacher model M infers the logit of D2 . In the case where the teacher model uses LR , the inference is simply an inner product between the model parameter θ and xjs , which can be efficiently computed with HE using slot-wise rotation ( See the Allsum algorithm in [ 23 ] ) . Denoting the teacher model as fs and logits of D1 and D2 as l1 and l2 , respectively , we can obtain l2 from D2 using the teacher model fs . However , because D1 and D2 have different distributions , also l1 and l2 might . Therefore , rather than directly using the logit distribution l2 , we suggest adding a regularization term β to l2 so that it can consider the difference between two distributions . The β should satisfy : ∑ i fs ( x i 1 ) ∑ i y i 1 = ∑ i ( fs ( x i 2 ) + β ) ∑ i y i 2 ( 3 ) Now applying the regularization term β , we can obtain the shifted logit l̃2 , where l̃i2 = fs ( x i 2 ) + β . Note that ( 3 ) can be seen as a simplified version of kernel mean matching reported in [ 17 ] , which is widely used in domain adaptation studies [ 24 ] . Kernel mean matching attempts to match the means of the distributions of xs in a kernel space . When directly applying mean matching to logits , the distributions of logits are forced to become similar , which has a negative effect on moving the prediction away from the true label . Therefore , we multiplied the logit by a weight , which reflects the distribution of the true label . Ridge regression Finally , we train a ridge regression on D2 , where the target variable is the shifted logit l̃2 , not y2 . Intuitively , using a well-estimated probability enables a training at least as accurate as that achieved when using a binary target ; our theoretical results reported in Chapter 4 support this claim . By encrypting a small private portion of the entire information , we can train an efficient ridge regression without having to explore the learning parameters . LetX = ( ( x12 ) T · · · ( xn22 ) T ) T ∈ Rn2× ( p+ ` ) , [ Hs ] = ( [ h1 ] · · · [ h ` ] ) be the encrypted ` columns ofX andX ( −s ) the other p columns . Then the ridge estimate is f̂ = X ( XTX+λIp+ ` ) −1XT [ l̃2 ] = XX T ( XXT+λIn2 ) −1 [ l̃2 ] = [ l̃2 ] −λ ( XXT+λIn2 ) −1 [ l̃2 ] . ( 4 ) BecauseXXT = X ( −s ) XT ( −s ) + [ Hs ] [ H T s ] , by applying the Sherman-Woodbury inversion formula we have ( XXT + λIn2 ) −1 = A−1 −A−1 [ Hs ] ( I ` + [ HTs ] A−1 [ Hs ] ) −1 [ HTs ] A−1 where A = X ( −s ) XT ( −s ) + λIn2 . Using the singular value decomposition ( SVD ) of X ( −s ) = UΣV T where U ∈ Rn2×n2 and V ∈ Rp×p are orthogonal matrices , and Σ ∈ Rn2×p is a diagonal matrix with diagonal entries σ1 ≥ · · · ≥ σp ≥ σp+1 = · · · = σn2 = 0 , we have A−1 = ( UΣΣTUT + λIn2 ) −1 = U ( ΣΣT + λIn2 ) −1UT . Then the ridge estimate can be rewritten as f̂ = [ l̃2 ] − λA−1 [ l̃2 ] + λ n2∑ j=1 uju T j [ Hs ] σ2j + λ ( I ` + [ Ξ ] ) −1 n2∑ j=1 1 σ2j + λ [ HTs ] uju T j [ l̃2 ] ( 5 ) where uj are the columns of U and [ Ξ ] = ∑n2 j=1 1 σ2j+λ [ HTs ] uju T j [ Hs ] . The result implies that , by separating the operations that involve encrypted data from those that do not , we can reduce the dimension of the matrix inversion from ( p+ ` ) × ( p+ ` ) to ` × ` . Using the Newton-Schulz iterative algorithm [ 3 ] for matrix inversion , multiplications between ( p + ` ) × ( p + ` ) matrices reduce to multiplications between ` × ` matrices , which results in a quadratic reduction in the computation cost . Moreover , it is known that Newton-Schulz type methods can fail with large matrices [ 15 ] , thus reducing the size of the matrix greatly improves the stability . It is notable that when ` = 1 or 2 , the matrix inverse reduces to a scalar inverse , and can be more easily calculated with Goldschmidt ’ s division algorithm [ 14 ] . Based on having the freedom to choose up+1 , . . . , un2 because σp+1 = · · · = σn2 = 0 , [ Ξ ] and f̂ can be further simplified as Algorithm 1 Training Ridge Regression with Encrypted Private Variable Input : unencypted dataX ( −s ) , encrypted private variable [ hs ] , encrypted target variable [ l ] Output : two ciphertexts [ ω ( −s ) ] , [ ωs ] such that the weight vector of Ridge regression is ωT = ( ωT ( −s ) ωs ) Pre-computations [ - ] Calculate matrixA−1 = ( X ( −s ) XT ( −s ) + λIn2 ) −1 using the collected user data X ( −s ) [ - ] Obtain orthogonal matrices U = ( u1 · · ·up ) , V = ( v1 . . .vp ) and diagonal entries σ = ( σ1 , · · · , σp ) using the SVD ofX ( −s ) Computations on Encrypted data [ - ] [ c1 ] ← MatVecProd ( V , σσ2+λ , U , [ l ] ) ; [ c2 ] ← MatVecProd ( V , σ σ2+λ , U , [ hs ] ) ; [ - ] [ c3 ] ← MatVecProd ( U , σ 2 σ2+λ , U , [ l ] ) ; [ c4 ] ← MatVecProd ( U , σ 2 σ2+λ , U , [ hs ] ) ; [ - ] [ c5 ] ← ( [ hs ] − [ c4 ] ) /λ ; [ c6 ] ← InnerProduct ( [ c5 ] , [ l ] ) ; [ c7 ] ← Inv ( InnerProduct ( [ c5 ] , [ hs ] ) + 1 ) [ - ] Output [ ω ( −s ) ] ← [ c1 ] − [ c2 ] · [ c6 ] · [ c7 ] and [ ωs ] ← [ c6 ] · [ c7 ] [ Ξ ] = p+∑̀ j=1 1 σ2j + λ [ HTs ] uju T j [ Hs ] , f̂ = [ l̃2 ] − λA−1 [ l̃2 ] + λ p+∑̀ j=1 uju T j [ Hs ] σ2j + λ ( Is + [ Ξ ] ) −1 p+∑̀ j=1 1 σ2j + λ [ HTs ] uju T j [ l̃2 ] . ( 6 ) by choosing ûp+k = hk − p+k−1∑ i=1 ui ( u T i hk ) , up+k = ûp+k/‖ûp+k‖ , k = 1 , ... , ` . Algorithm Algorithm 1 describes the procedure of training ridge regression , assuming that there is one private variable . For convenience , our method was described as obtaining the ridge estimate , but to infer the test data , the algorithm is slightly modified to output the weight vector . We assume that all n2 samples of private variables are encoded into a single ciphertext to enable SIMD operations . If the number of samples is larger than the maximum number of samples which can be packed into a ciphertext , we can naturally split the samples into multiple ciphertexts . Excluding addition and multiplication operations for the ciphertext , the algorithm consists of three functions : InnerProduct , Inv , and MatVecProduct . InnerProduct returns a ciphertext in which the values of all plaintext slots are the same as the inner product of the two vectors . Inv evaluates Goldschmidt ’ s division algorithm stated in Algorithm 1 ; however , in the case where s ≥ 3 , it evaluates an iterative method for matrix inversion . MatVecProduct evaluates efficient matrix-vector multiplication using SVD of the input matrix . The detailed algorithms of these functions are provided in the Appendix . Extension to nonlinear models A limitation of our framework is that it can only model the linear relationship between the input variables and target variable . However , when partially encrypting private variables , our method has the capacity to model the nonlinear relationship between variables using kernel methods . The formulation opf the extension , and some experimental results are provided in the Appendix .

Summary: This study considers a situation that a part of unencrypted samples are available while the other samples are provided with encrypted by FHE (or LHE). First, the modeler trains a logistic regression model with unencrypted samples. By using this model, given encrypted samples,  the modeler obtains the encrypted logits. To absorb the difference of sample generating distributions of non-private samples and private samples, moment matching is applied on encrypted logits (I could not understand why this works).  Then, ridge regression is solved with the encrypted logits with labels, in that the solution can be obtained in an analytic form instead of iterative methods as logistic regression regularly requires. **Due to the lack of a detailed description of the security model and global information flow of the protocol, this summary might be wrong.