Parameter-free HE-friendly Logistic Regression

1 Introduction . Machine learning on encrypted data ( MLED ) is an effective method to ensure privacy for both machine learning models and data used for the model training . Unlike other privacy-preserving machine learning methods , MLED does not require decryption of intermediate products of the algorithm or data transfer between participants , enabling a complete outsourcing of machine learning . As data have become increasingly valuable , MLED models have been actively studied in an attempt to combine data and computing power that are separated from each other . With the development of efficient homomorphic encryption ( HE ) schemes that enable MLED , recent studies [ 4 , 9 , 38 , 30 , 29 , 31 ] cover most machine learning models , from simple linear regression to deep neural networks ( DNN ) . However , current research on MLEDs remains lacking owing to the limitations of the proposed HE schemes . Operations on ciphertexts are very inefficient in terms of computation time and memory consumption , compared to operations on plaintexts . In addition , after performing a certain number of operations , a costly procedure called bootstrapping is required [ 19 , 5 ] . Therefore , the majority of studies are limited to the inference phase , assuming a situation that provides machine learning inference as a service . The training phase should be studied more deeply top extract the maximum value from the data . One of the MLED models , whose training steps have been commonly studied , is logistic regression ( LR ) . LR is a simple classification model that linearly divides the data space , which is used effectively in various fields , including medicine , marketing , and geology [ 8 , 10 , 28 ] . [ 23 ] trained an encrypted LR model with gradient descent and approximated the sigmoid function with the most similar polynomials within a certain range . Despite performing well on several benchmark datasets , their ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . methodology raises some concerns because the training parameters , such as learning rate , number of iterations , and range of sigmoid function approximation should be predetermined . Indeed , this problem is common for all proposed MLED training models . In this study , we propose an effective privacy-preserving LR method that is free from most hyperparameter selections . First , we train an unencrypted classification model to extract the prediction probability for each label . Then , we solve for a ridge regression that predicts the logit result of the estimated probability . Our theoretical results provide a new generalization error bound , which can be optimized by properly estimating the logit and our proposed mean matching , which aims to reduce the gap between the distributions of encrypted and unencrypted data . To benefit from the trade-off between security and efficiency , we define private variables , whose privacy is more important than the privacy of others , which has been widely recognized in other fields but has rarely been introduced into MLED . The experimental results show that , compared to the prior encrypted LR models , our method achieves better classification results and lower training latency . 2 Preliminary Information . Throughout this paper , vectors are written in bold lowercase and matrices in bold uppercase . A ciphertext is indicated in the form of the corresponding message enclosed in brackets ( e.g . [ m ] ) . Operations between ciphertexts or between ciphertext and plaintext are both represented same as those between plaintexts . Fully Homomorphic Encryption A fully HE ( FHE ) scheme consists of four procedures : KGen , Enc , Dec and Eval . KGen generates secret key sk and public key pk [ 18 ] . A message m is encrypted to a ciphertext [ m ] = Enc ( m , pk ) , and a ciphertext [ m′ ] is decrypted back to a plaintext m′ = Dec ( [ m′ ] , sk ) . Eval evaluates a function f with a vector of ciphertexts ( [ m1 ] , . . . , [ mk ] ) as an input ; thus , it holds that Dec ( Eval ( ( [ m1 ] , . . . , [ mk ] ) , f , pk ) , sk ) = f ( m1 , . . . , mk ) . An FHE scheme naturally enables addition and multiplication ; therefore , non-polynomial operations must be approximated by polynomial operations . Noise is introduced when a message is first encrypted , which grows as the operation between ciphertexts proceeds . If the noise exceeds a certain level , the ciphertext can not be decrypted correctly . FHE allows an unlimited number of operations through bootstrapping which resets the noise , but bootstrapping incurs a large computational cost . Therefore , leveled HE ( LHE ) which avoids bootstrapping is preferred in many studies , with the number of operations required and corresponding scheme parameters determined in advance . In the case of using gradient descent in MLED training , LHE is not suitable because the number of iterations is very limited ; whereas our method is suitable for LHE because the number of required operations is not too large and can be determined before implementation . Logistic Regression with HE The loss function of LR is the negative log-likelihood , which is given by J ( θ ) = n∑ i=1 log ( 1 + exp ( −yiθTxi ) ) ( 1 ) where θ is the weight vector , xi is i-th input and yi ∈ { +1 , −1 } is the label of i-th input . [ 22 ] minimized ( 1 ) with Nesterov ’ s accelerated gradient , which is a slight modification of the gradient descent algorithm . The gradient descent algorithm can be written as follows : θt+1 = θt − ηgt , gt = ∂J ∂θ = − n∑ i=1 sig ( −yiθTt xi ) · yixi ( 2 ) where η is the learning rate and sig ( x ) = 1/ ( 1 + exp ( −x ) ) is the sigmoid function . When implementing this model with HE , there are two major limitations . The first is the selection of the learning rate and number of iterations for the gradient descent . In Figure 1 ( a ) , we plotted how the loss changes according to the number of iterations for different learning rates . Observe that , the performance can vary drastically depending on the learning rate . In addition , the optimal learning rate is different depending on the number of iterations ( compare the orange line ( η = 0.1 ) and pink line ( η = 0.01 ) ) . When dealing with plaintext , the optimal parameters can be found through cross validation , but this is impossible in MLED because the the weight vector can not be decrypted during training . On the other hand , our method can obtain a closed-form solution without using gradient descent . Another limitation is that the sigmoid function can not be evaluated efficiently using HE . Because the low-order Taylor expansion results in a good approximation only in a localized domain , [ 22 ] approximated the sigmoid function with a polynomial showing the lowest mean squared error in the domain [ −8 , 8 ] . Figure 1 ( b ) shows the different approximation results according to the domain range when approximated by a 3rd order polynomial . The maximum error between the polynomial and the sigmoid function is 0.11 when the range is [ −8 , 8 ] ( blue line ) , which increases to 0.26 when the range is [ −20 , 20 ] ( red line ) . Therefore , it is essential to obtain a tight bound for the input value of the sigmoid function , which is not possible for encrypted data . Conversely , our method solves ridge regression on the ciphertext ; thus , there is no need to evaluate the sigmoid function . Partial Encryption of Private Variables It has been widely accepted that privacy for some variables is more important than security for other variables . In the field of privacy-preserving data mining , k-anonymity [ 40 ] transforms the published variables to prevent exposure of personal identity . Subsequent concepts , such as l-diversity [ 32 , 41 ] and t-closeness [ 27 ] were proposed to protect other private variables , including personal income and medical records . In fair machine learning studies [ 25 , 42 ] , which have gained popularity recently , variables that may cause discrimination are defined as sensitive variables , which are required to be not disclosed [ 20 , 21 ] . In cryptography research , [ 39 , 13 ] presented security criteria and variables were selectively encrypted accordingly . When partially encrypting some private variables , the main purpose is to achieve a trade-off between security and computation & storage efficiency . In Chapter 3 , we propose an efficient ridge regression method with a partially encrypted dataset that can not be directly applied to LR . We separate operations on encrypted and non-encrypted variables , whereas in LR , the input of the sigmoid function is the weighted sum of all variables . In addition , simply applying the existing LR algorithm to partially encrypted data does not result in sufficient efficiency nor does it solve the problems of parameter selection . We solve this problem by replacing the LR with ridge regression for the pre-calculated logit . 3 Framework . In this study , we propose a new framework that can effectively handle classification tasks , while preserving the privacy of private variables . We set two different datasets for the same variables , D1 = { ( xi1 , yi1 ) |i = 1 , . . . , n1 } and D2 = { ( x j 2 , y j 2 ) |j = 1 , . . . , n2 } , where x j 2 = ( xj21 , . . . , x j 2p , [ x j 2 ( p+1 ) ] , . . . , [ x j 2 ( p+ ` ) ] ) contains ` encrypted private variables . Threat Model The participants of our framework consist of data owners O , a modeler M and a crypto-service provider C. We assume that the participants are honest-but-curious and do not collude , which is widely accepted in MLED studies [ 37 , 34 , 1 , 35 ] . Our security goals are as follows : • Neither M nor C should obtain information on private variables . • Neither O nor C should obtain information on the learned model . Protocol The details of our protocol to achieve the security goal are as follows : • ( Teacher modeling ) M trains a teacher model fs with an unencrypted dataset D1 , where D1 is owned by M or publicly available . • ( Encryption ) C generates keys ( pk , sk ) and sends pk to O and M. O encrypts the private variables of their dataset D2 and sends D2 to M. • ( Training on encrypted data ) M infers encrypted logit Enc ( l2 ) = fs ( D2 ) and evaluates Enc ( l̃2 ) = Enc ( l2 ) +Enc ( β ) through mean matching . M trains the privacy-preserving ridge regression on D2 and Enc ( l̃2 ) and obtains Enc ( ω ) . • ( Decryption ) M generates a random polynomial r and sends Enc ( ω + r ) to C. C decrypts w+ r , adds a random discrete Gaussian noise e , and sends w+ r+ e back to M. M subtracts r , and the final weight is obtained as w + e. Similar to [ 12 , 16 ] , the security of our protocol follows directly from the semantic security ( against passive adversaries ) of the underlying HE scheme . In our protocol , e is added to defend against attack proposed by [ 26 ] , which , according to [ 6 ] , with a high probability causes only < 1 bits of precision loss . Our framework consists of three steps , excluding encryption and decryption . The first step is teacher modeling , which is the first of our protocol , and the second and third steps are mean matching and ridge regression , which corresponds with the third protocol . Teacher modeling In the first step , M trains a classification model with D1 , which mimics the first phase of the knowledge distillation in the first model ; then , we extract the soft probability from the target label . However , our method has several major differences from knowledge distillation . First , knowledge distillation aims to train a rather simple model that performs comparably to a complex teacher model , in which extracting the probability is a means to achieve the goal . On the other hand , in our framework the probability plays a more important role . We transform the classification task into a regression problem through the probability . Another difference is that in our method , the teacher model does not need to be a complex model . There are several advantages to using a simple model as a teacher model , which we will demonstrate later . Meanwhile , we should assume that M should possess an unencrypted D1 that can be used in step 1 . Considering that M is depicted as a company seeking to profit from their model in many studies [ 30 , 29 , 31 ] , there may be individuals who provide their private information to M in return . To be more realistic , we assumed there are relatively few people who publish their private information . Moreover , the underlying distributions of D1 and D2 are heterogeneous because they have different features than those who do not want to provide private information . To fit the former setting , we use a simple model that generalizes sufficiently with a small amount of data as the teacher model . Mean matching In the second step , using the teacher model M infers the logit of D2 . In the case where the teacher model uses LR , the inference is simply an inner product between the model parameter θ and xjs , which can be efficiently computed with HE using slot-wise rotation ( See the Allsum algorithm in [ 23 ] ) . Denoting the teacher model as fs and logits of D1 and D2 as l1 and l2 , respectively , we can obtain l2 from D2 using the teacher model fs . However , because D1 and D2 have different distributions , also l1 and l2 might . Therefore , rather than directly using the logit distribution l2 , we suggest adding a regularization term β to l2 so that it can consider the difference between two distributions . The β should satisfy : ∑ i fs ( x i 1 ) ∑ i y i 1 = ∑ i ( fs ( x i 2 ) + β ) ∑ i y i 2 ( 3 ) Now applying the regularization term β , we can obtain the shifted logit l̃2 , where l̃i2 = fs ( x i 2 ) + β . Note that ( 3 ) can be seen as a simplified version of kernel mean matching reported in [ 17 ] , which is widely used in domain adaptation studies [ 24 ] . Kernel mean matching attempts to match the means of the distributions of xs in a kernel space . When directly applying mean matching to logits , the distributions of logits are forced to become similar , which has a negative effect on moving the prediction away from the true label . Therefore , we multiplied the logit by a weight , which reflects the distribution of the true label . Ridge regression Finally , we train a ridge regression on D2 , where the target variable is the shifted logit l̃2 , not y2 . Intuitively , using a well-estimated probability enables a training at least as accurate as that achieved when using a binary target ; our theoretical results reported in Chapter 4 support this claim . By encrypting a small private portion of the entire information , we can train an efficient ridge regression without having to explore the learning parameters . LetX = ( ( x12 ) T · · · ( xn22 ) T ) T ∈ Rn2× ( p+ ` ) , [ Hs ] = ( [ h1 ] · · · [ h ` ] ) be the encrypted ` columns ofX andX ( −s ) the other p columns . Then the ridge estimate is f̂ = X ( XTX+λIp+ ` ) −1XT [ l̃2 ] = XX T ( XXT+λIn2 ) −1 [ l̃2 ] = [ l̃2 ] −λ ( XXT+λIn2 ) −1 [ l̃2 ] . ( 4 ) BecauseXXT = X ( −s ) XT ( −s ) + [ Hs ] [ H T s ] , by applying the Sherman-Woodbury inversion formula we have ( XXT + λIn2 ) −1 = A−1 −A−1 [ Hs ] ( I ` + [ HTs ] A−1 [ Hs ] ) −1 [ HTs ] A−1 where A = X ( −s ) XT ( −s ) + λIn2 . Using the singular value decomposition ( SVD ) of X ( −s ) = UΣV T where U ∈ Rn2×n2 and V ∈ Rp×p are orthogonal matrices , and Σ ∈ Rn2×p is a diagonal matrix with diagonal entries σ1 ≥ · · · ≥ σp ≥ σp+1 = · · · = σn2 = 0 , we have A−1 = ( UΣΣTUT + λIn2 ) −1 = U ( ΣΣT + λIn2 ) −1UT . Then the ridge estimate can be rewritten as f̂ = [ l̃2 ] − λA−1 [ l̃2 ] + λ n2∑ j=1 uju T j [ Hs ] σ2j + λ ( I ` + [ Ξ ] ) −1 n2∑ j=1 1 σ2j + λ [ HTs ] uju T j [ l̃2 ] ( 5 ) where uj are the columns of U and [ Ξ ] = ∑n2 j=1 1 σ2j+λ [ HTs ] uju T j [ Hs ] . The result implies that , by separating the operations that involve encrypted data from those that do not , we can reduce the dimension of the matrix inversion from ( p+ ` ) × ( p+ ` ) to ` × ` . Using the Newton-Schulz iterative algorithm [ 3 ] for matrix inversion , multiplications between ( p + ` ) × ( p + ` ) matrices reduce to multiplications between ` × ` matrices , which results in a quadratic reduction in the computation cost . Moreover , it is known that Newton-Schulz type methods can fail with large matrices [ 15 ] , thus reducing the size of the matrix greatly improves the stability . It is notable that when ` = 1 or 2 , the matrix inverse reduces to a scalar inverse , and can be more easily calculated with Goldschmidt ’ s division algorithm [ 14 ] . Based on having the freedom to choose up+1 , . . . , un2 because σp+1 = · · · = σn2 = 0 , [ Ξ ] and f̂ can be further simplified as Algorithm 1 Training Ridge Regression with Encrypted Private Variable Input : unencypted dataX ( −s ) , encrypted private variable [ hs ] , encrypted target variable [ l ] Output : two ciphertexts [ ω ( −s ) ] , [ ωs ] such that the weight vector of Ridge regression is ωT = ( ωT ( −s ) ωs ) Pre-computations [ - ] Calculate matrixA−1 = ( X ( −s ) XT ( −s ) + λIn2 ) −1 using the collected user data X ( −s ) [ - ] Obtain orthogonal matrices U = ( u1 · · ·up ) , V = ( v1 . . .vp ) and diagonal entries σ = ( σ1 , · · · , σp ) using the SVD ofX ( −s ) Computations on Encrypted data [ - ] [ c1 ] ← MatVecProd ( V , σσ2+λ , U , [ l ] ) ; [ c2 ] ← MatVecProd ( V , σ σ2+λ , U , [ hs ] ) ; [ - ] [ c3 ] ← MatVecProd ( U , σ 2 σ2+λ , U , [ l ] ) ; [ c4 ] ← MatVecProd ( U , σ 2 σ2+λ , U , [ hs ] ) ; [ - ] [ c5 ] ← ( [ hs ] − [ c4 ] ) /λ ; [ c6 ] ← InnerProduct ( [ c5 ] , [ l ] ) ; [ c7 ] ← Inv ( InnerProduct ( [ c5 ] , [ hs ] ) + 1 ) [ - ] Output [ ω ( −s ) ] ← [ c1 ] − [ c2 ] · [ c6 ] · [ c7 ] and [ ωs ] ← [ c6 ] · [ c7 ] [ Ξ ] = p+∑̀ j=1 1 σ2j + λ [ HTs ] uju T j [ Hs ] , f̂ = [ l̃2 ] − λA−1 [ l̃2 ] + λ p+∑̀ j=1 uju T j [ Hs ] σ2j + λ ( Is + [ Ξ ] ) −1 p+∑̀ j=1 1 σ2j + λ [ HTs ] uju T j [ l̃2 ] . ( 6 ) by choosing ûp+k = hk − p+k−1∑ i=1 ui ( u T i hk ) , up+k = ûp+k/‖ûp+k‖ , k = 1 , ... , ` . Algorithm Algorithm 1 describes the procedure of training ridge regression , assuming that there is one private variable . For convenience , our method was described as obtaining the ridge estimate , but to infer the test data , the algorithm is slightly modified to output the weight vector . We assume that all n2 samples of private variables are encoded into a single ciphertext to enable SIMD operations . If the number of samples is larger than the maximum number of samples which can be packed into a ciphertext , we can naturally split the samples into multiple ciphertexts . Excluding addition and multiplication operations for the ciphertext , the algorithm consists of three functions : InnerProduct , Inv , and MatVecProduct . InnerProduct returns a ciphertext in which the values of all plaintext slots are the same as the inner product of the two vectors . Inv evaluates Goldschmidt ’ s division algorithm stated in Algorithm 1 ; however , in the case where s ≥ 3 , it evaluates an iterative method for matrix inversion . MatVecProduct evaluates efficient matrix-vector multiplication using SVD of the input matrix . The detailed algorithms of these functions are provided in the Appendix . Extension to nonlinear models A limitation of our framework is that it can only model the linear relationship between the input variables and target variable . However , when partially encrypting private variables , our method has the capacity to model the nonlinear relationship between variables using kernel methods . The formulation opf the extension , and some experimental results are provided in the Appendix .

Summary: This paper introduces an interesting new approach for learning a classifier on homomorphically encrypted data. A few important assumptions are made: only very few features (one to three maybe) are privacy-protected and the modeler is able to build a regression model on a part of the dataset. Next, further training data including private features are used to convert the initial model into a Ridge regression model. The key innovation is that the encrypted computation step can be isolated to make it very simple when few features are encrypted.