# Privacy-Preserving Vertical Collaborative Logistic Regression without Trusted Third-Party Coordinator

Authors: Xiaopeng Yu; Wei Zhao; Dianhua Tang; Kai Liang
Journal: Security and Communication Networks
 (2022)

---

## Abstract

Collaborative learning is an emerging distributed learning paradigm, which enables multiple parties to jointly train a shared machine learning (ML) model without causing the disclosure of the raw data of each party. As one of the fundamental collaborative learning algorithms, privacy-preserving collaborative logistic regression has recently gained attention from industry and academia, which utilizes cryptographic techniques to securely train joint logistic regression models across data from multiple parties. However, existing schemes have high communication and computational overhead, lose the ability to deal with high-dimensional sparse samples, cut down the accuracy of the model, or exist the risk of leaking private information. To overcome these issues, considering vertically distributed data, we propose a privacy-preserving vertical collaborative logistic regression (P2 VCLR) based on approximate homomorphic encryption (HE), which enables two parties to jointly train a shared model without a trusted third-party coordinator. Our scheme utilizes batching method in approximate HE to encrypt multiple data into a single ciphertext and enable a parallel processing through single instruction multiple data (SIMD) manner. We evaluate our scheme by using three publicly available datasets, the experimental results indicate that our scheme outperforms existing schemes in terms of training time and model performance.


---

## Body

## 1. Introduction

Machine learning (ML) [1] is a method for analyzing large-scale data and is widely used in practice to train predictive models for practical applications. As one of the basic machine learning algorithms, logistic regression (LR) [2] has attracted much attention for its powerful ability to solve classification problems in practical applications, such as disease diagnosis [3], credit evaluation [4].In recent years, in order to obtain massive data for training better-performing models [5], there is growing interest in machine learning by combining the data from different institutions [6]. For instance, different hospitals would like to combine health data to jointly train models to facilitate more accurate disease diagnosis; different financial companies want to collaborate to train more effective credit card scoring and fraud detection models. Unfortunately, due to regulatory and competitive reasons, it is difficult or even impossible to directly exchange data of different parties for model training in practice [7]. That is, the data of different organizations is isolated. To eliminate the issue of “data isolation“, the idea of collaborative learning [8] is introduced. Its goal is to cooperatively train a shared ML model on distributed data while complying with regulation and protecting privacy. The security, privacy, and efficiency concerns remain main challenges for practical applications. Recently, as a fundamental collaborative learning algorithm, privacy-preserving collaborative logistic regression (PPCLR) [9–24] has received considerable attention recently, which utilizes cryptographic primitives such as homomorphic encryption (HE) algorithm [25] and multi-party computation (MPC) protocol [26] to securely train a joint logistic regression model across data from multiple parties. However, for the HE-based schemes [9–11], model weights are exposed to all parties at each iterative update of global model parameters during training, which is able to be utilized to deduce additional private information [27]; for the MPC-based schemes [12, 14], after using secret sharing (SS) [28] on training samples of all parties, even previously sparse samples become dense, so they are not able to efficiently handle sparse samples and require high communication complexity when the training data becomes large.To solve the problems mentioned above, in a two-party setting, considering two vertically distributed training data with the same sample distributions but different feature distributions, we construct a privacy-preserving vertical collaborative logistic regression (P2 VCLR) based on the HE for arithmetic of approximate numbers [29]. The main contributions are as follows:(1)
Firstly, we construct aP2 VCLR framework for collaborative learning of vertical distributed features, which can securely realize the joint modeling of both parties without the assistance of a trusted third-party (TTP), and hence greatly reduces the system complexity.(2)
Secondly, to improve the training efficiency, using the batching technique in HE [29], the proposed scheme can pack multiple samples into a single plaintext with multiple slots, encrypt it into a single ciphertext, and enable a parallel computing through using SIMD.(3)
Finally, we conduct performance evaluations on three datasets [30], and the experimental evaluation results indicate that our scheme achieves a significant improvement in efficiency and performance than existing schemes [9, 21]. Specifically, the training time of the model is decreased by almost 32.3%-72.5%; the accuracy, F1-score, and AUC of the model have nearly 0.3% - 3.0%, 0.1% - 2.7% and 0 - 0.03 improvement, respectively. Furthermore, the security analysis indicates that the proposed P2 VCLR scheme is secure against semi-honest adversaries, and neither of the both parties can know each other's raw data.The rest of this work is arranged as follows. Several works related to our scheme are introduced in Section [2]. In Section [3], we review some preliminaries. In Section [4], our scheme is described. In Section [5], the evaluations for our scheme are presented. The security analysis of our scheme is shown in Section [6]. In Section [7], we conclude this work.


## 2. Related Works

There are several works that have been made to joint train a LR model across multiple data owners. In general, a common approach is to implement secure logistic regression by using cryptographic primitives like HE [25] and MPC [26]. The existing works [9–24] can be divided into two categories: PPCLR with a TTP coordinator [9–16] and PPCLR without a TTP coordinator [17–24]. A summary of the existing works [9–24] follows.As for the PPCLR with TTP coordinator [9–16], Hardy et al. [9] described a privacy-preserving federated LR scheme by employing additively HE scheme [25], which centralizes two vertically distributed training data in one TTP coordinator, but the approximation of non-polynomial function reduces the model accuracy. Yang et al. [10] shown a quasi-Newton way for achieving vertical federated LR model based on the additively HE scheme [25]. Using an additive HE [31] and an aggregation method [32], Mandal et al. [11] built a privacy-preserving regression analysis protocol on the horizontally distributed high-dimensional data. Employing an additive secret sharing technique [33], Zhang et al. [12] proposed a privacy-preservation collaborative learning for ensuring local training data and model information. Liu et al. [13] introduced a collaborative learning platform, which supports multiple institutions to build machine learning models collaboratively over large-scale horizontally and vertically partitioned data. By means of MPC from additive secret sharing [34, 35], Cock et al. [14] proposed a protocol for securely training LR model over distributed parties, where TTP initializer assigns relevant random values to two computing severs. Based on multi-key fully HE [36], Wang et al. [15] designed a secure cloud-edge collaborative LR system, which employs the cloud centre and edge nodes to collaboratively train a LR model over encrypted data. Zhu et al. [16] proposed a value-blind LR training method in a collaborative setting based on HE [25], where the central server updates model parameters without access to the training data and intermediate values, and model parameters are shared among the central server with collaborating parties. However, it's inherently difficult to establish a third party trusted by any data owners in a real-world scenario. Moreover, data interactions between data owners and TTP raise the risk of leakage of sensitive data of the data owner.To decrease the complexity of training a joint model for any two parties, by removing the TTP coordinator, Yanget al. [17] constructed a parallel distributed LR method for vertical federated learning based on HE [25], which allows two parties to jointly train models without the help of a TTP coordinator. Using the secure MPC protocol and ciphertext domain conversion protocol [37], Chen et al. [18] presented a collaborative learning system for jointly building better models over vertically partitioned multiple data. Based on the HE scheme [29], Li et al. [19] introduced a collaborative learning method on encrypted data, which could securely train LR models over vertically distributed data from both data owners. Based on asynchronous gradient sharing and HE algorithm [29], Wei et al. [20] designed a two-parties collaborative LR protocol, which can train securely joint model on the vertically partitioned data. Chen et al. [21] combined the HE [25] and secret sharing [38] to build securely LR model on the vertically distributed large-scale sparse training data. Over the horizontally partitioned data, based on secure MPC protocol, Ghavamipour et al. [22] described two methods to train LR model in a privacy-preserving manner. However, each data owner requires to compute multiple shares of its sensitive training data and sends them separately to each non-collusion computation party, this leads to heavy communication costs. He et al. [23] constructed a vertical federated LR method through a HE algorithm [25], which uses a piecewise function to ensure the accuracy of the loss function, but this results in a loss of efficiency. With the HE scheme [25] and differential privacy algorithm [39], Sun et al. [24] introduced a vertical federated LR solution, which alleviates the constraints on feature dimensions. However, the existing PPCLR schemes [17–24] without a TTP coordinator lead to high communication and computational overhead.


## 3. Preliminaries

### 3.1. System Architecture

For ease of reading, the definitions of the symbols in ourP2 VCLR scheme are displayed in Table 1. As is shown in Figure 1, the system architecture of our P2 VCLR includes two semi-trusted entities: Pa and Pb. Pa and Pb hold the vertically distributed datasets Sa and Sb, respectively. Sa and Sb have the same sample space but different feature distribution, namely, Pa holds the part of the features, Pb holds another part of the features and the label. Pa cooperates with Pb to train a shared LR model without disclosing the privacy of training data. Specifically, Pa generates ska,pka,rka,gka←KeyGenN,Q [29], sends polynomial-degree N, coefficient-modulus Q, scaling factor Δ, public key pka, relinearization key rka, galois key gka to Pb, and securely store secret key ska. Then, Pa encrypts its own data with pka, and sends the encrypted data to Pb. Finally, Pa and Pb jointly execute P2 VCLR algorithm to obtain the training result.Table 1
The definition of the symbol.
NotationDefinitionxMessage vectorx0,…,xN/2−1xiThei-th element of xxA ciphertext ofxXA list of message vectorsx0,…,xn−1XiThei-th message vector of XXA list of ciphertextsx0,x1,…,xn−1XiThei-th ciphertext of Xx∗yx0⋅y0,…,xN/2−1⋅yN/2−1x+yx0+y0,…,xN/2−1+yN/2−1x−yx0−y0,…,xN/2−1−yN/2−1x⋅yx0⋅y0+⋯+xN/2−1⋅yN/2−1Figure 1
System architecture.


### 3.2. Homomorphic Encryption

HE allows direct operations on ciphertext without decryption, and can ensure that the computation on the ciphertext is consistent with the computation on the plaintext. Cheonet al. [29] introduced an approximate HE algorithm from ring learning with errors (RLWE) [40], which supports the following operations.(1)
ski,pki,rki,gki←Key_GenN,Q: Given the parameters N,Q, it generates ski, pki, rki, gki for Pi.(2)
x←Encx,pki: Given a message vector x and pki, it generates a ciphertext x.(3)
x←Decx,ski: Given x and ski, it generates a message vector x.(4)
x+y←Addx,y: Given x and y, it generates a ciphertext x+y=x+y.(5)
x+y←Add_Plainx,y: Given x and a message vector y, it generates a ciphertext x+y=x+y.(6)
x0+⋯+xn−1←Add_Manyx,y: Given a ciphertext list X=x0,…,xn−1, it generates a ciphertext x0+⋯+xn−1=x0+⋯+xn−1.(7)
x−y←Subx,y: Given x and y, it generates a ciphertext x−y=x−y.(8)
x−y←Sub_Plainx,y: Given x and y, it generates a ciphertext x−y=x−y.(9)
x∗y←Mulx,y,rki: Given x, y and rki, it generates a ciphertext x∗y=x∗y.(10)
x∗y←Mul_Plainx,y,rki: Given x, y and rki, it generates a ciphertext x∗y=x∗y.(11)
y←Rot_Vectorx,k,gki: Given x=x0,…,xN/2−1, k and gki, it rotates x left by rotation value k, and generates a ciphertext y=xk,…,xN/2−1,x0,…,xk−1.


### 3.3. Logistic Regression

Let a datasetS includes m samples xi,1,…,xi,n,yi|i∈m=xi,yi|i∈m, where an input xi maps to a binary dependent variable yi∈0,1, the goal of binary LR is to compute weights ξ=ξ0,ξ1,…,ξn that minimizes the log-likelihood loss function Jξ=−1/m⋅∑i=1m1−yi⋅1−logσzi⋅ξ+yi⋅logσzi⋅ξ, where zi=1,xi. Assuming that ξk and αk denote the model weights and learning rate at iteration k, respectively, the gradient descent (GD) is able to be used to compute the extremum of Jξ by ξk+1←ξk−αk/m⋅∑i=1mσzi⋅ξk−yi⋅zi. Since the HE scheme (CKKS) [29] is not able to effectively support non-polynomial arithmetic operations, we use a 7-degree polynomial function fx=w0+w1x+w3x3+w5x5+w7x7 to approximate sigmoid function σx=1/1+e−x over the domain [-8, 8], where w0=1/2, w1=1.73496/8, w3=4.19407/83, w5=5.43402/85, and w7=2.50739/87.


## 3.1. System Architecture

For ease of reading, the definitions of the symbols in ourP2 VCLR scheme are displayed in Table 1. As is shown in Figure 1, the system architecture of our P2 VCLR includes two semi-trusted entities: Pa and Pb. Pa and Pb hold the vertically distributed datasets Sa and Sb, respectively. Sa and Sb have the same sample space but different feature distribution, namely, Pa holds the part of the features, Pb holds another part of the features and the label. Pa cooperates with Pb to train a shared LR model without disclosing the privacy of training data. Specifically, Pa generates ska,pka,rka,gka←KeyGenN,Q [29], sends polynomial-degree N, coefficient-modulus Q, scaling factor Δ, public key pka, relinearization key rka, galois key gka to Pb, and securely store secret key ska. Then, Pa encrypts its own data with pka, and sends the encrypted data to Pb. Finally, Pa and Pb jointly execute P2 VCLR algorithm to obtain the training result.Table 1
The definition of the symbol.
NotationDefinitionxMessage vectorx0,…,xN/2−1xiThei-th element of xxA ciphertext ofxXA list of message vectorsx0,…,xn−1XiThei-th message vector of XXA list of ciphertextsx0,x1,…,xn−1XiThei-th ciphertext of Xx∗yx0⋅y0,…,xN/2−1⋅yN/2−1x+yx0+y0,…,xN/2−1+yN/2−1x−yx0−y0,…,xN/2−1−yN/2−1x⋅yx0⋅y0+⋯+xN/2−1⋅yN/2−1Figure 1
System architecture.


## 3.2. Homomorphic Encryption

HE allows direct operations on ciphertext without decryption, and can ensure that the computation on the ciphertext is consistent with the computation on the plaintext. Cheonet al. [29] introduced an approximate HE algorithm from ring learning with errors (RLWE) [40], which supports the following operations.(1)
ski,pki,rki,gki←Key_GenN,Q: Given the parameters N,Q, it generates ski, pki, rki, gki for Pi.(2)
x←Encx,pki: Given a message vector x and pki, it generates a ciphertext x.(3)
x←Decx,ski: Given x and ski, it generates a message vector x.(4)
x+y←Addx,y: Given x and y, it generates a ciphertext x+y=x+y.(5)
x+y←Add_Plainx,y: Given x and a message vector y, it generates a ciphertext x+y=x+y.(6)
x0+⋯+xn−1←Add_Manyx,y: Given a ciphertext list X=x0,…,xn−1, it generates a ciphertext x0+⋯+xn−1=x0+⋯+xn−1.(7)
x−y←Subx,y: Given x and y, it generates a ciphertext x−y=x−y.(8)
x−y←Sub_Plainx,y: Given x and y, it generates a ciphertext x−y=x−y.(9)
x∗y←Mulx,y,rki: Given x, y and rki, it generates a ciphertext x∗y=x∗y.(10)
x∗y←Mul_Plainx,y,rki: Given x, y and rki, it generates a ciphertext x∗y=x∗y.(11)
y←Rot_Vectorx,k,gki: Given x=x0,…,xN/2−1, k and gki, it rotates x left by rotation value k, and generates a ciphertext y=xk,…,xN/2−1,x0,…,xk−1.


## 3.3. Logistic Regression

Let a datasetS includes m samples xi,1,…,xi,n,yi|i∈m=xi,yi|i∈m, where an input xi maps to a binary dependent variable yi∈0,1, the goal of binary LR is to compute weights ξ=ξ0,ξ1,…,ξn that minimizes the log-likelihood loss function Jξ=−1/m⋅∑i=1m1−yi⋅1−logσzi⋅ξ+yi⋅logσzi⋅ξ, where zi=1,xi. Assuming that ξk and αk denote the model weights and learning rate at iteration k, respectively, the gradient descent (GD) is able to be used to compute the extremum of Jξ by ξk+1←ξk−αk/m⋅∑i=1mσzi⋅ξk−yi⋅zi. Since the HE scheme (CKKS) [29] is not able to effectively support non-polynomial arithmetic operations, we use a 7-degree polynomial function fx=w0+w1x+w3x3+w5x5+w7x7 to approximate sigmoid function σx=1/1+e−x over the domain [-8, 8], where w0=1/2, w1=1.73496/8, w3=4.19407/83, w5=5.43402/85, and w7=2.50739/87.


## 4. Privacy-Preserving Vertical Collaborative Logistic Regression

Over vertically distributed datasetsSa and Sb, we propose a P2 VCLR scheme based on an approximate HE [29]. Using batching method in approximate HE, the proposed scheme packs a message vector with multiple messages into a plaintext with multiple plaintext slots, and performs parallel training based on SIMD. For ease of readability, we give the Algorithm , which can be found in Appendix. We assume that the samples of Sa and Sb held by Pa and Pb have been aligned, namely,(1)Sa=x0,1x1,1⋮xm−1,1x0,2x1,2⋮xm−1,2⋯⋯⋱⋯x0,n1x1,n1⋮xm−1,n1Sb=x0,n1+1x1,n1+1⋮xm−1,n1+1⋯⋯⋱⋯x0,n1+n2x1,n1+n2⋮xm−1,n1+n2y0y1⋮ym−1.Sa and Sb consist of m samples of the form xi,1,xi,2,…,xi,n1 and xi,n1+1,…,xi,n1+n2,yi, respectively, where i=0,1,…,m−1. Each column of Sa denote the features. The last column of Sb represents the label, and other columns of Sb represent the features. Pa cooperates with Pb to train a shared LR model without revealing the data privacy. Suppose 2n1+n2+1≤N, the details of the proposed P2 VCLR are described below.Input: Sa and Sb for Pa and Pb respectivelyOutput: ξ0s,ξ1s,…,ξn1s and ξn1+1s,ξn1+2s,…,ξn1+n2s for Pa and Pb respectivelyPreprocessing:1:Pa computes l=2logn1+n2+1, u=N/2l, v=m/u, letsxi′=1,xi,1,xi,2,…,xi,n1,0,0,…,0|i=0,1,…,m−1,generatesska,pka,rka,gka←KeyGenN,Q, encrypts dataset Sa into v ciphertextsxa,i=Encxi⋅u′,xi⋅u+1′,…,xi+1⋅u−1′−N/2,pka|i=0,1,…,v−2,xa,v−1=Encxv−1⋅u′,xv−1⋅u+1′,…,xm′,0,0,…,0−N/2,pka,letsξ′0=ξ00,ξ10,…,ξn10,0,0,…,0−l,encrypts the initial weightξ′0 into one ciphertextξa0=Encξ′0,ξ′0,…,ξ′0−N/2,pka,and sendsN, Q, Δ, pka, rka, gka, s, xa,i|i=0,1,…,v−1, ξa0 to Pb.2:Pb computes l=2logn1+n2+1, u=N/2l, v=m/u, letsxi′′=0,0,…,0−n1+1,xi,n1+1,xi,n1+2,…,xi,n1+n2,0,0,…,0−li=0,1,…,m−1,yi=yi,0,0,…,0−li=0,1,…,m−1,sets data setSb into 2v message vectorsxb,i=xi⋅u″,xi⋅u+1″,⋯,xi+1⋅u−1″−N/2i=0,1,…,v−2,xb,v−1=xv−1⋅u′′,xv−1⋅u+1′′,…,xm′′,0,0,…,0−N/2,yi=yi⋅u,yi⋅u+1,…,yi+1⋅u−1−N/2i=0,1,…,v−2,yv−1=yv−1⋅u,yv−1⋅u+1,…,ym,0,0,…,0−N/2,letsξ''0=0,0,…,0−n1+1,ξn1+10,ξn1+20,…,ξn1+n20,0,0,…,0−l,sets the initial weightξ''0 into one message vectorξb0=ξ′′0,ξ′′0,…,ξ′′0−N/2,sets the learning rateα into one message vectorα/m=α/m,α/m,…,α/m−n1+n2,0,0,…,0−N/2,letswi=wi,0,0,…,0i=0,1,3,5,7−l,sets the message vectorsωi=wi,wi,…,wii=0,1,3,5,7−N/2,sets the listsXa=xa,0,xa,1,…,xa,v−1,Xb=xb,0,xb,1,…,xb,v−1,Y=y0,y1,…,yv−1.Training:3:Pb computes ξ0=Add_Plainξa0,ξb04:for (i=0 to v−1) do5:Pb computes Xi=Add_PlainXa,i,Xb,i6:end for7:for (j=0 to s−1) do8:for (i=0 to v−1) do9:Pb computes Di=Mulξj,Xi,rka10:Pb computes Ei=Rot_Sum_1Di,l,gka11:Pb computes Fi=Approx_SigmoidEi,ω0,ω1,ω3,ω5,ω7,rka12:Pb computes Gi=Sub_PlainFi,Yi13:Pb computes Hi=Rot_Sum_2Gi,l,gka14:Pb computes Ii=MulHi,Xi,rka15:end for16:Pb computes a=Add_ManyI17:Pb computes b=Rot_Sum_3a,l,gka18:Pb computes c=Mul_Plainb,α/m,rka19:Pb computes ξ^j+1=Subξj,c20:Pb chooses random message vector δj+1=δ0j+1,δ1j+1,…,δN/2−1j+121:Pb computes φj+1=Sub_Plainξ^j+1,δj+122:Pb sends φj+1 to Pa23:Pa computes φj+1=Decφj+1,ska to Pa24:Pa sets φ^j+1=φ0j+1,φ1j+1,…,φn1+n2j+1,0,0,…,0−l25:Pa sets φ˜j+1=φ^j+1,φ^j+1,…,φ^j+1−N/226:Pa sets φ˜j+1=Encφ˜j+1,pka27:Pa sends φ˜j+1 to Pb28:Pb sets δ^j+1=δ0j+1,δ1j+1,…,δn1+n2j+1,0,0,…,0−l29:Pb sets δ˜j+1=δ^j+1,δ^j+1,…,δ^j+1−N/230:Pb computes ξj+1=Add_Plainφ˜j+1,δ˜j+131:end forReconstructing:32:Pa sends φn1+1s,φn1+2s,…,φn1+n2s to Pb33:Pb computes ξn1+1s,ξn1+2s,…,ξn1+n2s=φn1+1s,φn1+2s,…,φn1+n2s+δn1+1s,δn1+2s,…,δn1+n2s34:Pb sends δ0s,δ1s,…,δn1s to Pa35:Pa computes ξ0s,ξ1s,…,ξn1s=φ0s,φ1s,…,φn1s+δ0s,δ1s,…,δn1s36:return: ξ0s,ξ1s,…,ξn1s and ξn1+1s,ξn1+2s,…,ξn1+n2s for Pa and Pb respectively


## 5. Performance Evaluation

We execute the performance comparisons among ourP2 VCLR scheme and existing schemes [9, 21]. We perform all experiments on a 64-bits Linux system machine with i7 CPU and 16 GB memory. For all experiments, we choose the initial weights ξ00,ξ10,…,ξn10=0,0,…,0, ξn1+1s,ξn1+2s,…,ξn1+n2s=0,0,…,0, the learning rate α=0.15, and the maximum number of iterations s=20. The schemes [9, 21] choose the Paillier cryptosystem [25] to provide the additive HE operations, the proposed scheme uses the Microsoft SEAL library [41] to instantiate the HE operations [29]. To achieve κ=80 bits security, for the schemes [9, 21], we set the prime number p,q=512 bits and n=1024 bits; for the proposed scheme, we choose the polynomial-degree N=215, the coefficient-modulus Q=520, and the scaling factor Δ=240. On three publicly available datasets [30]: Θ1 - Umaru Impact Study, Θ2 - Myocardial Infarction from Edinburgh, and Θ3 - Nhanes III, we compare the proposed scheme and schemes [9, 21] in terms of training time, accuracy, F1-score, AUC. Pa has the first 4 features x1−x4 of all samples of Θ1, Pb has the last 4 features and labels x5−x8,y of all samples of Θ1; Pa has the first 5 features x1−x5 of all samples of Θ2, Pb has the last 4 features and labels x6−x9,y of all samples of Θ2; Pa has the first 8 features x1−x8 of all samples of Θ3, Pb has the last 7 features and labels x9−x15,y of all samples of Θ3. We get the validity of the experimental results by using 5-fold cross-validation. All experiment results are shown as the average of 10 experiments. The performance comparisons between the proposed scheme and schemes [9,21} are described in Table 2, in which '' √'' denotes ''satisfied'' and '' ×'' means ''unsatisfied''. From Table 2, we can see that our P2 VCLR scheme outperforms existing schemes [9, 21] in both training time and model performance, and does not need a TTP coordinator.Table 2
Performance comparisons.
SaSbschemeTraining timeAccuracyF1-scoreAUCNo TTP[9]1.27 min74.1 %85.1 %0.57×Θ1:x1−x4Θ1:x5−x8,y[21]2.18 min74.1 %85.1 %0.56√Ours0.86 min74.4 %85.2 %0.58√[9]2.20 min91.3 %77.5 %0.96×Θ2:x1−x5Θ1:x6−x9,y[21]3.41 min90.9 %75.3 %0.96√Ours1.49 min92.3 %78.0 %0.96√[9]11.13 min82.7 %60.1 %0.88×Θ3:x1−x8Θ1:x9−x15,y[21]21.32 min82.7 %60.1 %0.89√Ours5.87 min85.7 %61.9 %0.91√From Figure2, we can get that, for dataset Θ1, in our scheme, the training time of the model is 0.86 min, which is decreased by nearly 32.3% and 60.6% compared with that of [9, 21], respectively; for dataset Θ2, in our scheme, the training time of the model is 1.49 min, which is reduced by almost 32.3% and 56.3% in comparison to that of [9, 21], respectively; for dataset Θ3, in our scheme, the training time of the model is 5.87 min, which is nearly 47.3% and 72.5% less than that of [9, 21], respectively.Figure 2
The training time of the model.From Figure3, we can get that, for dataset Θ1, in our scheme, the accuracy of the model is 74.4%, which has nearly 0.3% and 0.3% improvement compared with that of [9, 21], respectively; for dataset Θ2, in our scheme, the accuracy of the model is 92.3%, which has an increase of almost 1.0% and 1.4% in comparison to that of [9, 21], respectively; for dataset Θ3, in our scheme, the accuracy of the model is 85.7%, which is nearly 3.0% and 3.0% higher than that of [9, 21], respectively.Figure 3
The accuracy of the model.From Figure4, we can get that, for dataset Θ1, in our scheme, the F1-score of the model is $85.2\%$, which has nearly 0.1% and 0.1% improvement compared with that of [9, 21], respectively; for dataset Θ2, in our scheme, the F1-score of the model is 78.0%, which has an increase of almost 0.5% and 2.7% in comparison to that of [9, 21], respectively; for dataset Θ3, in our scheme, the F1-score of the model is 61.9%, which is nearly 1.8% and 1.8% higher than that of [9, 21], respectively.Figure 4
The F1-score of the model.From Figure5, we can get that, for dataset Θ1, in our scheme, the AUC of the model is 0.58, which has nearly 0.01 and 0.02 improvement compared with that of [9, 21], respectively; for dataset Θ2, in our scheme, the AUC of the model is 0.96, which is the same as that of [9, 21]; for dataset Θ3, in our scheme, the AUC of the model is 0.91, which is nearly 0.03 and 0.02 higher than that of [9, 21], respectively.Figure 5
The AUC of the model.


## 6. Security Analysis

In the semi-honest model [42], we let the parties Pa and Pb know pka, rka, gka, and only Pa has ska. The proposed P2 VCLR scheme belongs to secure two-party computation, which denotes an objective functionality ℱ⋅ℱa,ℱb. For the inputs xa,xb, where xa is from party Pa and xb is from party Pb, the outputs ℱaxa,xb,ℱbxa,xb are random. ℱaxa,xb is the output for Pa, and ℱbxa,xb is for Pb, and neither party can know more private information than its output. According to the simulation-based security [43], we perform a security analysis of our P2 VCLR scheme.Definition 1.
Letℱ be a deterministic functionality and Π be a secure two-party computation protocol to compute ℱ. Given Pa's input xa, Pb's input xb, and security level κ, the views of Pa and Pb in the protocol Π are denoted as Va=κ,xa,xb=ska,pka,rka,gka,xa,xb,ya and Vb=κ,xa,xb=pka,rka,gka,xb,yb, where ya and yb are the messages received by Pa and Pb. We think that, in semi-honest model, Π can securely calculate ℱ if there are the probabilistic polynomial-time (PPT) simulators Sa and Sb, such that(2)Sa1κ,xa,ℱaxa,xbκ,xa,xb≅Vaκ,xa,xbκ,xa,xbSb1κ,xb,ℱbxa,xbκ,xa,xb≅Vbκ,xa,xbκ,xa,xb.Theorem 1.
Assuming that thePa and Pb do not collude with each other, and the HE scheme (CKKS) [29] satisfies the semantic security, our P2 VCLR scheme can ensure the security in semi-honest model.
Security Proof. Security proof of our P2 VCLR scheme follows the simulation-based security [43]. We prove that we are able to build Sa and Sb, such that(3)SAa1κ,ya,skaκ,z,ska≅VAaκ,z,skaκ,z,skaSAb1κ,z,ybκ,z,ska≅VAbκ,z,skaκ,z,ska,where VAa and VAb denote the views of Aa and Ab, respectively. Next, we show that the above two equations are indistinguishable for the corrupted parties Aa and Ab, respectively.
Against corruptedAa: We build Sa that, when given κ, Aa's input ska and Aa's output ya, is able to simulate Aa's view in the execution of the protocol. In this respect, we then analyze Aa's view VAaκ,z,ska in the execution of the protocol. The only message Aa gets is the ciphertext z. Therefore, VAaκ,z,ska consists of Aa's secret key ska, random message vector ra and ciphertext ya. Given κ, ska, and ya, SAa generates a simulation of VAaκ,z,ska. SAa encrypts SAa with pka into ya′, and generates the output ska,ra,ya′. Therefore, we can obtain two equations as follows:(4)VAaκ,z,ska=ska,ra,ya′SAa1κ,ya,ska=ska,ra,ya′.
Through the above analysis, we are able to get that probability distribution ofAa's view and SAa's output is indistinguishable. Therefore, the proposed P2 VCLR scheme is secure against the corrupted Aa in semi-honest model.
Against corruptedAb: We build Sb that, when given κ, Ab's input z and Ab's output yb, is able to simulate Ab's view in the execution of the protocol. For this reason, we analyze Ab's view VAbκ,z,ska in the execution of the protocol. Ab does not receive any message vectors from Aa. Therefore, VAbκ,z,ska consists of Ab's input z and random message vector rb. Given κ, z, and yb, SAb generates a simulation of VAbκ,z,ska by outputting z,rb. Therefore, we have the following two equations:(5)VAbκ,z,ska=z,rbSAb1κ,z,yb=z,rb.
Through the above analysis, we are able to get that probability distributions ofAb's view and SAb's output are indistinguishable. Therefore, the proposed P2 VCLR scheme is secure against the corrupted Ab in the semi-honest model.


## 7. Conclusion

In this paper, to improve the efficiency of the collaborative LR, based on an approximate HE algorithm, we propose aP2 VCLR over vertically distributed data while realizing the security of training data and the privacy of model parameters for all parties. We then evaluate the proposed scheme on the public datasets. The evaluation results show that our P2 VCLR scheme achieves a better performance in terms of joint training time and model performance in comparison to that of existing schemes [9, 21]. Specifically, the training time of the model is decreased by almost 32.3%-72.5%; the accuracy, F1-score, and AUC of the model have nearly 0.3% - 3.0%, 0.1% - 2.7% and 0 - 0.03 improvement, respectively. In the future, we will extend our method for supporting more complex ML, and deploy our scheme for practical applications.



---

*Source: 5094830-2022-10-12.xml*