Label Leakage and Protection in Two-party Split Learning

1 INTRODUCTION . With increasing concerns over data privacy in machine learning , federated learning ( FL ) ( McMahan et al. , 2017 ) has become a promising direction of study . Based on how sensitive data are distributed among parties , FL can be classified into different categories , notable among which are horizontal FL and vertical FL ( Yang et al. , 2019 ) . In contrast to horizontal FL where the data are partitioned by examples , vertical FL considers data partitioned by features ( including labels ) . As a canonical example of vertical FL , consider an online media platform A which displays advertisements from company B to its users , and charges B for each conversion ( e.g. , a user clicking the ad and buying the product ) . In this case , both parties have different features for each user : A has features on the user ’ s media viewing records , while B has the user ’ s conversion labels . B ’ s labels are not available to A because each user ’ s purchase behaviors happen entirely on B ’ s website/app . If both parties want to jointly learn a model to predict conversion without data sharing , split learning ( Gupta & Raskar , 2018 ; Vepakomma et al. , 2018 ) can be used to split the execution of a deep network between the parties on a layer-wise basis . In vanilla split learning , before training begins , both parties use Private Set Intersection ( PSI ) protocols ( Kolesnikov et al. , 2016 ; Pinkas et al. , 2018 ) to find the intersection of their data records and achieve an example ID alignment . This alignment paves the way for the split training phase . During training ( Figure 1 ) , the party without labels ( nonlabel party ) sends the intermediate layer ( cut layer ) outputs rather than the raw data to the party with labels ( label party ) , and the label party completes the rest of the forward computation to obtain the training loss . To compute the gradients with respect to model parameters , the label party initiates backpropagation from its training loss and computes its own parameters ’ gradients . To allow the non-label party to also compute gradients of its parameters , the label party also computes the gradients with respect to the cut layer outputs and communicates this information back to the non-label party . As a result of the ID alignment , despite not knowing the label party ’ s raw label data , the non-label party can identify the gradient value returned by the label party for each example . At first glance , the process of split learning appears privacy-preserving because only the intermediate computations of the cut layer—rather than raw features or labels—are communicated between the two parties . However , such “ gradient sharing ” schemes have been shown to be vulnerable to privacy leakage in horizontal FL settings ( e.g. , Zhu et al. , 2019 ) . In vertical FL ( and specifically split learning ) , it remains unclear whether the raw data can similarly be leaked during communication . In particular , as the raw labels often contain highly sensitive information ( e.g. , what a user has purchased ( in online advertising ) or whether a user has a disease or not ( in disease prediction ) Vepakomma et al . ( 2018 ) ) , developing a rigorous understanding of the threat of label leakage and its protection is particularly important . Towards this goal , we make the following contributions : 1 . We formalize a threat model for label leakage in two-party split learning in the context of binary classification ( Section 3.1 ) , and propose specific privacy quantification metrics to measure the severity of such threats ( Section 3.2 ) . 2 . We identify two simple and realistic methods within this threat model which can accurately recover the label party ’ s private label information ( Section 3.3 ) . 3 . We propose several random perturbation techniques to limit the label-stealing ability of the non-label party ( Section 4 ) . Among them , our principled approach Marvell directly searches for the optimal random perturbation noise structure to minimize label leakage ( as measured via our quantification metric ) against a worst-case adversarial non-label party . 4 . We experimentally demonstrate the effectiveness of our protection techniques and MARVELL ’ s improved privacy-utility tradeoffs compared to other protection baselines ( Section 5 ) . 2 RELATED WORK . Privacy leakage in split learning . Although raw data is not shared in federated learning , sensitive information may still be leaked when gradients and/or model parameters are communicated between parties . In horizontal FL , Zhu et al . ( 2019 ) showed that an honest-but-curious server can uncover the raw features and labels of a device by knowing the model architecture , parameters , and communicated gradient of the loss on the device ’ s data . Based on their techniques , Zhao et al . ( 2020 ) showed that the ground truth label of an example can be extracted by exploiting the directions of the gradients of the weights connected to the logits of different classes . Here we study a different setting—two-party split learning ( in vertical FL ) ( Yang et al. , 2019 ) , where no party has access to the model architecture or model parameters of the other party . In this setting , Vepakomma et al . ( 2019 ) studied how the forward communication of feature representations can leak the non-label party ’ s raw data to the label party . We instead study whether label information may be leaked from the label party to the non-label party during the backward communication . Despite the importance of maintaining the privacy of these labels , we are unaware of prior work that has studied this problem . Privacy protection and quantification . Techniques to protect communication privacy in FL generally fall into three categories : 1 ) cryptographic methods such as secure multi-party computation ( e.g. , Bonawitz et al. , 2017 ) ; 2 ) system-based methods including trusted execution environments ( Subramanyan et al. , 2017 ) ; and 3 ) perturbation methods that shuffle or modify the communicated messages ( e.g. , Abadi et al. , 2016 ; McMahan et al. , 2018 ; Erlingsson et al. , 2019 ; Cheu et al. , 2019 ; Zhu et al. , 2019 ) . Our protection techniques belong to the third category , as we add random perturbations to the gradients to protect the labels . Many randomness-based protection methods have been proposed in the domain of horizontal FL . In this case , differential privacy ( DP ) ( Dwork , 2006 ; Dwork et al. , 2014 ) is commonly used to measure the proposed random mechanisms ’ ability to anonymize the identity of any single participating example in the model iterates . However , in split learning , after PSI , both parties know exactly the identity of which example has participated in a given gradient update . As we explain in Section 3.1 , the object we aim to protect ( the communicated cut layer gradients ) , unlike the model iterates , is not an aggregate function of all the examples but are instead example-specific . As a result , DP and its variants ( e.g . label DP ( Chaudhuri & Hsu , 2011 ; Ghazi et al. , 2021 ) ) are not directly applicable metrics in our setting , and we instead propose a different metric ( discussed in Section 3.2 ) . 3 LABEL LEAKAGE IN SPLIT LEARNING . We first introduce the two-party split learning problem for binary classification , and then formally describe our threat model and privacy quantification metrics with two concrete attack examples . 3.1 TWO-PARTY SPLIT LEARNING IN BINARY CLASSIFICATION . Problem setup . Consider two parties learning a composition model h ◦ f jointly for a binary classification problem over the domain X × { 0 , 1 } ( Figure 1 ) . The non-label party owns the representation function f : X → Rd and each example ’ s raw feature X ∈ X while the label party owns the logit function h : Rd → R and each example ’ s label y ∈ { 0 , 1 } 1 . Let ` = h ( f ( X ) ) be the logit of the positive class whose predicted probability is given through the sigmoid function : p̃1 = 1/ ( 1 + exp ( − ` ) ) . We measure the loss of such prediction through the cross entropy loss L = log ( 1 + exp ( − ` ) ) + ( 1 − y ) ` . During model inference , the non-label party computes f ( X ) and sends it to the label party who will then execute the rest of forward computation in Figure 1 . Model training ( Figure 1 : backward gradient computation ) . To train the model using gradient descent , the label party starts by first computing the gradient of the loss L with respect to the logit dL d ` = ( p̃1 − y ) . Using the chain rule , the label party can then compute the gradient of L with respect to its function h ’ s parameters and perform the gradient updates . To also allow the non-label party to learn its function f , the label party needs to additionally compute the gradient with respect to cut layer feature f ( X ) and communicate it to the non-label party . We denote this gradient by g : = ∇f ( X ) L = ( p̃1 − y ) ∇zh ( z ) |z=f ( X ) ∈ Rd ( by chain rule ) . After receiving g , the non-label party continues the backpropagation towards f ’ s parameters and also perform the gradient updates . Why Not Differential Privacy ? Note that for a given iteration , the non-label party randomly chooses B example IDs to form a batch . Therefore , the identity of which examples are used is known to the non-label party by default . In addition , the communicated features f ( X ) and returned gradients g will both be matrices in RB×d with each row belonging to a specific example in the batch . The different gradients ( rows of the matrix ) are not with respect to the same model parameters , but are instead with respect to different examples ’ cut-layer features ; thus , no averaging over or shuffling of the rows of the gradient matrix can be done prior to communication to ensure correct computation of f ’ s parameters on the non-label party side . This example-aware and example-specific nature of the communicated gradient matrix makes differential privacy ( which focuses on anonymizing an example ’ s participation in an aggregate function ) inapplicable for this problem ( see also Section 2 ) . 3.2 THREAT MODEL AND PRIVACY QUANTIFICATION . Below we specify several key aspects of our threat model , including the adversary ’ s objective and capabilities , our metric for quantifying privacy loss , and the possible inclusion of side information . Adversary ’ s objective . At a given moment in time during training ( with f and h fixed ) , since the communicated cut layer gradient g is a deterministic function of f ( X ) and y ( see Section 3.1 ) , we consider an adversarial non-label party whose objective is to recover the label party ’ s hidden label y based on the information contained in g for every training example . Adversary ’ s capability . We consider an honest-but-curious non-label party which can not tamper with training by selecting which examples to include in a batch or sending incorrect features f ( X ) ; instead , we assume that the adversary follows the agreed-upon split training procedure while trying to guess the label y . This can be viewed as a binary classification problem where the ( input , output ) distribution is the induced distribution of ( g , y ) . We allow the adversary to use any binary classifier q : Rd → { 0 , 1 } to guess the labels . This classifier can be represented by a ( scoring function r , threshold t ) tuple , where r : Rd → R maps an example ’ s cut layer gradient to a real-valued score and the threshold t ∈ R determines a cut-off so that q ( g ) = 1 if r ( g ) > t and q ( g ) = 0 if r ( g ) ≤ t. Moving forward , we use this tuple representation to describe adversarial non-label party classifiers . Privacy loss quantification . As we consider binary classification , a natural metric to quantify the performance of an adverary ’ s scoring function r is the AUC of its ROC curve . Denote the unperturbed class-conditional distributions of the cut-layer gradients by P ( 1 ) and P ( 0 ) for the 1To simplify notation , we assume no additional features in the label party to compute the logit . The data leakage problem still holds true for other more complicated settings ( see WDL experiment setting in Section 5 ) . positive and negative class , respectively . The ROC curve of a scoring function r is a parametric curve t 7→ ( FPRr ( t ) , TPRr ( t ) ) ∈ [ 0 , 1 ] 2 which maps a threshold value t ∈ R to the corresponding ( False Positive Rate , True Positive Rate ) tuple of the classifier represented by ( r , t ) , with FPRr ( t ) : = P ( 0 ) ( { g : r ( g ) > t } ) and TPRr ( t ) : = P ( 1 ) ( { g : r ( g ) > t } ) . The AUC of the ROC curve of a scoring function r ( denote by AUC ( r ) ) can be expressed as an integral : AUC ( r ) = ∫ −∞ ∞ TPRr ( t ) dFPRr ( t ) ∈ [ 0 , 1 ] ( Leak AUC ) ( more details on this expression see Appendix A.1 . ) We use this value as the privacy loss quantification metric for a specific adversary scoring function r and refer to it as the leak AUC . This metric summarizes the predictive performance of all classifiers that can be constructed through all threshold values t and removes the need to tune this classifier-specific hyperparameter . The leak AUC being close to 1 implies that the corresponding scoring function r can very accurately recover the private label , whereas a value of around 0.5 means r is non-informative in predicting the labels . In practice , during batch training , the leak AUC of r can be estimated at every gradient update iteration using the minibatch of cut-layer gradients together with their labels . Side information . Among all the scoring functions within our threat model , it is conceivable that only some would recover the hidden labels accurately . Picking such effective ones would require the non-label party to have population-level side information specifically regarding the properties of ( and distinction between ) the positive and negative class ’ s cut-layer gradient distributions . Since we allow the adversary to pick any specific ( measurable ) scoring function , we implicitly allow for such population-level side information for the adversary . However , we assume the non-label party has no example-level side information that is different example by example . Thus we also don ’ t use local DP for privacy quantification ( detailed explanation in Appendix A.8 ) . Next we provide two example scoring functions which use population-level side-information to effectively recover the label .

Summary: ## Summary of Contributions  This paper studies the multiparty setting where there are two parties: the first party (aka non-label party) holds the feature vectors $X_i$ of user $i$ whereas the second party (aka label party) holds the corresponding labels $y_i \in \{0, 1\}$. Together they wish to jointly train a model in such a way that the non-label party does not learn of the users' labels. This is especially relevant in online advertisement (where the label party corresponds to the advertiser who knows whether each user converts and the non-label party is e.g. the publisher who shows the ads) and in medical applications (where the label party is the hospital).  The paper focuses on *split training* which is a setting where the model is divided among two parties with the first layers (denoted as a function $f$) is with the first party and the remaining layers (denoted as a function $h$) are with the second party. Here the training is performed as follows. First, the non-label party computes $f(X)$ (referred to as the "cut layer") and send this to the label party. The label party then compute the gradient w.r.t. $h$ and update its parameters and furthermore it computes the gradient $g$ w.r.t. the cut layer $f(X)$ and sends it back to the non-label party. The non-label party can then use backpropagation starting with $g$ to update its own parameters. The main concern tackled in this paper is that, in such a scheme, the gradient $g$ sent back to the non-label party can leak the information about the label. Specifically, the authors observe and experimentally show that the following two attacks are very effective in predicting the labels given the intermediate gradient $g$: - Consider the norm $\\|g\\|_2$ and make prediction based on whether it exceeds a certain threshold. - Consider the cosine similarity between $g$ and another (fixed) gradient, and then threshold. In fact, the two above attacks get near-perfect predictions in some examples. The exact measure they use to evaluate their model is the area-under-curve (AUC) of the (false positive rate, true positive rate) curve.  The authors then propose to add noise in order to mitigate such attacks. The baseline here would be to add isotopic Gaussian noise (with a certain scale). The authors then formulate an optimization problem aiming to achieve better privacy-utility tradeoff. Roughly speaking, this is an optimization problem of the form "minimize AUC" such that "utility is at least ...". It is hard to make this precise (and it is probably inefficient), so the authors use certain proxy for both the objective and the constraint. For the objective, the authors show that AUC is upper bound by a certain quantity involving the KL divergence of the distributions of the gradients from the two labels. For the utility constraint, the authors instead try to restrict the amount of the noise added by having an upper bound on the trace of their covariance matrices. Furthermore, the authors show that when the (unperturbed) gradient distributions are assumed to be Gaussians and only Gaussian noises are considered, then the optimization problem simplifies greatly to just a single constant-size optimization problem (Theorem 2). This final form constitutes their so-called Marvell algorithm. Finally, the paper concludes by several experimental results showing that Marvell is very effective against protecting the two aforementioned attacks in real-world datasets and models, while preserving reasonable level of privacy.