Label Leakage and Protection in Two-party Split Learning
Authors: Oscar Li
Jiankai Sun
Xin Yang
Weihao Gao
Hongyi Zhang
Junyuan Xie
Virginia Smith
Chong Wang
Venue: ICLR 2022 Poster (2022)
Keywords: Split Learning
label leakage
privacy
privacy protection

Abstract: Two-party split learning is a popular technique for learning a model across feature-partitioned data. In this work, we explore whether it is possible for one party to steal the private label information from the other party during split training, and whether there are methods that can protect against such attacks. Specifically, we first formulate a realistic threat model and propose a privacy loss metric to quantify label leakage in split learning. We then show that there exist two simple yet effective methods within the threat model that can allow one party to accurately recover private ground-truth labels owned by the other party. To combat these attacks, we propose several random perturbation techniques, including $\texttt{Marvell}$, an approach that strategically finds the structure of the noise perturbation by minimizing the amount of label leakage (measured through our quantification metric) of a worst-case adversary. We empirically demonstrate the effectiveness of our protection techniques against the identified attacks, and show that $\texttt{Marvell}$ in particular has improved privacy-utility tradeoffs relative to baseline approaches.

Published as a conference paper at ICLR 2022
LABEL LEAKAGE AND PROTECTION IN TWO-PARTY SPLIT LEARNING
Oscar Jiankai Sun Xin Yang', Weihao Gao², Hongyi Zhang", Junyuan Xie², Virginia Smith', Chong Wang"
Carnegic Mellon University 2 BayeDanne Inc.
ABSTRACT
Two-party split learning is a popular technique for learning a model across feature- partitioned data. In this work, we explore whether it is possible for one party steal the private label information from the other party during split training, and whether there are methods that can protect against such attacks. Specifically, we first formulate a realistic threat model and propose privacy loss metric to quan- tify label leakage in split learning. We then show that there exist two simple yet effective methods within the threat model that can allow one party to accurately recover private ground-truth labels owned by the other party. To combat these at- tacks, we propose several random perturbation techniques, including Marve 11, an approach that strategically finds the structure of the noise perturbation by min- imizing the amount of label leakage (measured through our quantification metric) of worst-case adversary. We empirically' demonstrate the effectiveness of our protection techniques against the identified attacks, and show that Marvell in particular has improved privacy-utility tradeoff's relative to baseline approaches.
INTRODUCTION
With increasing concerns over data privacy in machine baariing.federatio learning (FL) (McMahan et al., 2017) has become promising direction of study. Based on how sensitive data are distributed among parties, FL can classified into different categories, notable among which are horizontal FL and vertical FL (Yang et al., 2019). In contrast to horizontal FL where the data partitioned by examples, vertical FL considers data partitioned by features (including labels). As a canonical example of vertical FL, consider online media platform A which displays advertisements from company B to its users, and charges B for each conversion (e-g., a user clicking the ad and buying the product). In this case, both parties have different features for each user: A has features the user's 's media viewing records, while B has the user's conversion labels. B's labels are not available A because each user's purchase behaviors happen entirely on B's website/app.
If both parties want to jointly learn a model to predict conversion without data sharing, split learn- ing (Gupta & Raskar, 2018; Vepakomma et al., 2018) can be used to split the execution of deep network between the parties on a layer-wise wise basis. In vanilla split learning, before training begins, both parties use Private Set Intersection (PSI) protocols (Kolesnikov et al., 2016; Pinkas al., 2018) to find the intersection of their data records and achieve an example ID alignment. This alignment paves the way for the split training phase. During training (Figure 1), the party without labels (non- label party) sends the intermediate layer (cul layer) outputs rather than the raw data to the party with labels (label party), and the label party completes the rest of the forward computation to obtain the training loss. To compute the gradients with respect to model parameters, the label party initiates buckpropagation from its training loss and computes its own parameters' gradients. To allow the non-label party to also compute gradients of its parameters, the label party also computes the gradi- ents with respect to the cut layer outputs and communicates this information back to the non-label party. As result of the ID alignment, despite not knowing the label party's 's raw label data, the non-label party can identify the gradient value returned by the label party for each example.
At first glance, the process of split learning appears ate computations of the cut layer-rather than raw features or labels-are communicated between the two parties. However, such "gradient sharing" schemes have been shown to be vulnerable to privacy leakage in horizontal FL settings (c.g., Zhu et al., 2019). In vertical FL (and specifically split learning). remains unclear whether the raw data can similarly be leaked during communi-
because only the intermedi-
"wor done during internship at ByteDance Inc. dixcarli@cmu.co jjiankai sun @ bytedance.com Code available htts://gthuh.com//xcaree pataeaaaaaaalllllleeepttttttttti

---

Published as a conference paper at ICLR 2022
non-label party
cut layer
label party
posítive logit é h(f(x)
cross entropy
features
user
f(x)
communication
f(X)
ft
Can g leak y? Can we protect against this?
{0,
1 m
no
buy
weights of
buy
of
computation
backward
= V.
dl.
dl
computation
V, Atz)
de
dL
Figure I: Communication diagram of two-party split training for example of online advertising. We study whether it is possible for the communicated gradient g to leak private label information.
cation. In particular, as the raw labels often contain highly sensitive information (e.g.. what user has purchased (in online advertising) or whether a user has a disease or not (in disease predic- tion) Vepakomma et al. (2018)), developing a rigorous understanding of the threat of label leakage and its protection is particularly important. Towards this goal, we make the following contributions:
1. We formalize threat model for label leakage in two-party split learning in the context of binary classification (Section 3.1), and propose specific privacy quantification metries to measure the severity of such threats (Section 3.2).
2. We identify two simple and realistic methods within this threat model which can accurately recover the label party's 's private label information (Section 3.3).
3. We propose several random perturbation techniques to limit the label-stealing ability of the non-label party (Section 4). Among them, principled approach Marvell directly searches for the optimal random perturbation noise structure to minimize label leakage (as measured via our quantification metrie) against worst-case adversarial non-label party.
4. We experimentally demonstrate the effectiveness of our protection techniques and MARVELL'S 's improved privacy-utility tradeoffs compared to other protection baselines (Section 5).
2 RELATED WORK
Privacy leakage in split learning, Although raw data is not shared in federated learning, sensitive information may still be leaked when gradients and/or model parameters are communicated between parties. In horizontal FL, Zhu et al. (2019) showed that an conest-but-curious server can uncover the raw features and labels of a device by knowing the model architecture, parameters, and com- municated gradient of the loss on the device's 's data. Based on their techniques, Zhao et al. (2020) showed that the ground truth label of an example can be extracted by exploiting the directions of the gradients of the weights connected to the logits of diffferent classes. Here we study different Betting-two-par split learning (in vertical FL) (Yang et al., 2019), where no party has access to the model architecture model parameters of the other party. In this setting, Vepakomma et al. (2019) studied how the forward communication of feature representations can leak the non-label party's raw data to the label party. We instead study whether label information may be leaked from the label party to the non-label party during the backward communication. Despite the importance of maintaining the privacy of these labels, we are unaware of prior work that has studied this problem.
Privacy protection and quantification. Techniques to protect communication privacy in FL gen- erally fall into three categories: 1) cryptographic methods such as secure multi-party computation (e.g., Bonawitz et al., 2017); 2) system- based methods including trusted execution environments (Subramanyan et al., 2017); and 3) perturbation methods that shuffle or modify the communicated messages (e.g., Abadi et al., 2016; McMahan et al., 2018 Erlingsson et al., 2019; Cheu et al., 2019; Zhu et al., 2019). Our protection techniques belong to the third category, as we add random per- turbations to the gradients to protect the labels. Many randomness-based protection methods have been proposed in the domain of horizontal FL. In this case, differential privacy (DP) (Dwork, 2006; Dwork et al., 2014) is commonly used to measure the proposed random mechanisms' ability to anonymize the identity of any single participating example in the model iterates. However, in split learning, after PSI, both parties know exactly the identity of which example has participated in given gradient update. As we explain in Section 3.1, the object we aim to protect (the communi- cated cut layer gradients), unlike the model iterates, is not an aggregate function of all the examples but are instead example-specific. As a result, DP and its variants (e.g. label DP (Chaudhuri & Hsu, 201 Ghazi et al., 2021)) are not directly applicable metries in our setting, and we instead propase a different metric (discussed in Section 3.2).
2

---

Published as a conference paper at ICLR 2022
3 LABEL LEAKAGE IN SPLIT LEARNING
We first introduce the two-party split learning problem for binary classification, and then formally describe our threat model and privacy quantification metrics with two concrete attack examples.
3.1 TWO-PARTY SPLIT LEARNING IN BINARY CLASSIFICATION
Problem setup. Consider two parties learning a composition model h o f jointly for a binary classification problem over the domain X' x {0, 1} (Figure 1). The non-label party owns the repre- sentation function f X + R² and each example's raw feature X X' while the label party owns the logit function h : IR + R. and each example's label y {0. 132. Let & h(f(X)) be the logit of the positive class whose predicted probability is given through the sigmoid function: Pi 1/(1 + exp(-E)). We measure the loss of such prediction through the cross entropy loss L = log(1 1 + exp(-()) + (1 y).. During model inference, the non-label party computes f f(X) and sends it to the label party who will then execute the rest of forward computation in Figure 1.
Model training (Figure I: backward gradient computation). To train the model using gradient descent, the label party starts by first computing the gradient of the loss L with respect to the logit d = (pi y). Using the chain rule, the label party can then compute the gradient of L with respect to its function hts parameters and perform the gradient updates. To also allow the non-label party to learn its function f. the label party needs to additionally compute the gradient with respect to cut layer feature f(X) and communicate it to the non-label party. We denote this gradient by = FXxx- = (p1 y) (=)((==)=sst E Rd (by chain rule). After receiving g. the non-label party continues the backpropagation towards f's parameters and also perform the gradient updates.
Why Not Differential Privacy? Note that for a given iteration, the non-label party randomly chooses B example IDs to form a batch. Therefore, the identity of which examples are used is known to the non-label party by default. In addition, the communicated features f f(X) and returned gradients g will both be matrices in R Bxd with each row belonging to specific example in the batch. The different gradients (rows of the matrix) are not with respect to the same model parameters, but are instead with respect to different examples' cut-layer features; thus, no averaging over or shuffling of the rows of the gradient matrix can be done prior to communication to ensure correct computa- tion of f's parameters on the non-label party side. This example- awwre and ccample-peccife nature of the communicated gradieni mairix makes differential privacy (which focuses on anonymizing an example's participation in an aggregate function) inapplicable for this problem (see also Section 2).
3.2 THREAT MODEL AND PRIVACY QUANTIFICATION
Below we specify several key aspects of our threat model, including the adversary's objective and capabilities, our metric for quantifying privacy loss, and the possible inclusion of side information.
Adversary's objective. At given moment in time during training (with and h fixed), since the communicated cut layer gradient is a deterministic function of (see Section 3.1), we consider an adversarial non-label party whose objective is to recover the label party's hidden label y based on the information contained in for every training example.
Adversary's capability. We consider an honest- prreest-butcuune burtius non-label party which cannot tamper with training by selecting which examples to include in a batch or sending incorrect features f(X); instead, we assume that the adversary follows the agreed-upon split training procedure while trying to guess the label y. This can be viewed as binary classification problem where the (input, output) distribution is the induced distribution of (g. y). We allow the adversary to use any binary classifier q : Rd + {0, 1} to guess the labels. This classifier can be represented by a (scoring function r, threshold f) tuple, where r Rdd + IR maps an example's 's cut layer gradient to a real-valued score and the threshold E R determines cut-off so that ((g) = 1 if r(g) > t and q(g) = 0 if r(g) < t. Moving forward, we use this tuple representation to describe adversarial non-label party classifiers.
Privacy loss quantification. As consider binary classification, a natural metric to quantify the performance of an adverary's scoring function r is the AUC of its ROC curve. Denote the unperturbed class-conditional distributions of the cut-layer gradients by P(1) and p(0) for the
TTo simplify notation, we assume additional features in the label party to compute the logit. The data leakage problem still holds true for other more complicated settings (see WDL experiment setting in Section 5).
3

---

Published as a conference paper at ICLR 2022
[P]
[8ght22 moll
118
100000 Roceboo
co5iV 0.hi2l)2--va
coslg.9.>
Figure 2: Distributions of quantitics discussed in Observations 1-4 after the first 100, 200, 300, 400, 500 steps of stochastic gradient descent training of the WDL model on Criteo (see experiments).
positive and negative class, respectively. The ROC curve of scoring function is a paramet- ric curve f + (t). (t)) E [0. 1J2 which maps a threshold value E R to the corre- sponding (False Positive Rate, True Positive Rate) tuple of the classifier represented by (r, t), with FPR, (t) ((g) r(g) t}) and (t), = ((9 r(g) > f}). The AUC of the ROC
curve of a scoring function r (denote by AUC(r)) can be expressed as an integral:
AUC(r) - TPR, (L) (I)R, E [0,
(Leak AUC)
(more details on this expression see Appendix A.I.) We this value as the privacy loss quantifi- cation metric for specific adversary scoring function r and refer to it as the leak AUC. This metric summarizes the predictive performance of all classifiers that can be constructed through all threshold values t and removes the need to tune this classifier-specifie hyperparameter. The leak AUC being close to 1 implies that the corresponding scoring function r can very accurately recover the private label, whereas a value of around 0.5 means r is non- -on-ifformative in predicting the labels. In practice, during batch training, the leak AUC of r can be estimated every gradient update iteration using the minibatch of cut-layer gradients together with their labels.
Side information. Among all the scoring functions within our threat model, is conceivable that only some would recover the hidden labels accurately. Picking such effective ones would require the non-label party to have population-level side information specifically regarding the properties of (and distinction between) the positive and negative class's 's cut-layer gradient distributions. Since allow the adversary to pick any specific (measurable) scoring function, we implicitly allow for such population-level level side information for the adversary. However, example-level side information that is different example by example. Thus we also don't use local DP for privacy quantification (detailed explanation in Appendix A.8). Next we provide two example scoring functions which use population-level side-information to effectively recover the label.
assume the non-label party has
3.3 PRACTICAL ATTACK METHODS
(aah(aal==fx)) lla. We Attack 1: Norm-based scoring function. Note that ll9ll2 po - yl 1
make the following observations for P1 - yl and [[ which hold true for a wide range of real-world learning problems.
Observation 1: Throughout training, the model tends to be less confident about a positive ex- ample being positive than a negative example being negative. In other words, the confidence gap of positive example 1 1-6 - 15 y (when y 1) is typically larger than the confidence gap of negative example 1 PO P1 [P] ---- yl (when y = 0) (see Figure 2(a)). This observation is particularly true for problems like advertising conversion prediction and disease prediction, where there is inherently more ambiguity for the positive class than the negative. For example, advertising, uninterested users of product will never click on its ad and convert, but those interested, even after clicking, might make the purchase only a fraction of the time depending on time/money constraints. (See A.2 for such an ambiguity even for a class-balanced setting.)
Observation 2: Throughout training, the norm of the gradient vector [ llz is on the same order of magnitude (has similar distribution) for both the positive and negative examples (Figure 2(b)). This is natural because vah(a)l==(()) not function of y.
4

---

Published as a conference paper at ICLR 2022
As a consequence of Observation 1 and 2, the gradient norm || of the positive instances generally larger than that of the negative ones (Figure 2(c)). Thus, the scoring function (g)g)= = 1gg12 is strong predictor of the unseen label y. We name the privacy loss (leak AUC) measured against the attack In the norm leak AUC. In Figure 2(c), the norm leak AUCs are consistently above 0.9, signaling high level of label leakage throughout training.
Attack 2: Direction-based scoring function. We now show that the direction of g (in addition to its magnitude) can also leak the label. For pair of examples, (Xa, Ma), (Xb, 2h),, let their respective predicted positive class probability be Pr.a. PI. and their communicated gradients be go 9b- Let ous Rd Rd - R denote the cosine similarity function cus(ya. go)) = gā 90 9 ||a||96 la). It is easy to see that cos(ge, 9h) sgn(p1... Ma) sgn(pi.. yo.) cos(( V=h(=))==((xx,) V. F= where sgn(x) the sign function which returns if x 2 O, and -1 i < O. We highlight two additional observations that can allow us to use cosine similarity to recover the label.
Observation 3: When the examples a, are of different classes, the term sgn(ØL. - yo) sgn(P1.. - yo) = -1 is negative. On the other hand, when examples a, b are of the same class (both positive/both negative), this product will have a value of and thus be positive.
Observation 4: Throughout training, for any two examples a, b, their gradients of the function h always form an acute angle, i.e. cos( (/l(VV/()===aa) neural networks that use monotonically increasing activation functions (such as ReLU, sigmoid, tanh), this is caused by the fact that the gradients of these activation functions with respect to its inputs are coordinatewise nonnegative and thus always lie in the first closed hyperorthant.
V v=h(=))==/(xx)) > O (Figure 2(d)). For
Since cos(ga- 9b) is the product of the terms from Observation 3 and 4, we see that for given example, all the examples that are of the same class result in a positive cosine similarity, while all opposite class examples result in negative cosine similarity. If the problem is class-imbalanced and the non-label party knows there are fewer positive examples than negative ones, can thus determine the label of each example: the class is negative if more than half of the examples result in positive cosine similarity; otherwise it is positive. For many practical applications, the non-label party may reasonably guess which class has more examples in the datasel priori without ever seeing any data- for example, in disease prediction, the percentage of the entire population having certain disease is almost always much lower than 50%; in online advertising conversion prediction, the conversion rate (fraction of positive examples) is rarely higher than 30%. Note that the non- cobelabel party doesn't 't need knowledge of the exact sample proportion of each class for this method to work.
To simplify this attack for evaluation, we consider an even worse oracle scenario where the non-label party knows the clean gradient of one positive example 94. Unlike the aforementioned practical majority counting attack which needs to first figure out the direction of one positive gradient, this oracle scenario assumes the non-label party is directly given this information. Thus, any protection method capable of defending this oracle attack would also protect against the more practical one. With g+ given, the direction- based scoring function ra is simply ra(g) cos(g, g+). We name the privacy loss (leak AUC) against this oracle attack ra the cosine leak AUC. In practice, we randomly choose a positive class clean gradient from each batch as 9+ for evaluation. For iterations in Figure 2(e), the cosine leak AUC all have the highest value of 1 (complete label leakage).
LABEL LEAKAGE PROTECTION METHODS
In this section, we first introduce a heuristic random perturbation approach designed to prevent the practical attacks identified in Section 3.3. We then propose a theoretically justified method that aims to protect against the entire class of scoring functions considered in our threat model (Section 3.2).
4.1 A HEURISTIC PROTECTION APPROACH
Random perturbation and the isotropic Gaussian baseline. To protect against label leakage, the label party should ideally communicate essential information about the gradient without commu- nicating its actual value. Random perturbation methods generally aim to achieve this goal. One obvious consideration for random perturbation is to keep the perturbed gradients unbiased. In other words, suppose is the perturbed version of an example's 's true gradient y, then we want Elğ 9] = g. By chain rule and linearity of expectation, this ensures the computed gradients of the label party's 's parameters ff will also be unbiased, a desirable property for stochastic optimization. Among unbiased perturbation methods, a simple approach is to add iid isotropic Gaussian noise to every gradient to mix the positive and negative gradient distribution before sending to the non-label party.
5

---

Published as a conference paper at ICLR 2022
Although isotropic Gaussian noise is valid option, it may not be optimal because 1) the gradients are vectors but not scalars, so the structure of the noise covariance matrix matters. Isotropic noise might neglect the direction information; 2) due to the asymmetry of the positive and negative gradi- ent distribution, the label party could add noise with different distributions to each class's ' gradients.
Norm-alignment heuristic. We now introduce improved heuristic approach of adding zero- mean Gaussian noise with non-isotropic and example-dependent covariance. [Magnitude choice] As we have seen that ||9]|2 can be different for positive and negative examples and thus leak label information, this heuristic first aims to make the norm of each perturbed gradient indistinguishable from one another. Specifically, we want to match the expected squared 2-norm of every perturbed gradient in a mini-batch to the largest squared 2-norm in this batch (denote by [|9 12). [Direction choice] In addition, as we have seen empirically from Figure 2(e), the positive and negative gradients lie close to a -ieddimenioonal line in R", with positive examples pointing in one direction and negative examples in the other. Thus we consider only adding noise (roughly speaking) along "this line". More concretely, for a gradient 9j in the batch, we add a zero-mean Gaussian noise vector 7 supported only on the one-dimensional space along the line of 9j. In other words, the noise's covariance is the rank-1 1 matrix Cov[n;] = -]9,9, To calculate jj. we aim to match Elllg, + l1å] Since we align to the maximum
| ll2. A simple calculation gives o, v Vllgtmax Bls lB norm, we name this heuristic protection method max norm. The advantage of max norm is that it has no parameter to tune. Unfortunately, it does not have a strong theoretical motivation, cannot flexibly trade-off between model utility and privacy, and may be broken by some unknown attacks.
4.2 OPTIMIZED PERTURBATION METHOD: MARVELL
Motivated by the above issues of max_norm, we next study how to achieve a more principled trade-off between model performance (utility) and label protection (privacy). To do so, we directly minimize the worst-case adversarial scoring function's 's leak AUC under a utility constraint. We name this protection method Marvell (optiMized perturbAtion to pRe Label Leakage).
Noise perturbation structure. Due to the distribution difference between the positive and nega- tive class's cut layer gradients, we consider having the label party additively perturb the randomly sampled positive (()) and negative g0)) gradients with independent zero-mean random noise vectors y1t)) and (7 with possibly different distributions (denoted by D and D( We use f00) and pro) to denote the induced perturbed positive and negative gradient distributions. Our goal is find the optimal noise distributions (1)) and y( by optimizing our privacy objective described below.
Privacy protection optimization objective. As the adversarial non-label party in our threat model is allowed to use any measurable scoring function r for label recovery, we aim to protect against all such scoring functions by minimizing the privacy loss of the worst case scoring function measured through our leak AUC metric. Formally, our optimization objective is (einpp))ppo _pto) max, AUC(r). Here to compute AUC(r), the FPR, (f) and (t), needs to be computed using the perturbed distributions f(0)) and pico) instead of the unperturbed p()) and pro) (Section 3.2). Since AUC is difficult to directly optimize, we consider optimizing an upper bound through the following theorem:
Theorem 1. For 0 S < 4 and any perturbed gradient distributions po) and p(o) that are abso- lutely continuous with respect to each other,
kL(( || p(0) + kL((5()) || (())) Se
implies max, (UC(r) -
From Theorem (proof in Appendix A.3), we see that as long as the sum KL divergence is below 4, the smaller sumKL is, the smaller max,- AUC(r) is. V2/2 - c/88 decreases as decreases.)
Thus we can instead minimize the sum KL divergence between the perturbed gradient distributions:
sumKL" := min KL(P()) ((0) + kL( (()).
(I)
Utility constraint. In an extreme case, we could add infinite noise to both the negative and positive gradients. This would minimize (1) optimally to and make the worst case leak AUC 0.5, which is equivalent to a random guess. However, stochastic gradient descent cannot converge under infinitely large noise, so it is necessary to control the variance of the added noise. We thus introduce the noise power constraint: p teiitpppprrrr (r(Cov(z(") + (1 - p) tr (r(Cov(n < P. where p is the fraction of positive examples (already known to the label party); tr (r(Covnn"") denotes the trace of the covariance matrix of the random noise ( and the upper bound P is a tunable hyperparameter to control the level of noise: larger P would achieve a lower SLmKl. and thus lower worst-case leak AUC and better privacy; however, it would also add more noise to the gradients, leading to slower optimization

---

Published as a conference paper at ICLR 2022
convergence and possibly worse model utility. We weight each class's noise level tr(Cov'n D by its
example proportion (p or 1-p) p) since, from an optimization perspective, we want to equally control every training example's 's gradient noise. The constrained optimization problem becomes:
min KL((()) n(0)
p00)- - KL(P(O) || (50)) s.t. p. p.tr(Cov(y/)) + (1 (1-p) tr(Covly (0 P. P. (2)
Optimizing the objective in practice. To solve the optimization problem we first introduce some modelling assumptions. We assume that the unperturbed gradient of each class follows a Gaussian distribution: gt) ~ NN vlaxa) and gf ~ N (N ulaxd). Despite this being an approximation. as we see later in Section it can achieve strong protection quality against our identified attacks. In addition, it makes the optimization easier (see below) and provides us with insight on the optimal noise structure. We also search for perturbation distributions that are Gaussian: D (0, 2n) and ( = (0, Mo) with commuting covariance matrices: L, = MnLi,. The commutative require- ment slightly restricts our search space but also makes the optimization problem more tractable. Our goal is to solve for the optimal noise structure, i.e. the positive semidefinite covariance matrices Eo,
Σn. Let Ag = g()) - g()) denote the difference between the positive and negative gradient's 's mean vectors. We now have the following theorem (proof and interpretation in Appendix A.4):
Theorem 2. The optimal Ei and Eig to (2) with the above assumptions have the form:
Σ 10 X
11))
X
(0)+
(Ag) + 22 Ia.
Σg
(Ag)(Ag) + X Ia,
(3)
A
where (2 () 2
(0)
(1)x X ) is the solution to the following 4-variable optimization problem:
(d- 1) 1 + u
+
x + a
||
( + ++ 1g
min
+ (d
A2
A +
1/2 +
x +v
x +
(1) p(d 1) (1 1)
+ + - p) 1) S P.
-2 S
0,
- S 0, ~2p00
1) S 0, - S 0, 100 x s 0,
50
Additional details of Marvell. By Theorem 2, our optimization problem over two positive semidefinite matrices is reduced to a much simpler variialle optimization problem. We include a detailed description of how the constants the problem are estimated in practice and what solver we use in full description of the Marvell algorithm in Appendix A.5. Beyond optimization de- tails, it is worth noting how to set the power constraint hyperparameter P in Equation 2 in practice. As directly choosing P requires knowledge of the scale of the gradients in the specific application and the scale could also shrink the optimization converges, instead express P = s A and tune for a fixed hyperparameter > O. This alleviates the need to know the scale of the gradients in advance, and the resulting value of P can also dynamically change throughout training as the distance between the two gradient distributions" mean |ggوي changes.
EXPERIMENTS
In this section, we first describe experiment setup and then demonstrate the label protection quality of Marvell as well as its privacy-utility trade-off relative to baseline approaches.
Empirical Setup. We use three real-world binary classification datasets for evaluation: Criteo and Avazu, two online advertising prediction datasets with millions of examples; and ISIC, a healthcare image dataset for skin cancer prediction. All three datasets exhibit severe label leakage problem without protection. (see Appendix 1. on dataset and preprocessing details). We defer similar results Avazu to Appendix A.7 and focus Criteo and ISIC in this section. For Criteo, we train a Wide& Deep model (Cheng et al., 2016) where the non-label party owns the embedding layers for input features and the first three 128-unit ReLU activated MLP layers (first half of the deep part) while the label party owns the remaining layers of the deep part and the entire wide part of the m For ISIC, we train a model with 6 convolutional layers each with 64 channels followed by a 64-unit ReLU MLP layer, and the cut layer is after the fourth convolutional layer. In this case, an example's 's cut layer feature f(X) and gradient are both in 5x5x61 We treat such tensors as vectors in Ig 1600 to fit into our analysis framework (for additional model architecture and training details see Appendix A.6.2, A.6.3).
IIn this setting, the label party will also process input features (through the wide part) just like the non-label party, further relaxing our formal split learning setup in Section 3.
7

---

Published as a conference paper at ICLR 2022
(a) ISIC cut layer
(b) ISIC cut layer
ISIC first layer
(4) ISC first layer
1.0
3.0
1.0
1.0
0.9
0.7
0.7
0.7
0.6
10000
20000 30000
10020
20005
30000
10000 20000
30000
0
10000
20000
30000
number of updstes
number of agd updaiess
numberal sgi updstes
number optleee
no.robse
Marvell, 5-0.1
Marvell - 5-0.25
Manvell 2-10
Monelll 2-40
Figure 3: Norm and cosine leak AUC (computed every batch) at the cut layer and the first layer under no protection vs. Marvell 1 with different scale hyperparameter s throughout the ISIC training.
5.1 LABEL LEAKAGE AND MARVELL'S 's STRONG AND FLEXIBLE PROTECTION
We first evaluate the protection quality of Marvell against the norm and cosine attacks dis- cussed in Section 3.3. We also compare against the leakage metrics when no protection is applied (no_noise). As the results across the three datasets are highly similar, we use ISIC an example (other datasets see Appendix 1.7.I). We see in Figure (a) that unlike no_no i se where the label information is completely leaked (leak AUC R 1) throughout training, Marve 11 achieves a flex- ible degree of protection (by varying s) against both the norm ((a) and direction attacks 2(b) on the cut layer gradients and has strong protection (leak AUC x 0.5) at 4.0. Additionally, is natural to ask whether the gradients of layers before the cut layer (on the non-label party side) can also leak the labels the non-labe party keeps back propagating towards the first layer. In Fig- ure (d)) we compute the leak AUC values when using the non-label party's 's first layer activation gradient as inputs the scoring functions predict y. Without protection, the first layer gradient still leaks the label very consistently. In constrast, Marvell 1 still achieves strong privacy protection at the first layer (s 4.0) despite the protection being analyzed at the cut layer.
5.2 PRIVACY- UTILITY TRADE-OFF COMPARISON
After showing Ma sarllll can provide strong privacy protection against our identified attacks, we now see how well it can preserve utility by comparing its privacy-utility tradcoff against other pro- tection baselines: no_noise, isotropic Gaussian (iso), and our proposed heuristic max_norm, rm. Similar to how we allow Marvell to use a power constraint to depend on the current iteration's 's gradient distribution through P s|| Algو we also allow iso to have such type of dependence- specifically, we add ~ No (t/d)- Taxa) to every gradient in a batch with a tunable privacy hyperparameter to be fixed throughout training. To trace out the complete tradeoff curve for Ma Marvell and iso, we conduct more than 20 training runs for each protection method with different value of privacy hyperparameter (s for Marvell, t for iso) in each run on every dataset. (Note that no.noisee and max.norm do not have privacy hyperparameters.)
We present the tradcoffs between privacy (measured through norm and cosine leak AUC at cut layer/first layer) and utility (measured using test loss and test AUC) in Figure 4. To summarize the leak AUC over a given training run, we pick the 95% quantile over the batch-computed leak AUCs throughout all training iterations. This quantile is chosen instead of the mean because we want to measure the most-leaked iteration's 's privacy leakage (highest leak AUC across iterations) to ensure the labels are not leaked any points during training. 95% quantile is chosen instead of the max (100%) as we want this privacy leak estimate to be robust against randomness of the training process.
Privacy-Utllite Tradeoff comparison results. In measuring the privacy-utility tradeoff. we aim to find a method that consistently achieves a lower leak AUC (better privacy) for the same utility value.
[arvelll vs iso] As shown in Figure 4, Marvell almost always achieves a better trade- off than iso against both of our proposed attacks at both the cut layer and the first layer on both the ISIC and Criteo datasets.
important to note that although the utility constraint is in terms of training loss optimization, Marvell's better tradeoff still translates to the gener- alization performance when the utility is measured through test loss or test AUC. Additionally, despite achieving reasonable (though still worse than Marvell) privacy-utilie tradeoff against the norm-base based attack, isc performs much worse against the direction-basee even after applying significant amount of isotropic noise (with > 20), iso's 's cosine leak AUC is still higher than 0.9 at the cut layer (Figure 4(b.f). In contrast, Marvell 11 is effective against this direction-bann based attack with a much lower cosine leak AUC 0.6.
attack: on ISIC,
8

---

Published as a conference paper at ICLR 2022
narm Jesk 1ESK-ACC
out layer
taak AUC
(b) ISI layer
95% quantie nom leak AUC 95% quantil norm leak AUC
10 Gitteo layer
teak AUC 93% euartiie bak AUC
(d) Gr cut layer
1.00
1.00
Marvel
0.90
Mareel
Harvel
may somm
3.90
max, rorm
9.90
mak rorm
1o noiee
0.00
ro no
ro no
95% quanttee nomm lesk AlC 99% quartin quantig
3.80
teak AUC 93% quantie
0.90
0.70
0,80
geag
0.70
0.70
0.70
8.00
0.650
1.
2-10
4.48
F-42
4.0
550
351
3.3
2.5
0.7
210
233
47a
(9) SIC
(h) Crico cuyee
1.00
Harvel
bo
1.00
0.90
Marvell
1.00
bo
Harvel
8.90
mac pamm
Toise
0.90
0.80
max pa
po
0.90
ro no
max, pomm
0,80
8.80
95% quantie
0.00
0.70
Quantie
Marvel
0.70
50
0.60
0.1
1.44
C22
325
CN
lost
best auc
lost
tost auc
() ISIC first Iayer
95% quantti la cosing twak AMC
ISIC first layer
(k) Criteo first layer
quantle cedieteen 30C
Critea first layer
1..0
2.00
1.00
Narvel
0.90
larvel
Marcel
0.90
no ncbe
mac nor
0.90
0.80
no. roloe
norm
0.90
mar nor
rolac
0.80
0.80
0.80
95% quantle
Marvel
0.70
mac mmm
8.60
95%
no. m o
0.60
test BUC
test auc
test auc
auc
Figure 4: Privacy (norm & cosine leak AUC) vs Utility (test loss & test AUC) trade-off of protection methods
(Marvel1, 1, iso, noise se, max norm) at the cut and first layer on ISIC and Criten.
[max x norm heuristic] Beyond Ma Marvell, 11, we see that our heuristic approach max norm can match and sometimes achieve even lower (Figure 4(a.f.i) leak AUC value than Ma rveel at the same utility level. We believe this specifically results from our norm and direction consideration when designing this heuristic. However, without tunable hyperparameter, max_ max_norm cannot tradeoff between privacy and utility, Additionally, unlike Marvell which is designed to protect against the entire class of adversarial scoring functions, max_norm might still fail to protect against other future attack methods beyond those considered here.
In summary, our principled method Marvell significantly outperforms the isotropic Gaussian baseline, and our proposed max norm heuristic can also work particularly well against the norm- and direction-based attacks which we identified in Section 3.3.
CONCLUSION
In this paper, we formulate a label leakage threat model in the two-party split learning binary clas- sification problem through a novel privacy loss quantification metric (leak AUC). Within this threat model, we provide two simple yet effective attack methods that can accurately uncover the private labels of the label party. To counter such attacks, we propose a heuristic random perturbation method max_no rm as well theoretically principled method Marvell which searches for the optimal noise distributions to protect against the worst-case adversaries in the threat model. We have con- ducted extensive experiments to demonstrate the effectiveness of Marvell and max_nori rm over the isotropic Gaussian perturbation baseline iso.
Open questions and future work. Our work is the first we are aware of to identify, rigorously quantify, and protect against the threat of label leakage in split-learning, and opens up number of worthy directions of future study. In particular, as the model parameters are updated every batch in our problem setup, the true gradient of an example and the gradient distribution would both change. An interesting question is whether the adversarial non-label party can remember the stale gradient of the same example from past updates (possibly separated by hundreds of updates steps) in order to recover the label information in the current iteration in a more complex threat model. It would also be interesting to build on results to study whether there exist attack methods when the classification problem is multiclass instead of binary, and when the split learning scenario involves more than two parties with possibly more complicated training communication protocols (e.g., Vepakomma al., 2018).

---

Published as a conference paper at ICLR 2022
Ethics Statement. In our paper, have identified realistic threat of label leakage in the two- party split learning binary classification problem. We aim to raise awareness about potential privacy issues in this problem domain, where many industrial applications have been deployed. Beyond making such threats clear, we have taken the first steps towards protection-we have proposed both heuristic and principled methods that can preserve label privacy. We hope our work will pave the way for future analyses that make the two-party split learning framework more effective and secure.
Reproducibility Statement. make our paper reproducible, we provide:
Proofs of our Theorem 1 and Theorem 2 in Appendix A.3,A.4;
Detailed experiment description including 1) data preprocessing, 2) model architecture, 3) training algorithm and hyperparameters in Appendix A.6.
Source code with running instructions (in README. md) at
htwp:hhwww.ooooosorraaa/aaaa/eee prottetto.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Iyy Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Securit, pp. 308-318, 2016.
Avazu. Avazu click-through rate prediction, 2015. URL https:/ / /ww. kaggla.ggg.............k........ com/ c/
avazu- ptr.pptiiiiiioonn//eoooom daaa..
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Kamn Seth. Practical secure aggregation for privacy- preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175-1191, 2017.
Kamalika Chaudhuri and Daniel Hsu. Sample complexity bounds for differentially private learn- ing. In Proceedings of the 24th Annual Conference on Learning Theory, pp. 155-186. JMLR Workshop and Conference Proceedings, 2011.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recom- mender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10, 2016.
Albert Cheu, Adam Smith, Jonathan Ullman, David Zeber, and Maxim Zhilyaev. Distributed differ- ential privacy via shuffling. In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pp. 375-403, Springer, 2019.
Criteo. Criteo display advertising challenge, 2014. URL https://www/ wwggka.ggg......... com c/ criteo-display-eeee chal challllegeeelleeeegeeeeeeeeeeeee
Cynthia Dwork. Differential privacy. In Michele Bugliesi, Bart Preneel, Vladimiro Sassone, and Ingo Wegener (eds.), Automata, Languages and Programming, pp. 1-12, Berlin, Heidelherg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-35908-1
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 93344:2111440 2014.
Ülfar Erlingsson, Vitaly Feldman, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Abhradeep Thakurta. Amplification by shuffling: From local to central differential privacy via anonymity. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algo- rithms, pp. 2468-2479. SIAM, 2019.
Badih Ghazi, Noah Golowich, Ravi Kumar, Pasin Manurangsi, and Chiyuan Zhang. On deep learn- ing with label differential privacy. arXiv preprint 102.v:210..06062 2021.
Otkrist Gupta and Ramesh Raskar, Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications, 116:1-8, 2018.
10

---

Published as a conference paper at ICLR 2022
ISIC.
Siim-isic melanoma classification, 2020. URL https://ww.kkggle..coo
data.
Vladimir Kolesnikov, Ranjit Kumaresan, Mike Rosulek, and Ni Trieu. Efficient batched oblivi- ous prf with applications private intersection. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS '16, pp. 818-829, 2016.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-fficieni learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282. PMLR, 2017.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. International Conference on Learning Representations, 2018. URL htttps://cpenrevieee net/ forum? 1ididdddBBB000000
Benny Pinkas, Thomas Schneider, and Michael Zohner. Scalable private set intersection based on of extension. ACM Transactions on Privacy and Security (TOPS), 21(2):1-35, 2018.
Pramod Subramanyan, Rohit Sinha, Ilia Lebedev, Srinivas Devadas, and Sanjit A Seshia. A formal foundation for secure remote execution enclaves. In Proceedings of the 2017 ACM SIGSAC Conference Computer and Communications Security, pp. 2435-24500 2017.
Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv: 18/2.00564 2018.
Praneeth Vepakomma, Otkrist Gupta, Abhimanyu Dubey, and Ramesh Raskar. Reducing leakage in distributed deep learning for sensitive health data. arXiv preprint
2019.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.0266 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems, pp. 14774-14784 2019.

---

Published as a conference paper at ICLR 2022
A
APPENDIX
Appendix Outline
A.1 Expressing AUC(r) integral
A.2 Toy Class-balanced Example of Positive Example Prediction Lacking Confidence
A.3 Proof of Theorem
A.4 Proof and Interpretation of Theorem 2
A.5 Marvell Algorithm Description
A.5.1 Marvell Time Complexity
A.5.2 Marvell Empirical Run Time
A.6 Data Setup and Experimental Details
A.6.1 Dataset Preprocessing
A.6.2 Model Architecture Details
A.6.3 Model Training Details
A.7 Complete Experimental Results
A.7.1 AUC Progression for Avazu and Criteo
A.7.2 Complete Privacy-Utility Tradeoff's
A.7.3 Impact of Activation Gradients' Layer Location on the Corresponding Leak AUC
A.8 Reasons why don't use local DP in our setup
12

---

Published as a conference paper at ICLR 2022
A.1 EXPRESSING AUC(r) AS AN INTEGRAL
Recall that the ROC curve of scoring function RR RR parametric curve cp. IR - [0, 10,
such that (t) = (t) TPR, (t)), with
FPR, + [0, 1], such that FPR, (t) p(o Rd r(g) (1 >1}
TPR, : + [0. 1), such that TPR, (t) p(1) [ R :)(g) > x}))
We notice that FPR, and TPR, are both monotonically decreasing functions with lim (t), = 1, lim (t), = t-500
lim (1),
lim ( 0.
However, FPR, and TPR, does not need to be differentiable everywhere with respect to f. Thus we cannot simply express the area under the curve using / (1) (P)( On the other hand, because both FPR, and TPR, are functions of bounded variations, we can express the area under the parametric curve e, through the Ceamann--tielljee. integral which involves integrating the function TPR, with respect to the function FPR,:
AUC(r) Here the integration boundary from oo to -00 in order ensure the integral evaluates to a positive value.
TPR, (t) dFPR, (t)
A.2 TOY CLASS-BALANCED EXAMPLE OF POSITIVE EXAMPLE PREDICTION LACKING CONFIDENCE
As mentioned in Observation of norm-based attack in Section 3.3, for many practical applications, there is inherently more ambiguity for the positive class than the negative class. To make this more concrete, let's consider simple toy example.
Suppose the binary classification problem is over the real line. Here, 50% of the data is positive and lies uniformly in the interval [0, 1J. On the other hand, the remaining negative data (the other 50% of the total data) is a mixture distribution: 10% of the negative data also lies uniformly in [0, 1]. while the rest 90% of the negative data lies uniformly in [1,2].
p y 1) 1(x E [o, 1)),
ply 1) 0.5.
p( 0) = 0.1 (xx E 1J) + 0.9 1(x E [1,
p(y 0) 0.5.
This setup mirrors our online advertising example where for all the users interested the product (with feature j0, 1)), only part of them would actually make the purchase after clicking its advertisement.
In this case, the best possible probabilistic classifiers CP that can be ever learned, by Bayes Rule, 10 would predict any example in [0, 1] to be of the positive class with probability TI while would predict any example in [1, to be of the negative class with probability 1:
10 Ti
if x E [0
C (y 11=1
if [1,2]
Thus even for this best possible classifier Copt every positive example would have confidence gap of IT while 90% of the negative examples (the ones in [1, 2)) would have confidence gap of O. Hence we see that in this toy example, our empirical observation the lack of prediction confidence for positive examples would still hold true.
Besides, it is important to notice that this lack of positive prediction confidence phenomenon hap- pens even in this bllanebl toy example (p(y 1) p(y 0) 0.5). Thus, Observation does not require the data distribution to be class-imbalanced to hold, which further demonstrates the generality of nore-based attack.
13

---

Published as a conference paper at ICLR 2022
A.3 PROOF OF THEOREM
(Theorem 1). For O I e < and any perturbed gradient distributions po)) and pr) that are
absolutely continuous with respect to each other,
1 (L(((1) || ((o) (L( || p(y S implies max AUC(r) 1 + 8
Ve
obtain up- per bound total variation distance by the symmetrized KI. divergence (sumKL) for pair of distributions (P, Q) that are absolutely continuous with respect to each other:
Proof of Theorem 1. Combining Pinsker's ' inequality with Jensen's 's inequality,
TV(P, Q) & 2 ([VLL(P Q1/2+ VKL |I P)/2) - (KL(P +)+++++++++ KL P).
(4)
2
By our assumption, this implies that TV(P(U), p(O)) < Tf1 By the equivalent definition of total variation distance TV(P p())) = maXACR" ((2) (A) - (A)), we know that for any C
Rd. (A) p I d
2 For any scoring function and any threshold value t, let A
L/c (g: rg)) > 1), then we have (t) (1), = P() (A) p(o) 2
Therefore, the AUC of the scoring function can be upper bounded in the following way:
AUC(r) =
(t),(t) (FPR,(t)
(5)
min (F), + VC
1) dFPR,
(6)
2
where in (6) we use the additional fact that TPR, I 1 for all t E R.
When [0, 4), we have 1 1- ve (0, As FPR, (8) a monotonically nonincreas-
ing function in with range in [0, 1J. the set {t: (L), S - x
+ Φ is not empty. Let
== inf t: (t) & 1 2 Again by FPR, (t) being monotonically nonincreasing function in I, break the integration in Equation (6) into terms:
AUC(r)
Ve min (PR, + 2
1) dFFRR.(1) + ( min (P), + VE 2
1) (FFRR
(7)
(P),(1) + VE
dFPRR (1) + S. d(FR.(()
<
(8)
2
(t]]
Ve + (1)
+ +FPR,
(9)
2
2
1=000
(1
VEY2
VE
1 - Ve
- (1
Ve
<
2
+
(10)
2
2
2
VE
(11)
2
Since this inequality is true for any scoring function it is true for the maximum value. Thus the proof is complete.
14

---

Published as a conference paper at ICLR 2022
A.4 PROOF AND INTERPRETATION OF THEOREM
(Theorem 2). The optimal Σx and Σg to the following problem
min KL(N(g()) vl + E1) N ul +oo)) + KL(N(T ul Eo) N vz . 21) ES
subject to
Σ9h1 = En Eo:
p-tr(2))+(11p)) (1- -tr((o $ P.
E ビ 0
En ビ 0.
have the form:
2 2/100 (1)+
A (0) 2
+ +10 (A))(Ag)T
2
A (Ag)(Ag)) + I Ia, (12)
where (A(0) 2/2 10)x (1)x , (1)))) is the solution to the following 4-variable optimization problem:
(d - 1) X/2
+ 1
+ (d 1) 11 +
1 + +u* + 1|
12
x1 + + A
min A0
+
1
\/2 +
1/2 +
1 +
1 +u
S.L. p\d (1) + p(d 1)11 +(1 -p)1 (0)
(1 p) 1)1 $ P.
x S 0, -1 (0) < 0, -1 < 0, -1 < 0,
( 1 S 0, (0) - 1 0 0
Proof of Theorem 2. By writing out the analytical close-form of the KL divergence between two Gaussian distributions, the optimization can be written as:
min tr((En + -l) (Eo + ul)) + tr((Eu + uI) - (Σ1 + xI))+ E18
(g - T (( + +u) + (Eo + u/) -1 (g - (0))
subject to
Σn = Eo,
p. p.tr(E)) + (1-p) (r(En)) < P,
(13)
Σ > 0,
Eg ビ 0.
By the commutative constraint on the two positive semidefinite matrices Σ₁ and Eo, we know that we can factor these two matrices using the same set of eigenvectors. We thus write:
En = QQ diag (0)
d
)2
Q
Σ Q diag (1)
(2
Q
(14)
where Q R is an orthogonal matrix and the eigenvalues x xx()) are nonnegative and de- creasing in value.
Using this alternative expression of Σn and Eg, can express the optimization in terms of (2) 7, 1.
(A(")
d + u
min (110 1QQ
(Q(G()) - g(>}} diag
1
+
((1)) ) Q(0()) (0)
1
15

---

Published as a conference paper at ICLR 2022
d
subject to pl
X + (1 - p)(
00) Σ
- P
==1
1=1
1 < 0, Wi W
1 2 0, Wi E (d.
1 2 1 , <j
\ > 1 Vi <j <i
Q orthogonal.
For any fixed feasible (A,0) ). { row to be the unit vector in the direction of Ag. Thus by first minimizing Q, the optimization objective reduces to:
(0) J. we see that the corresponding minimizing Q will set its first
+ u
+
g
g
+
+
+
1=1
+
\
+ u
X +
(0)
+u
(1)
+
+o+
Aggl
(0)
+ + | 2
1 (0)
1 +
+
+u
+u
is strietly convex over the line segment pA,"" (1) + (1 p) (0) for any nonnegative and attains the the when u 2 v. Suppose without loss of then for the optimal solution must have X,
We see that for the pair of variable (2 , (0)) (i 2 2), the function
0 10 + x
x
minimum value at x 0 when u < v and (0)
for all i 2 2. Under this condition, we notice that the function m(xx) z+e + z+2 is strictly convex on the positive reals.
generality
Thus for all {x that satisfies Ei x(1) c for fixed nonnegative c, by Jensen inequality, we have
1000 +
1
+
d
+
u
d
m
m(((1))
d
1
(1)
d
1=2
From this, we see that the optimal solution's variables } must take on the same value (++) for all i 2 2. The case when u < v is similar. As result, we have proved that at the optimal solution, must have:
1 = 10
(0) and ( 21", for i,j 2 2.
16

---

Published as a conference paper at ICLR 2022
{ 2
U
1
Hence, the optimization problem over the 2d variables an optimization problem over the four variables { 11 (0)
siz can be reduced to
==1
1 +
( + u + A
1 +
(d - 1) u
min x A A
+ (d -
+
A
12 +
X/2 +
x +
X (0)
+
(15)
subject to p\1" + p(d 1)11 + p}>1 (0)
+ (1 p)(d 1)(222)) S P
1111 0
21 0 0
22 0 0
1/ S 0
(1) 1 < 0
12
(0) ( < 0.
Given the optimal solution to the above 4-variable problem (A A/ 10)) 22(0)= >(00),, we can set Q to be any orthogonal matrix whose first row is the vector TAall2" Plugging this back into the expression of Σ, and Eo in Equation (14) gives us the final result.
Ag
Thus the proof is complete.
Remark (Interpreting the optimal 2j and Mg). From the form of the optimal solution in (12), we that the optimal covariance matrices are both linear combinations of two terms: rank one matrix (Ag) (Ag)T and the identity matrix Ig. Because a zero-mean Gaussian random vector with convariance matrix (A + B) can be constructed the sum of two independent zero-mean Gaussian random vectors with covariance matrices A and B respectively, we see that the optimal additive noise random variables ry( and (7 each consist of two independent components: one random component lies along the line that connects the positive and negative gradient mean vectors (whose covariance matrix is proportional to AgAg );; the other component is sampled from an isotropic Gaussian. The use of the first random directional component and the fact that the isotropic Gaussian component have different variance scaling for the positive and negative class clearly distinguishes Marvell from the isotropic Gaussian baseline iso.
Remark (How solve). By analyzing the KKT condition of this four variable problem we can find that the optimal solution must exactly lie on the hyperplane pA(") + p(d - 1) (1) + (1 p)1(0) +
if < v and > = if u 2 Thus the problem is further reduced to 3-variable problem. If we consider keeping one the remaining variables' values fixed, then the feasible region becomes restricted to line segment. We can simply perform a line search optimization of the convex objective and find the optimal values for the remaining two free variables. We can then alternate over which one of the three variables to fix and optimize over the other two. This optimization procedure would eventually converge and give us the optimal solution. This approach mimics the Sequential Minimal Optimization technique used solve the dual of the support vector machine (SVM).
1)-p)(d 1) P. From the proof above we additionally know that 111
17

---

Published as a conference paper at ICLR 2022
A.5 MARVELL ALGORITHM DESCRIPTION
We use E IR" to denote the vector with 1 in each coordinate. We use زiuو to denote the j-th row of the matrix y and y[5] denote the j-th coordinate of the vector y.
Algorithm 1: Marveell algorithm
input g E IR Bxd
size-B batch of unperturbed gradients
V E (0, 13
the label for each example in the batch
privacy hyperparameter for the power constraint P output: J E RR Bxd batch of perturbed gradients
8.
// Step 1: estimate the aptimization constants from the batch gradients using maximum likel food hood estimation (MLE)
1 1"v
/x positive fraction */
2 (()) + Po g y:
/+ positive mean */
g()) - B-1" g" (1 y):
/* negative mean */
Ag g( -
sut + y(' y) ا==1 v |1bb]] /00 :
/* positive convariance +/
== (1 vbi) |166]] (0) 2
u 4 31B-11
/* negative convariance +/
P |Ag
power constraint hyperparameter */ // Step 2: optimize the four-vart able problem
then
2 0:
/* this variable is optimal at 0 x/
Randomly initialize the optimization variables ( ( 10 in the feasible region in (15).
10
else
12
/+ this variable is optimal at 0 */ Randomly initialize the optimization variables ( 1()) ( in the feasible region in (15).
- 0:
13
end
15
while not converged do
16
Fix one of the newly updated optimization variables:
// The 4 4-variable optimal solution lies on hyperplans in R (see
Appendix A.4 Remark) so fixing two variables gives us a line-segment
17
Update the remaining two optimization variables by performing 1-d line-search minimization of the convex function (15) while satisfying the constraints;
18
end
// Step
compute the optimal covariance matrices
19 Σj ( x (Ag)(Ag)T
T + 1 La:
AA
(9)
Σ.
(Ag) + 1 La:
// Step 4:
perturb the gradients
δ 4- 0xxd
/x an empty matrix to store the perturbed gradients */
22
for j < 1 to B do
23
['y[j then
9 [[] + 17 (1), where ()) A/(0, Ej);
24
25
else // vbil =0
øbi + abil + (0), where ()) ~ N'0 Σ;);
26
27
end
end
18

---

Published as a conference paper at ICLR 2022
A.5.1 MARVELL TIME COMPLEXITY
The Marvell algorithm in 1 consists of the following steps:
Step 1.
Compute the positive and negative gradient mean and their difference (line 1 This amounts averaging over most B numbers each of the d gradient dimensions. Thus would have time complexity of (BB)).
Step 2. Compute the positive and negative covariance constants and (line 5, 6). This operation also takes O (Bd) as it averages over squares of coordinate differences.
Step 3. Use the results from step 1 and solve the 4-variable optimization problem (line 7-18)- Here because the constant size of the number of optimization variables. Solving this prob- lem up to a fixed precision takes constant time O(1).
Step 4. Perform the actual random perturbation (line 19 28). Because of the additive structure each class's covariance matrix, for every example's ' gradient in the batch, we can in- dependently sample one random Gaussian vector with rank-1 covariance and also another spherical Gaussian random vector and add both vectors to this gradient for perturbation. This step would take ((d) for each example and thus takes ((Bd) in total.
As a result, the entire Marvell algorithm has a time complexity of O(Bd). This can be further sped up through parallel computation using multi-treaddin/mmllioooom through the cut layer would also require ((Bd) time complexity, the Marvell algorithm would not slow down the split-learing process any significant way all.
Considering backpropagation
A.5.2 MARVELL EMPIRICAL RUN TIME
Empirically, in Table 1, present the average amount of time it takes Marvell with the privacy hyperparameter value 8 = for the three models (one for each dataset) we considered in experiments (s value is chosen
achieves good privacy protection shown with purple line in Figure 3, 5, 6). We also compare it to the average time it takes to run one update iteration. We additionally include the 95% confidence interval for the mean estimator of the run time. Here notice that the total run time of Marvell only takes up very small amount of the total training time for each method, further corroborating our algorithm's 's time efficiency advantage.
batch size
cut layer feature dimension
average
average update time (seconds iteration)
Ma rve 11 run time (scconds run)
B
1600
1.79 10-2
3.94 x 10-1
ISIC
128
= 5 x 5 x 64
+ 7.03 x 10-5
+ 4.05 x 10-3
1.68 10-2
4.84
Criteo
1024
128
+ 8.67 x 10-5
+ 3.28 x 10-1
1.75 x 10-2
6.22
Avazu
32768
128
1.47 x 10
3.71 x 10
Table 1: Average run time of Marvell (s and average total update time per iteration for models trained on ISIC, Criteo, and Avazu.
19

---

Published as a conference paper at ICLR 2022
A.6 DATA SETUP AND EXPERIMENTAL DETAILS
We first describe how we preprocess each of the datasets in A.6.1. We then describe the model architecture used for each dataset in A.6.2. Finally, we describe what the training hyperparameters are used for each dataset/model combination and the amount of compute used for the experiments in A.6.3.
A.6.1 DATASET PREPROCESSING
[Criteo] Every record of Criteo has 27 categorical input features and 14 real-valued input features. We first replace all the NA values in categorical features with a single new category (which we represent using the empty string) and replace all the NA values in real-valued features with O. For each categorical feature, we convert each of its possible value uniquely to an integer between O (inclusive) and the total number of unique categories (exclusive). For each real-valued feature, we linearly normalize it into [0, 1]. We then randomly sample 10% of the entire Criteo publicly provided training set as our entire dataset (for faster training to generate privacy-utility trade-off comparision) and further make the subsampled dataset into 90%- 10% train-test split.
[Avazu] Unlike Criteo, each record in Avazu only has categorical input features. We similarly replace all NA value with single new category (the empty string), and for each categorical feature, we convert each of its possible value uniquely to an integer between 0 (inclusive) and the total number of unique categories (exclusive). We use all the records in provided in Avazu and randomly split it into 90% for training and 10% for test.
[[SIC] The official SIIM-ISIC Melanoma Classification dataset has a total 33126 of skin lesion images with less than 2% positive examples. Because for image classification model training it is desirable to use a batch size of ~ 10², it is highly likely that there won't be any positive examples sampled in batch of such size. Thus to make the label leakage problem more severe, we modify the dataset by retaining all the 584 positive examples and randomly choosing 584 x 9 examples out of all the negative examples. By doing this, we enforce that there are 10% positive examples in this modified dataset. We randomly split these 5840 examples into 80%-20% training and test split. We also resize the images to size 84 x 84 for efficient model training.
A.6.2 MODEL ARCHITECTURE DETAILS
[Criteo, Avazu] We use popular deep learning model architecture WDL (Cheng et al.. 2016) for online advertising. Here the deep and wide part each first processes the categorical features in a given record by applying an embedding lookup for every categorical feature's 's value. We use an embedding dimension of 4 for the deep part and embedding dimension of 1 for the wide part. After the lookup, the deep/wide embeddings are then concatenated with the continuous features to form the raw input vectors for both the deep part and wide part respectively. (This step is skipped for Avazu as it has no continuous features.) Then the wide part computes the wide part logit value through real-valued linear function (with bias) of its raw input features using 6 U-activacd 128-unit MI.P layers before producing a single deep part logit. The two logits are summed up to form the final logic value. The cut layer is after the output of the 3rd ReLU layer on the deep part.
input vectors, while the deep part processes its
[[SIC] Every input image after resizing is of size 84 x.. We use a convolutional model with convolutional layers each with 64 channels 3 x 3 filter size with 1 x stride size. Each convolutional layer is followed by a ReLU activation function whose output is then max pooled with 2 x window and stride size 2 x 2. The max- max-pooled output of the 6th layer is then flattened and pass into 64-unit ReL.U-activated MLP layer before finally being linearly transformed into a single logit score. The cut layer is after the output of the 4th max pool layer. Thus the cut layer feature and gradient both of shape x 5 x 64.
A.6.3 MODEL TRAINING DETAILS
Because the protection mechanism requires adding noise to the cut layer gradient, the induced vari- ance of the gradients of non-label party's [paaraeterrs becomes larger. Thus to ensure smooth
20

---

Published as a conference paper at ICLR 2022
optimization and sufficient training loss minimization, we use slightly smaller learning rate than what normally used.
[Criteo] We the Adam optimizer with batch size of 1024 and learning rate of le-4 through- out the entire training of 5 epochs (approximately 20k stochastic gradient updates).
[[SIC] We use the Adam optimizer with batch size of 128 and a learning rate 1e-5 throughout the entire training of 1000 epochs (approximately 35k stochastic gradient updates).
[Avazu] We use the Adam optimizer with a batch size of 32768 and a learning rate le-4 throughout the entire training of epochs (approximately 5.5k stochastic gradient updates).
We conduct our experiments over 16 Nvidia 1080Ti GPU card. Each run of Avazu takes about 1 hours to finish 37 hours to finish on single GPU card using GB of GPU RAM. Each run of ISIC takes about 12 hours finish single GPU card occupying 4GB of GPU RAM.
single GPU card occupying 8GB of GPU RAM. Each of Criteo takes about
21

---

Published as a conference paper at ICLR 2022
A.7 COMPLETE EXPERIMENTAL RESULTS
A.7.1 LEAK AUC PROGRESSION FOR AVAZU AND CRITEO
In addition to the leak AUC progression on ISIC shown in Figure 3 in the main paper, we also show the leak AUC progression on the Avazu and Criteo datasets throughout training here in Figure 5 and Figure 6. We similarly compare Marvell with different levels of protection strength (s values) against the no protection baseline no no_noi se. As we can see, Marv Marvell still achieves strong and flexible privacy protection these two datasets against our label attacks at different model layers.
layer
Avazu layer
layer
Avazu first
0.9
003 Took
0.8
0.8
0.5
0.5
2000
4000
2000
4000
2000
4000
2,000
4000
number of apd updalls
number uptates
numberal updates
number sgd updanss
no moise
Manvoli 5-3.1
Marvell. 5-0.25
Manvell, 2=1.0
Morvlll 5-40
Figure Norm and cosine lcak AUC (computed every batch) the cut layer and at the first layer of
no_noi se (no protection) vs. Marvell with different scale hyperparameter a throughout the Avazu training.
(a) Criteo cut cuylayer
[b] Criteo cut layer
[c) Criteo first layer
[d] Criteo first layer
1.4
1.0
1.0
1.0
0.8
0.8
0.6
0.6
5000
10000
15000
20000
5000
10000
15000
20000
5000 10000 15000 20000
5000 10000 15000 20000 number of uge updates
number of agd lppdttt
numberoo agd upiiates
number of uptatees
molse
Marvel, a-0.1
Marvel, am0.25
Marvel 1=1
Marvel, xm6,5
Figure 6: Norm and cosine leak AUC (computed every batch) at the cut layer and at the first layer of oo_e (no protection) vs. Marvell with different scale hyperparameter throughout the Criteo training.
A.7.2 COMPLETE PRIVACY-UTILITY TRADEOFFS
We show additional Privacy-Utility tradeoff results for all three datasets considered in this paper. (Some of the plots have already been shown in the main paper but we still include them here for completeness and ease of reference.) For each dataset, we compare the privacy-utiiity tradeoff over multiple measures of privacy and utility:
Privacy
We consider our introduced privacy metrics using the activation gradient from the cut layer and the first layer of the non-label party:
95% norm leak AUC at cut layer
95% cosine leak AUC at cut layer
95% norm leak AUC at first layer
95% cosine leak AUC at first layer
Utility
We consider three metrics of utility:
training loss (train loss): the lowest loss achieved on the training set throughout training- This directly measures how much the random protection perturbation influences the opti- mization.
test loss. Because only control the training optimization stochastic gradient's variance, measuring test loss directly tells us how much impact the training optimization random per- turbation influences beyond optimization but on the learned model's generalization ability. test AUC. As we are dealing with binary classification problem (where performance is commonly measured through test AUC), we also naturally consider it utility metric.
22

---

Published as a conference paper at ICLR 2022
As shown in Figure 7,8,9, Marvell consistently outperforms the isotropic Gaussian baseline over all the different privacy-utility definitions. In addition, our proposed heuristic max_norn is also attacks.
particularly effective against our identified norm and
95 quantie romm lakk AUC
1.00
be Marvel
max maam
95% quartte codne teak
9.50
1.00
layer
quantie rorm 10 auc
20.50
3.00
bc
Movvel
max max
guartte cogine teak
..00
3.90
layer
bis
Hovell
max
0.80
0.80
0.20
0.80
0.70
9.70
Forvel
8.70
2.70
max
0.60
8.50
333
0,45
9:3
941
335
K3
loss
loss
DUNNRi norm leak AUC
1.00
layer
95% cuandie leak AUC
1.00
Avazu out layer
5 ouantil rorm 1e3h AAVC
1,000
(g) Avazu layer
bo
1,00
Avazu first layer
0.90
max, sorm
Marvell
0.90
20.90
Mareel
max, rorm
0.90
mak norm
Marvel
0.80
no noise
0.80
0.80
no no
0.80
no noin
0.70
0.70
0.70
quamtle
0.70
97 30.50
0.60
3:45 E72
333
E55
154
C.2
0.60
3
3.3
133
CS
ES4
0..5
0.60
333
121
332
024
0.3g
95%
0.60
1:4
3:33
E26
121
CS4
giamble rarm eak AVC
layer
AUC
layer
grantle narm eak AVC
first layer
first layer
1.00
1.00
1.00
1.00
0.90
Marvel
mac norm
0.90
max nom
Marvell
0.90
max, norm
Harvel
0.80
0.80
0.80
ro so
8.70
93% g-antle
0.70
0.70
0.70
6 0.50
0.60
3.78
0.60
4733
1778
0.50
0.60
95%
3.660
KINE
Figure 7: Privacy (norm and cosine leak AUC) vs Utility (train loss, test loss, and test AUC) trade-off of protection methods (Marvell, iso, no noise,
at the cut layer and first layer on Avazu.
95% quanttr nome leak MUC
Criteo
quantle teak ALC
(bl Criteo cut layer
Quantee nom lesk AUC
(cl Crites first layer
quandle teak ALC
(d) Criteo first layer
1.00
1.00
Myvvll
Marvel
0.90
Marvel
Marvel mac
max_ torm
0.90
max, roem
max_rorm
0,90
no. poice
poice
0.80
ro yo
poie
0.80
0,.00
2.70
0.70
0.70
0.70
0.60
0.60
0.60
0.30
0.30
8.35
0.30
0.55
CCC 0.35
train liss
train Toss
95% quanbilaeeeee narm leak AUC
Cribeo tnyer
92% quantle cestre teak AUC
01 Criteo cut
Ontec first
92% quantle cetiee leak AUC
(hi Oriee first Inyer
Marvel
50
1.00
Marvel
95% ouantle nom wemmleeaaaaaa
0.90
50
Marvel
1.00
Go
Harvel
max: 10rm
0.90
max rorm
max rorm
0.90
max
moras
no
0.80
no
0.80
0.80
0.70
0.70
0.70
6.70
3.60
0.60
0.60
0.60
0.75
375
test loss
test loss
test
test 1055
95% quartle norm bakk AUC
(1) Criaa cut. layer
92% quartte AlC
St G Ree cut layer
99% quartle nomm balk AUC
(3) Critee first layer
( Cr iieo first layer
1000
Marvdli
1.00
0.90
Marvell
1.00
Po
Manvel
max
0.90
rograsse
rorm
0.80
nax nar
0.90
no roymis
maxnoon
0.90
0.80
8.70
9.70
9.70
0.60
0.50
16-8 mUC
lest auc
test aUc
test auc
Figure 8: Privacy (norm and cosine leak AUC) vs Utility (train loss, test loss, and test AUC) trade-off of protection methods (Marvell, isc, no_no max_norm) at the cut layer and first layer on Critco.
23

---

Published as a conference paper at ICLR 2022
95% qUAMtin norm Jesk AUC
layer
tak AUC
(b) ISC layer
layer
layer
95% quantia norm Jeak
1.00
Marvel
max, somm <o noiee
0.90
0.90
9.80
93% euantie
0.90
0.80
93% quantlle
0.70
3.70
0.70
1.59
144
4.4
848
0.52
0.18
95% guantic rarm lesk-alkc
1.00
byyer
teak AUC
1.00
(n out layer
guantie rorm leslekkkk
1.00
(g) layer
1,00
byer
Marvel
max, tomm
0.90
ro.ross
8.80
90% gantl
0.80
0.80
0.70
0.70
92%
0.70
95%
9.70
3:2
C-6
622
C36
CA2
626
3.0
0.30
340
6/2
95% gannile nem lak AUC
layer
AUC
95% gantle nom leak AUC
JUC
.00
1.00
L.00
1.00
Marvell
Marvell
mac norm
max, nom
0.90
roise
no
0.80
0.80
93%
0.70
0.70
95%
0.70
313
0.3
C.0
0,35
6/6
000
030
0.5
315
test
Figure 9: Privacy (norm and cosine leak AUC) vs Utility (train loss, test loss, and test AUC) trade-off of
protection methods (Marve iso, noinsss,, se, max.nerm) at the cut layer and first layer on ISIC.
24

---

Published as a conference paper at ICLR 2022
A.7.3 IMPACT OF ACTIVATION GRADIENTS' LAYER LOCATION THE CORRESPONDING LEAK AUC
In Section 5.1, we have shown that the activation gradients of not only the cut layer but also the first layer can leak the label if no protection is applied. this section, we analyze the effect of this layer location on the degree of label leakage under our proposed attacks measured through leak AUC. Here, for the convolutional neural network trained on ISIC, we have allocated 4 Conv-Rel.U- MaxPool layers on the non-label party side. Here each of these 4 layers' activation (from ReL.U output) gradients can be used to infer the label. Figure 10, we plot the progression of norm and cosine leak AUC for layer 2 and 3 in addition to the first layer (layer 1) and the cut layer (layer 4) when applying no protection or Marvell with s = 4. We notice that when no protection is applied, the norm and cosine leak AUC are approximately the same across the layers and both at a very high level, indicating a strong degree of label leakage. In contrast, when Marvell is applied, the leak AUCs are much lowered layers. Empirically, we notice that the leak AUCs for given training iteration are close among all the layers with a maximum difference around 0.05. Here gradients would leak the labels more than the cut layer or not - for norm leak AUC, the carlier layers have higher AUC values (more leakage) than the cut layer, while for cosine leak AUC, the earlier layers have lower AUC values (less leakage) than the cut layer. Further understanding the relationship between these leak AUC values across different layers an open question motivated by our observations here.
reasonable level of around 0.6 throughout all the
unclear whether the earlier layers'
ISIC, non-label party's convolution network with four Conv-ReLU-MaxPer layers
no_noise
(b) no_noise
(c) Marvell, 5m4.0
Marvell, 5,4.0
0.64
MM
ncmm leak auve
0.99
norm auc
costre tcak auc
0.98
leak
0.98
0.56
0.62
0.97
0.97
0.54
0.60
0.95
0.95
0.58
30000
30000
number of sgd updates
number of sgd updates
number of sgd updales
number of sgd updates
layer layer)
layer
layer
layer (cut layer)
Figure 10: For non-label party's four-layer convolutional architecture trained on ISIC, we plot the progression of the norm cosine leak AUC computed using the activation gradients from each of the four layers throughout training (a). (b) when random protection is applied and in (c). (d) when using Marvell with privacy hyperparameter s = at cut layer (layer 4). The four curves overlap (b) at the value 1.0. For each layer's data in each figure, a 1-d Gaussian kernel with standard deviation of is convolved with the raw 1-d aray of leak AUC values to smooth out the fluctuations and make the different layers' degree of leakage more visually distinct.
Beyond the four layer convolutional neural network trained on ISIC, we also train new MLP model architecture on Criteo. Here instead of having 3 128-unit ReLU-actiiated MLP layers for the non- label party's 's function as described in Section A.6.2, we use such layers for f while keeping every other architectural component the same. We plot the progression of leak AUC values on each of the eight layers Figure 1 I. Here the observations are very similar to what we see in Figure 10, confirming our aforementioned conclusions in this section.
25

---

Published as a conference paper at ICLR 2022
Criteo, non-label party's MLP network with eight 128-unit ReLU-actiivaedd MLP layers
(a)
noise
(c) Marvell, s=4.0
(d) Marvell, 5=4.0
1.00
1.005
0.60
norm lesk suc
1.000
norm leakeek aut
cosine leak auc
0.65
0.95
tcak
0.55
0.64
0.990
0.50
0.90
0.985
0.45
0.60
5000 10000 15000 20000
5000 10000 15002 20000
10200
20000
10000
20000
sgd
sgd updates
number updates
sgd
layer 1 (first layer)
layer
layer
layer
layer
layer
layer layer)
Figure 11: For a non-label party's 's eight-layer MLP architecture trained on Criteo, we plot the progression of the norm or cosine leak AUC computed using the activation gradients from each of the eight layers throughout training in (a). (b) when random protection is applied and in (c), (d) when using MarvelL with privacy 4 the layer (layer 4). The eight
hyperparameter
overlap (b) at the value 1.0. For each
layer's data each figure, 1-d Gaussian kernel with standard deviation of 100 is convolved with the raw arry of leak AUC values to smooth out the fluctuations and make the different layors' degrec of Icakage more visually distinct.
26

---

Published as a conference paper at ICLR 2022
A.8 REASONS WHY WE DON'T USE LOCAL DP OUR SETUP
is possible to use a local differential privacy definition to analyze our problem setup if we treat any single training example dataset and two examples (dataset) adjacent if they share the same X but have the opposite label y. Then one can analyze the activation gradient computation function's 's local DP properties. However, we choose not use local DP in our setup for the following two reasons:
aware of any attacks associated with local DP that are suitable for practical use in our problem setup. In contrast, our leak AUC privacy quantification metrics have corresponding concrete, realistic attack methods that the non-label party can use practice. 2. Even if there exists practical attack associated with local DP, this attack would require example-level side information as the activation gradient distributions would be example- specific. Distinguishing between such gradient distributions for pair of adjacent examples (with same X but opposite label y/) would require knowledge about the two distributions specific to this example pair itself. As have mentioned in Section 3.2, do not assume the non-label party would realistically have access to such fine-grained knowledge. Instead, in setup, the non-label party uses scoring function that only takes in the communicated cut layer gradient without any additional information.
1. We
27


Reviews:
[{"review_id": "jlmUAba6xjs", "reviewer": "Reviewer_GVaD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper considers a novel setting in split learning: stealing binary labels during training and the setting could not be solved directly by differential privacy. To begin with, the authors first explore the possibility of stealing label information via some heuristic observations of the gradient information of positive/negative examples. The effectiveness of the proposed attack is shown in the experiments. The authors then propose an optimized random perturbation approach to mitigate the attack and the defense is shown to achieve the best trade-offs among the baselines. The paper is well-written and clearly organized. Overall, I think the paper makes solid contributions. \n\nMy biggest concern is that the paper is limited to (class-unbalanced) binary classification. I carefully check the proposed attacks, but find it difficult to extend the multi-class classification. For example, observations 1-4 are not necessarily extended to multi-class classification. \n\nBesides, since the proposed defense MARVELL at each update step requires solving an optimization problem, the (time) efficiency of MARVELL should also be discussed and demonstrated in the experiments. I am also curious about how the locations of the cut layer could affect the performance of the proposed attacks and defense (not limited to the first layer). \n", "summary_of_the_paper": "The authors in the paper consider stealing the private label information from the party that does not know the label of training data during split training for binary classification. Specifically, the attack methods are based on differences of gradient information between positive and negative examples and the defense method MARVELL is based on optimal random perturbation solved by an optimization problem. Experimental results suggest that the proposed method could effectively defend against the label stealing attacks.", "summary_of_the_review": "1. The paper makes solid contributions in privacy-preserving machine learning.\n2. Time efficiency of the proposed methods should be discussed and more ablation study can be done.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7j-4UaRiZZw", "reviewer": "Reviewer_ercK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n* The paper studies an interesting problem. \n\n* The paper is well written and organized. \n\nWeaknesses: \n\n* The setup for attack models (e.g., observations in attacks 1 and 2 in Section 3.3) seem very restrictive, and may only work for a specific set of binary classification problems with highly imbalanced classes, such as the disease prediction example given in the paper. What if the labels are more homogeneous, what would be the impact on observation 1? \n\n* The privacy objective is defined as a function of KL divergence with respect to any underlying probability distribution on the perturbed gradients. This appears as a rather difficult objective to achieve in general. In fact, the paper then assumes a Gaussian distribution as the underlying distribution (and the perturbations) for each of the classes while evaluating the objective function. This assumption is not clear. The first thought that comes to mind is the central limit theorem, but then, as mentioned in the paper, one class may have very few samples (e.g., the disease prediction example). Moreover, it appears that the empirical mean of each class is treated as the true mean. However, the validity of this assumption would also depend on the number of samples collected for each class, which could be highly imbalanced between the two classes according to the motivated problem setup. This, combined with the fact that the max_norm heuristic seems to perform as well as Marvell, seems to require a stronger motivation for Marvell. \n \n\n\n\n\n", "summary_of_the_paper": "This paper formulates a threat model on two-party split learning (parties have different features, with one party holding the labels) for binary classification, and provides insights about how simple functions on the gradients can be used to extract confidential label information. The authors then proceed with defenses to these attacks, based on random perturbations of the communicated gradients from one party to another. In doing so, the authors motivate a new measure for privacy termed the leak AUC, based on the AUC of the ROC curve.", "summary_of_the_review": "Understanding the label leakage problem and protection against it under a split learning setup has the potential to start an intriguing line of work. The main drawback of this work is the restricted setup on which the attack/defense models are built on.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-prtfn2C2nH", "reviewer": "Reviewer_SgGa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- Label privacy is an overlooked problem that has many practical applications. This paper provides a formulation for this problem together with a fairly strong attacks and a promising mitigation.\n\n## Weaknesses\n\n- I think the current split learning setting seems strange: if our goal is strictly to protect just the labels and not the feature vectors, it would also be fine to the non-label party to send the features directly to the label party; then, the latter can train the entire model by itself without leaking any label at all. This seems much better for label-privacy and also for communication complexity.\n\n## Comments for Authors\n\n- As stated above, it would be good to justify the current split learning setting more. For example, is the goal here to also protect the X? If so, then shouldn't we also employ some method for that? And how would such a method, in conjunction with those presented in the paper, effect the utility?\n\n- It is stated (on pages 2 and 3) that \"DP and its variants are not directly applicable metric in our setting\". I do not think this statement is true. In fact, *local* DP would still be applicable to non-aggregate data. The paper [Ghazi et al. 2021] you cited also yield label DP in the local setting without aggregation (by flipping the labels). Similarly, the isotropic Gaussian baseline discussed in the paper also achieve some level of local DP. I think this point has to be clarified in the revised version.\n", "summary_of_the_paper": "## Summary of Contributions\n\nThis paper studies the multiparty setting where there are two parties: the first party (aka non-label party) holds the feature vectors $X_i$ of user $i$ whereas the second party (aka label party) holds the corresponding labels $y_i \\in \\{0, 1\\}$. Together they wish to jointly train a model in such a way that the non-label party does not learn of the users' labels. This is especially relevant in online advertisement (where the label party corresponds to the advertiser who knows whether each user converts and the non-label party is e.g. the publisher who shows the ads) and in medical applications (where the label party is the hospital).\n\nThe paper focuses on *split training* which is a setting where the model is divided among two parties with the first layers (denoted as a function $f$) is with the first party and the remaining layers (denoted as a function $h$) are with the second party. Here the training is performed as follows. First, the non-label party computes $f(X)$ (referred to as the \"cut layer\") and send this to the label party. The label party then compute the gradient w.r.t. $h$ and update its parameters and furthermore it computes the gradient $g$ w.r.t. the cut layer $f(X)$ and sends it back to the non-label party. The non-label party can then use backpropagation starting with $g$ to update its own parameters. The main concern tackled in this paper is that, in such a scheme, the gradient $g$ sent back to the non-label party can leak the information about the label. Specifically, the authors observe and experimentally show that the following two attacks are very effective in predicting the labels given the intermediate gradient $g$:\n- Consider the norm $\\\\|g\\\\|_2$ and make prediction based on whether it exceeds a certain threshold.\n- Consider the cosine similarity between $g$ and another (fixed) gradient, and then threshold.\nIn fact, the two above attacks get near-perfect predictions in some examples. The exact measure they use to evaluate their model is the area-under-curve (AUC) of the (false positive rate, true positive rate) curve.\n\nThe authors then propose to add noise in order to mitigate such attacks. The baseline here would be to add isotopic Gaussian noise (with a certain scale). The authors then formulate an optimization problem aiming to achieve better privacy-utility tradeoff. Roughly speaking, this is an optimization problem of the form \"minimize AUC\" such that \"utility is at least ...\". It is hard to make this precise (and it is probably inefficient), so the authors use certain proxy for both the objective and the constraint. For the objective, the authors show that AUC is upper bound by a certain quantity involving the KL divergence of the distributions of the gradients from the two labels. For the utility constraint, the authors instead try to restrict the amount of the noise added by having an upper bound on the trace of their covariance matrices. Furthermore, the authors show that when the (unperturbed) gradient distributions are assumed to be Gaussians and only Gaussian noises are considered, then the optimization problem simplifies greatly to just a single constant-size optimization problem (Theorem 2). This final form constitutes their so-called Marvell algorithm. Finally, the paper concludes by several experimental results showing that Marvell is very effective against protecting the two aforementioned attacks in real-world datasets and models, while preserving reasonable level of privacy.", "summary_of_the_review": "## Recommendation\n\nI think formalizing attack vectors and proposing mitigations for label privacy is an important contribution that may have a large impact. On the other hand, the current formulation/attack/mitigation is tailored towards the specific split learning setting which (as stated above) might require more motivation. Taken both into account, I'm giving a weak accept for now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers concur on the fact that the paper contains solid ideas. The discussion helped clarify the case of class-imbalance and no major concerns remained after discussion phase. I thank the authors for the additional details on execution time / complexity.

On a separate note and perhaps to dig further in the paper's ideas,

1- the validity of the Gaussian assumption carried in the paper was raised (e.g. ercK), but I would like to point out that Theorem 2 can also be derived for general exponential families given the objective in (2), with perhaps a reformulation of the trace constraint (still, this would imply the knowledge of the exponential family for the KL divergence to simplify).

2- when it comes to protecting labels, the authors might want to have a look at the rich literature on learning from label proportions, which shows that the knowledge of the class is not necessary to learn a supervised model (see for example Patrini et al, NeurIPS / NIPS 2014). Thus, protecting the class could in fact be more achievable than by just considering that learning “needs observed classes”.
Decision: Accept (Poster)