Deep Learning with Label Differential Privacy

1 Introduction . The widespread adoption of machine learning in recent years has increased the concerns about the privacy of individuals whose data is used during the model training . Differential privacy ( DP ) [ 34 , 33 ] has emerged as a popular privacy notion that has been the basis of several practical deployments in industry [ 35 , 82 , 43 , 9 , 29 ] and the U.S. Census [ 3 ] . A classical algorithm—that predates DP and was initially designed to eliminate evasive answer biases in survey aggregation—is Randomized Response ( RR ) [ 96 ] : when the input is an element from a finite alphabet , the output is equal to the input with a certain probability , and is a uniform random other element from the alphabet with the remaining probability . This simple algorithm is shown to satisfy the strong notion of local DP [ 38 , 58 ] , whereby the response of each user is protected , in contrast with the so-called central DP setting where a curator has access to the raw user data , and only the output of the curator is required to be DP . We note that schemes building on RR have been studied in several previous works on DP estimation ( e.g. , [ 30 , 57 ] ) , and have been deployed in practice [ 82 ] . Meanwhile , the large error incurred by RR ( e.g. , [ 17 ] ) has stimulated significant research aiming to improve its accuracy , mostly by relaxing to weaker privacy models ( e.g. , [ 36 , 24 , 4 ] ) . In this work , we use a different approach and seek to improve RR by leveraging available prior information . ( A recent work of Liu et al . [ 63 ] also used priors to improve accuracy , but in the context of the DP multiplicative weights algorithm , which applies in the central DP setting . ) The prior information can consist of domain-specific knowledge , ( models trained on ) publicly available data , or historical ⇤Part of this work was done while at Google Research . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . runs of a training algorithm . Our algorithm is presented in Section 3 ( Algorithm 2 ) . At a high level , given a prior distribution p on the alphabet , the algorithm uses p to prune the alphabet . If the prior is reliable and even if the alphabet is only minimally pruned , the probability that the output equals the input is larger than when RR is applied to the entire alphabet . On the other hand , if the prior is uniform over the entire alphabet , then our algorithm recovers the classical RR . To implement the above recipe , one needs to specify how to effect the pruning using p. It turns out that the magnitude of pruning can itself vary depending on p , but we can obtain a closed-form formula for determining this . Interestingly , by studying a suitable linear program , we show that the resulting RRWithPrior strategy is optimal in that among all `` -DP algorithms , it maximizes the probability that the output equals the input when the latter is sampled from p ( Theorem 3 ) . 1.1 Applications to Learning with Label Differential Privacy . There have been a great number of papers over the last decade that developed DP machine learning algorithms ( e.g. , [ 19 , 102 , 88 , 83–85 , 78 ] ) . In the case of deep learning , the seminal work of Abadi et al . [ 2 ] introduced a DP training framework ( DP-SGD ) that was integrated into TensorFlow [ 80 ] and PyTorch [ 92 ] . Despite numerous followup works , including , e.g. , [ 75–77 , 67 , 98 , 71 , 20 , 93 ] , and extensive efforts , the accuracy of models trained with DP-SGD remains significantly lower than that of models trained without DP constraints . Notably , for the widely considered CIFAR-10 dataset , the highest reported accuracy for DP models is 69.3 % [ 93 ] , which strikingly relies on handcrafted visual features despite that in non-private scenarios learned features long been shown to be superior . Even using pre-training with external ( CIFAR-100 ) data , the best reported DP accuracy , 73 % 2 , is still far below the non-private baselines ( > 95 % ) . The performance gap becomes a roadblocker for many real-world applications to adopt DP . In this paper , we focus on a more restricted , but important , special case where the DP guarantee is only required to hold with respect to the labels , as described next . In the label differential privacy ( Labe DP ) setting , the labels are considered sensitive , and their privacy needs to be protected , while the input points are not sensitive . This notion has been studied in the PAC setting [ 18 , 13 ] and for the particular case of sparse linear regression [ 94 ] , and it captures several practical scenarios . Examples include : ( i ) computational advertising where the impressions are known to the Ad Tech3 , and thus considered non-sensitive , while the conversions reveal user interest and are thus private ( see , e.g. , Nalpas and Dutton [ 70 ] and [ 8 ] ) , ( ii ) recommendation systems where the choices are known , e.g. , to the streaming service provider , but the user ratings are considered sensitive , and ( iii ) user surveys and analytics where demographic information ( e.g. , age , gender ) is non-sensitive but income is sensitive—in fact , this was the motivating reason for Warner [ 96 ] to propose RR many decades ago ! We present a novel multi-stage algorithm ( LP-MST ) for training deep neural networks with Labe DP that builds on top of RRWithPrior ( see Section 3 and Algorithm 3 ) , and we benchmark its empirical performance ( Section 5 ) on multiple datasets , domains , and architectures , including the following . • On CIFAR-10 , we show that it achieves 20 % higher accuracy than DP-SGD4 . • On the more challenging CIFAR-100 , we present the first non-trivial DP learning results . • On MovieLens , which consists of user ratings of movies , we show improvements via LP-MST . In some applications , domain specific algorithms can be used to obtain priors directly without going through multi-stage training . For image classification problems , we demonstrate how priors computed from a ( non-private ) self-supervised learning [ 22 , 23 , 44 , 53 , 16 ] phase on the input images can be used to achieve higher accuracy with a Labe DP guarantee with extremely small privacy budgets ( `` 0.1 , see Section 5.2 for details ) . We note that due to the requirement of DP-SGD to compute and clip per-instance gradient , it remains technically challenging to scale to larger models or mini-batch sizes , despite numerous attempts to minigate this problem [ 42 , 5 , 26 , 90 ] . On the other hand , our formulation allows us to use state-of- 2For DP parameters of `` = 8 and = 10 5 , cited from Abadi et al . [ 2 , Figure 6 ] . For a formal definition of DP , we refer the reader to Definition 2.1 . 3Ad tech ( abbreviating Advertising Technology ) comprises the tools that help agencies and brands target , deliver , and analyze their digital advertising efforts ; see , e.g. , b og.hubspot.com/marketing/what-is-ad-tech . 4We remark that the notion of `` -DP in [ 2 , 76 , 75 ] is not directly comparable to `` -Label DP in our work in that they use the addition/removal notion whereas we use the substitution one . Please see the Supplementary Material for more discussion on this . the-art deep learning architectures such as ResNet [ 51 ] . We also stress that our LP-MST algorithm goes beyond deep learning methods that are robust to label noise . ( See [ 87 ] for a survey of the latter . ) Our empirical results suggest that protecting the privacy of labels can be significantly easier than protecting the privacy of both inputs and labels . We find further evidence to this by showing that for the special case of stochastic convex optimization ( SCO ) , the sample complexity of algorithms privatizing the labels is much smaller than that of algorithms privatizing both labels and inputs ; specifically , we achieve dimension-independent bounds for Labe DP ( Section 6 ) . We also show that a good prior can ensure smaller population error for non-convex loss . ( Details are in the Supplementary Material . ) 2 Preliminaries . For any positive integer K , let [ K ] : = { 1 , . . . , K } . Randomized response ( RR ) [ 96 ] is the following : let `` 0 be a parameter and let y 2 [ K ] be the true value known to RR '' . When an observer queries the value of y , RR '' responds with a random draw ỹ from the following probability distribution : Pr [ ỹ = ŷ ] = ( e '' e '' +K 1 for ŷ = y , 1 e '' +K 1 otherwise . ( 1 ) In this paper , we focus on the application of learning with label differential privacy . We recall the definition of differential privacy ( DP ) , which is applicable to any notion of neighboring datasets . For a textbook reference , we refer the reader to Dwork and Roth [ 32 ] . Definition 2.1 ( Differential Privacy ( DP ) [ 33 , 34 ] ) . Let `` , 2 R 0 . A randomized algorithm A taking as input a dataset is said to be ( `` , ) -differentially private ( ( `` , ) -DP ) if for any two neighboring datasets D and D0 , and for any subset S of outputs of A , it is the case that Pr [ A ( D ) 2 S ] e `` · Pr [ A ( D0 ) 2 S ] + . If = 0 , then A is said to be `` -differentially private ( `` -DP ) . When applied to machine learning methods in general and deep learning in particular , DP is usually enforced on the weights of the trained model [ see , e.g. , 19 , 59 , 2 ] . In this work , we focus on the notion of label differential privacy . Definition 2.2 ( Label Differential Privacy ) . Let `` , 2 R 0 . A randomized training algorithm A taking as input a dataset is said to be ( `` , ) -label differentially private ( ( `` , ) -Labe DP ) if for any two training datasets D and D0 that differ in the label of a single example , and for any subset S of outputs of A , it is the case that Pr [ A ( D ) 2 S ] e '' · Pr [ A ( D0 ) 2 S ] + . If = 0 , then A is said to be `` -label differentially private ( `` -Labe DP ) . All proofs skipped in the main body are given in the Supplementary Material . 3 Randomized Response with Prior . In many real world applications , a prior distribution about the labels could be publicly obtained from domain knowledge and help the learning process . In particular , we consider a setting where for each ( private ) label y in the training set , there is an associated prior p = ( p1 , . . . , pK ) . The goal is to output a randomized label ỹ that maximizes the probability that the output is correct ( or equivalently maximizes the signal-to-noise ratio ) , i.e. , Pr [ y = ỹ ] . The privacy constraint here is that the algorithm should be `` -DP with respect to y . ( It need not be private with respect to the prior p. ) We first describe our algorithm RRWithPrior by assuming access to such priors . 3.1 Algorithm : RRWithPrior We build our RRWithPrior algorithm with a subroutine called RRTop-k , as shown in Algorithm 1 , which is a modification of randomized response where we only consider the set of k labels i with largest pi . Then , if the input label y belongs to this set , we use standard randomized response on this set . Otherwise , we output a label from this set uniformly at random . The main idea behind RRWithPrior is to dynamically estimate an optimal k⇤ based on the prior p , and run RRTop-k with k⇤ . Specifically , we choose k⇤ by maximizing Pr [ RRTop-k ( y ) = y ] . It is not Algorithm 1 RRTop-k Input : A label y 2 [ K ] Parameters : k 2 [ K ] , prior p = ( p1 , . . . , pK ) 1 . Let Yk be the set of k labels with maximum prior probability ( with ties broken arbritrarily ) . 2 . If y 2 Yk , then output y with probability e `` e '' +k 1 and output y 0 2 Yk \ { y } with probability 1 e '' +k 1 . 3 . If y 62 Yk , output an element from Yk uniformly at random . hard to see that this expression is exactly equal to e `` e '' +k 1 · ⇣P ỹ2Yk pỹ ⌘ if y ⇠ p. RRWithPrior is presented in Algorithm 2 . Algorithm 2 RRWithPrior Input : A label y 2 [ K ] Parameters : prior p = ( p1 , . . . , pK ) 1 . For k 2 [ K ] : ( a ) Compute wk : = e `` e '' +k 1 · ⇣P ỹ2Yk pỹ ⌘ , where Yk is the set of k labels with maximum prior probability ( ties broken arbritrarily ) . 2 . Let k⇤ = argmaxk2 [ K ] wk . 3 . Return an output of RRTop-k ( y ) with k = k⇤ .

Summary: This paper studies a Randomized Response algorithm with prior (RRwithPrior), which takes as input a label y, a prior distribution p on the possible labels [K], and outputs a randomized label aiming to maximize the probability that the output is y, while preserving epsilon-DP. The first claim of the paper is that the proposed algorithm is optimal among all epsilon-DP algorithms, given that the label y is actually distributed according to the prior distribution p. This is proven by arguing that RRwithPrior gives the optimal solution to a Linear Program that models this problem.   Then, the authors use the RRwithPrior algorithm as a subroutine to LP-MST, a multi-stage algorithm for training NNs with label differential privacy (LabelDP). LabelDP is a weaker notion of privacy where neighboring data sets differ on the label of a single example. The algorithm proceeds in stages, in each of which a model is trained on disjoint partitions of the dataset and the private labels are protected using RRwithPrior. An important thing to note is that LP-MST does not require a prior and is essentially boosting the prior that RRwtihPrior uses: it starts with the uniform distribution over all [K] classes, then labels the first batch of examples according to RRwithPrior with this uniform prior and moves to the next stage using the new model as a prior. LP-MST with one stage (called LP-1ST) is equivalent to Randomized Response (without prior). The performance of the epsilon-LabelDP model output by variations of LP-MST is empirically evaluated against other methods, demonstrating increased accuracy. The authors also experiment with new advances in self-supervised learning to retrieve useful priors.   The above are the main contributions of the paper, but this work also includes a study of SGD with LabelDP. The authors give a version of SGD with Randomized Response which satisfies epsilon-LabelDP and achieves a excess true risk in the order of K/(epsilon*sqrt(n)), hiding the dependence on the diameter of the parameter space and the Lipschitz constant. They also give a (not very tightly matching) lower bound of 1/\sqrt(epsilon*n). They extend the study to (epsilon, delta)-LabelDP, saving a sqrt(K) factor from the upper bound. The lower bound above works for (epsilon, delta)-LabelDP as well so in this case the gap between the two is smaller. In the process of deriving the lower bound, the authors give lower bounds on the empirical loss and give an extension of a known reduction from private SCO to private ERM from Bassily et al. to the case of LabelDP. They finally consider a slightly different algorithm and give bounds that depend on the quality of the priors.