Deep Learning with Label Differential Privacy

1 Introduction . The widespread adoption of machine learning in recent years has increased the concerns about the privacy of individuals whose data is used during the model training . Differential privacy ( DP ) [ 34 , 33 ] has emerged as a popular privacy notion that has been the basis of several practical deployments in industry [ 35 , 82 , 43 , 9 , 29 ] and the U.S. Census [ 3 ] . A classical algorithm—that predates DP and was initially designed to eliminate evasive answer biases in survey aggregation—is Randomized Response ( RR ) [ 96 ] : when the input is an element from a finite alphabet , the output is equal to the input with a certain probability , and is a uniform random other element from the alphabet with the remaining probability . This simple algorithm is shown to satisfy the strong notion of local DP [ 38 , 58 ] , whereby the response of each user is protected , in contrast with the so-called central DP setting where a curator has access to the raw user data , and only the output of the curator is required to be DP . We note that schemes building on RR have been studied in several previous works on DP estimation ( e.g. , [ 30 , 57 ] ) , and have been deployed in practice [ 82 ] . Meanwhile , the large error incurred by RR ( e.g. , [ 17 ] ) has stimulated significant research aiming to improve its accuracy , mostly by relaxing to weaker privacy models ( e.g. , [ 36 , 24 , 4 ] ) . In this work , we use a different approach and seek to improve RR by leveraging available prior information . ( A recent work of Liu et al . [ 63 ] also used priors to improve accuracy , but in the context of the DP multiplicative weights algorithm , which applies in the central DP setting . ) The prior information can consist of domain-specific knowledge , ( models trained on ) publicly available data , or historical ⇤Part of this work was done while at Google Research . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . runs of a training algorithm . Our algorithm is presented in Section 3 ( Algorithm 2 ) . At a high level , given a prior distribution p on the alphabet , the algorithm uses p to prune the alphabet . If the prior is reliable and even if the alphabet is only minimally pruned , the probability that the output equals the input is larger than when RR is applied to the entire alphabet . On the other hand , if the prior is uniform over the entire alphabet , then our algorithm recovers the classical RR . To implement the above recipe , one needs to specify how to effect the pruning using p. It turns out that the magnitude of pruning can itself vary depending on p , but we can obtain a closed-form formula for determining this . Interestingly , by studying a suitable linear program , we show that the resulting RRWithPrior strategy is optimal in that among all `` -DP algorithms , it maximizes the probability that the output equals the input when the latter is sampled from p ( Theorem 3 ) . 1.1 Applications to Learning with Label Differential Privacy . There have been a great number of papers over the last decade that developed DP machine learning algorithms ( e.g. , [ 19 , 102 , 88 , 83–85 , 78 ] ) . In the case of deep learning , the seminal work of Abadi et al . [ 2 ] introduced a DP training framework ( DP-SGD ) that was integrated into TensorFlow [ 80 ] and PyTorch [ 92 ] . Despite numerous followup works , including , e.g. , [ 75–77 , 67 , 98 , 71 , 20 , 93 ] , and extensive efforts , the accuracy of models trained with DP-SGD remains significantly lower than that of models trained without DP constraints . Notably , for the widely considered CIFAR-10 dataset , the highest reported accuracy for DP models is 69.3 % [ 93 ] , which strikingly relies on handcrafted visual features despite that in non-private scenarios learned features long been shown to be superior . Even using pre-training with external ( CIFAR-100 ) data , the best reported DP accuracy , 73 % 2 , is still far below the non-private baselines ( > 95 % ) . The performance gap becomes a roadblocker for many real-world applications to adopt DP . In this paper , we focus on a more restricted , but important , special case where the DP guarantee is only required to hold with respect to the labels , as described next . In the label differential privacy ( Labe DP ) setting , the labels are considered sensitive , and their privacy needs to be protected , while the input points are not sensitive . This notion has been studied in the PAC setting [ 18 , 13 ] and for the particular case of sparse linear regression [ 94 ] , and it captures several practical scenarios . Examples include : ( i ) computational advertising where the impressions are known to the Ad Tech3 , and thus considered non-sensitive , while the conversions reveal user interest and are thus private ( see , e.g. , Nalpas and Dutton [ 70 ] and [ 8 ] ) , ( ii ) recommendation systems where the choices are known , e.g. , to the streaming service provider , but the user ratings are considered sensitive , and ( iii ) user surveys and analytics where demographic information ( e.g. , age , gender ) is non-sensitive but income is sensitive—in fact , this was the motivating reason for Warner [ 96 ] to propose RR many decades ago ! We present a novel multi-stage algorithm ( LP-MST ) for training deep neural networks with Labe DP that builds on top of RRWithPrior ( see Section 3 and Algorithm 3 ) , and we benchmark its empirical performance ( Section 5 ) on multiple datasets , domains , and architectures , including the following . • On CIFAR-10 , we show that it achieves 20 % higher accuracy than DP-SGD4 . • On the more challenging CIFAR-100 , we present the first non-trivial DP learning results . • On MovieLens , which consists of user ratings of movies , we show improvements via LP-MST . In some applications , domain specific algorithms can be used to obtain priors directly without going through multi-stage training . For image classification problems , we demonstrate how priors computed from a ( non-private ) self-supervised learning [ 22 , 23 , 44 , 53 , 16 ] phase on the input images can be used to achieve higher accuracy with a Labe DP guarantee with extremely small privacy budgets ( `` 0.1 , see Section 5.2 for details ) . We note that due to the requirement of DP-SGD to compute and clip per-instance gradient , it remains technically challenging to scale to larger models or mini-batch sizes , despite numerous attempts to minigate this problem [ 42 , 5 , 26 , 90 ] . On the other hand , our formulation allows us to use state-of- 2For DP parameters of `` = 8 and = 10 5 , cited from Abadi et al . [ 2 , Figure 6 ] . For a formal definition of DP , we refer the reader to Definition 2.1 . 3Ad tech ( abbreviating Advertising Technology ) comprises the tools that help agencies and brands target , deliver , and analyze their digital advertising efforts ; see , e.g. , b og.hubspot.com/marketing/what-is-ad-tech . 4We remark that the notion of `` -DP in [ 2 , 76 , 75 ] is not directly comparable to `` -Label DP in our work in that they use the addition/removal notion whereas we use the substitution one . Please see the Supplementary Material for more discussion on this . the-art deep learning architectures such as ResNet [ 51 ] . We also stress that our LP-MST algorithm goes beyond deep learning methods that are robust to label noise . ( See [ 87 ] for a survey of the latter . ) Our empirical results suggest that protecting the privacy of labels can be significantly easier than protecting the privacy of both inputs and labels . We find further evidence to this by showing that for the special case of stochastic convex optimization ( SCO ) , the sample complexity of algorithms privatizing the labels is much smaller than that of algorithms privatizing both labels and inputs ; specifically , we achieve dimension-independent bounds for Labe DP ( Section 6 ) . We also show that a good prior can ensure smaller population error for non-convex loss . ( Details are in the Supplementary Material . ) 2 Preliminaries . For any positive integer K , let [ K ] : = { 1 , . . . , K } . Randomized response ( RR ) [ 96 ] is the following : let `` 0 be a parameter and let y 2 [ K ] be the true value known to RR '' . When an observer queries the value of y , RR '' responds with a random draw ỹ from the following probability distribution : Pr [ ỹ = ŷ ] = ( e '' e '' +K 1 for ŷ = y , 1 e '' +K 1 otherwise . ( 1 ) In this paper , we focus on the application of learning with label differential privacy . We recall the definition of differential privacy ( DP ) , which is applicable to any notion of neighboring datasets . For a textbook reference , we refer the reader to Dwork and Roth [ 32 ] . Definition 2.1 ( Differential Privacy ( DP ) [ 33 , 34 ] ) . Let `` , 2 R 0 . A randomized algorithm A taking as input a dataset is said to be ( `` , ) -differentially private ( ( `` , ) -DP ) if for any two neighboring datasets D and D0 , and for any subset S of outputs of A , it is the case that Pr [ A ( D ) 2 S ] e `` · Pr [ A ( D0 ) 2 S ] + . If = 0 , then A is said to be `` -differentially private ( `` -DP ) . When applied to machine learning methods in general and deep learning in particular , DP is usually enforced on the weights of the trained model [ see , e.g. , 19 , 59 , 2 ] . In this work , we focus on the notion of label differential privacy . Definition 2.2 ( Label Differential Privacy ) . Let `` , 2 R 0 . A randomized training algorithm A taking as input a dataset is said to be ( `` , ) -label differentially private ( ( `` , ) -Labe DP ) if for any two training datasets D and D0 that differ in the label of a single example , and for any subset S of outputs of A , it is the case that Pr [ A ( D ) 2 S ] e '' · Pr [ A ( D0 ) 2 S ] + . If = 0 , then A is said to be `` -label differentially private ( `` -Labe DP ) . All proofs skipped in the main body are given in the Supplementary Material . 3 Randomized Response with Prior . In many real world applications , a prior distribution about the labels could be publicly obtained from domain knowledge and help the learning process . In particular , we consider a setting where for each ( private ) label y in the training set , there is an associated prior p = ( p1 , . . . , pK ) . The goal is to output a randomized label ỹ that maximizes the probability that the output is correct ( or equivalently maximizes the signal-to-noise ratio ) , i.e. , Pr [ y = ỹ ] . The privacy constraint here is that the algorithm should be `` -DP with respect to y . ( It need not be private with respect to the prior p. ) We first describe our algorithm RRWithPrior by assuming access to such priors . 3.1 Algorithm : RRWithPrior We build our RRWithPrior algorithm with a subroutine called RRTop-k , as shown in Algorithm 1 , which is a modification of randomized response where we only consider the set of k labels i with largest pi . Then , if the input label y belongs to this set , we use standard randomized response on this set . Otherwise , we output a label from this set uniformly at random . The main idea behind RRWithPrior is to dynamically estimate an optimal k⇤ based on the prior p , and run RRTop-k with k⇤ . Specifically , we choose k⇤ by maximizing Pr [ RRTop-k ( y ) = y ] . It is not Algorithm 1 RRTop-k Input : A label y 2 [ K ] Parameters : k 2 [ K ] , prior p = ( p1 , . . . , pK ) 1 . Let Yk be the set of k labels with maximum prior probability ( with ties broken arbritrarily ) . 2 . If y 2 Yk , then output y with probability e `` e '' +k 1 and output y 0 2 Yk \ { y } with probability 1 e '' +k 1 . 3 . If y 62 Yk , output an element from Yk uniformly at random . hard to see that this expression is exactly equal to e `` e '' +k 1 · ⇣P ỹ2Yk pỹ ⌘ if y ⇠ p. RRWithPrior is presented in Algorithm 2 . Algorithm 2 RRWithPrior Input : A label y 2 [ K ] Parameters : prior p = ( p1 , . . . , pK ) 1 . For k 2 [ K ] : ( a ) Compute wk : = e `` e '' +k 1 · ⇣P ỹ2Yk pỹ ⌘ , where Yk is the set of k labels with maximum prior probability ( ties broken arbritrarily ) . 2 . Let k⇤ = argmaxk2 [ K ] wk . 3 . Return an output of RRTop-k ( y ) with k = k⇤ .

Summary: The paper introduces a Randomized response (RR) concept to provide more accurate results at the same level of privacy protection of LabelDP. The method can be incorporated into LP-MST and SSL frameworks to improve model performance. To achieve that, the authors propose RRWithPrior, which maximizes the probability that the output label is correct, and a novel LP-MST for training deep neural networks with LabelDP and RRWithPrior. They show formal proof of optimality of RRWithPrior and some open research questions.