Antipodes of Label Differential Privacy: PATE and ALIBI

1 Introduction . Sophisticated machine learning models perform well on their target tasks despite of—or thanks to—their predilection for data memorization [ 39 , 2 , 17 ] . When such models are trained on non-public inputs , privacy concerns become paramount [ 31 , 6 , 7 ] , which motivates the actively developing area of privacy-preserving machine learning . With some notable exceptions , which we discuss in the related works section , existing research has focused on an “ all or nothing ” privacy definition , where all of the training data , i.e. , both features and labels , is considered private information . While this goal is appropriate for many applications , there are several important scenarios where label-only privacy is the right solution concept . A prominent example of label-only privacy is in online advertising , where the goal is to predict conversion of an ad impression ( the label ) given a user ’ s profile and the spot ’ s context ( the features ) . The features are available to the advertising network , which trains the model , while the labels ( data on past conversion events ) are visible only to the advertiser . More generally , any two-party setting where data is vertically split between public inputs and ( sensitive ) outcomes is a candidate for training with label-only privacy . In this work we adopt the definition of differential privacy as our primary notion of privacy . The definition , introduced in the seminal work by Dwork et al . [ 13 ] , satisfies several important properties that make it well-suited for applications : composability , robustness to auxiliary information , preservation under post-processing , and graceful degradation in the presence of correlated inputs ( group privacy ) . Following Ghazi et al . [ 18 ] , we refer to the setting of label-only differential privacy as Label DP . ∗ { manimalek , imironov , krp , shilov } @ fb.com , Facebook AI . †tramer @ cs.stanford.edu , Stanford University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our contributions . We explore two approaches—with very different characteristics—toward Label DP . These approaches and our methodology for estimating empirical privacy loss via label memorization are briefly reviewed below : • PATE : We adapt the PATE framework ( Private Aggregation of Teacher Ensembles ) of Papernot et al . [ 28 ] to the Label DP setting by observing that PATE ’ s central assumption— availability of a public unlabeled dataset—can be satisfied for free . Indeed , a public unlabeled dataset is simply the private dataset with the labels removed ! By instantiating PATE with a state-of-the-art technique for semi-supervised learning , we demonstrate an excellent tradeoff between empirical privacy loss and accuracy . • ALIBI ( Additive Laplace with Iterative Bayesian Inference ) : We propose applying additive Laplace noise to a one-hot encoding of the label . Since differential privacy is preserved under post-processing , we can de-noise the mechanism ’ s output by performing Bayesian inference using the prior provided by the model itself , doing so continuously during the training process . ALIBI improves , particularly in a high-privacy regime , Ghazi et al. ’ s approach based on randomized response [ 18 ] . • Empirical privacy loss . We describe a memorization attack targeted at the Label DP scenario that efficiently extracts lower bounds for the privacy parameter that , in some cases , come within a factor of 2.5 from the theoretical , worst-case upper bounds against practically relevant , large-scale models . For comparison , recent , most advanced black-box attacks on DP-SGD have approximately a 10× gap between the upper and lower bounds [ 20 , 26 ] . Both algorithms , PATE and ALIBI , come with rigorous privacy analyses that , for any fixed setting of parameters , result in an ( ε , δ ) -DP bound . We emphasize that these bounds are just that—they are upper limits on an attacker ’ s ability to breach privacy ( in a precise sense ) on worst-case inputs . Even though these bounds can be pessimistic , intuitively , a smaller ε for the same δ corresponds to a more private instantiation of a mechanism , and indeed this is likely the case within an algorithmic family . Can the privacy upper bounds be used to compare diverse mechanisms with widely dissimilar analyses , such as ours ? We argue that focusing on the upper bounds alone may be misleading to the point of favoring a less private algorithm with tighter analysis over a more private one whose privacy analysis is less developed . For a uniform perspective , we estimate the empirical privacy loss of all trained models by evaluating the performance of a black-box membership inference attack [ 31 , 20 , 26 ] . Our attack directly measures memorization of deliberately mislabeled examples , or canaries [ 6 ] . Following prior work [ 26 , 20 ] , we use the success rate of our attack to compute a lower bound on the level ε of label-DP achieved by the training algorithm . To avoid the prohibitive cost of retraining a model for each canary , we propose and analyze a heuristic that consists in inserting multiple canaries into one model , and measuring an attacker ’ s success rate in guessing each canary ’ s label . In some settings , we find that the empirical privacy of PATE ( as measured by our attack ) is significantly stronger than that of ALIBI , even though the provable DP bound for PATE is much weaker . Finally , to characterize setups where label-only privacy may be applicable , it is helpful to distinguish between inference tasks , where the label is fully determined by the features , and prediction , where there is some uncertainty as to the eventual outcome , despite ( hopefully ) some predictive value of the feature set . We are mostly concerned with the latter scenario , where the outcome is not evident from the features , or protecting choices made by individuals may be mandated or desirable ( for connection between privacy and self-determination see Schwartz [ 30 ] ) . While we do use public data sets ( CIFAR-10 and CIFAR-100 ) in which it is possible to “ infer ” the label , we do so only to evaluate our approaches on standard benchmarks . We completely remove the triviality of inference by adding mislabeled canaries in our attack , thereby introducing a measure of non-determinism . 2 Notation and Definitions . Throughout the paper we consider standard supervised learning problems . The inputs are ( x , y ) pairs where x ∈ X are the features and y ∈ Y are the labels . ( If some labels are absent , the problem becomes semi-supervised learning . ) The cardinality of the set Y is the number of classes C. The task is to learn a model M parameterized with weights W that predicts y given x . To this end the training algorithm has access to a training set of n samples { ( xi , yi ) } n. Differential privacy due to Dwork et al . [ 13 ] is a rigorous , quantifiable notion of individual privacy for statistical algorithms . ( See Dwork and Roth [ 14 ] and Vadhan [ 33 ] for book-length introductions . ) Definition 2.1 ( Differential Privacy ) . We say that a randomized mechanismM : U → R satisfies ( ε , δ ) -differential privacy ( DP ) if for all adjacent inputs D , D′ ∈ U that differ in contributions of a single sample , the following holds : ∀S ⊂ R Pr [ M ( D ) ∈ S ] ≤ eε · Pr [ M ( D′ ) ∈ S ] + δ . If δ = 0 , we refer to it as ε-DP . The notion of adjacency between databases D and D′ is domain- and application-dependent . To instantiate the Label DP setting , we say that D and D′ are adjacent if they consist of the same number of examples { ( xi , yi ) } n and are identical except for a difference in a single label at index i∗ : ( xi∗ , yi∗ ) ∈ D and ( xi∗ , y′i∗ ) ∈ D′ . 3 Background and related work . DP and machine learning . Differential privacy aligns extremely well with the goal of machine learning , which is to produce valid models or hypotheses on the basis of observed data . In fact , the relationship can be made precise . An important metric in ML is generalization error , which measures the model ’ s performance on previously unseen samples . Differentially private mechanisms provably generalize [ 11 ] . For an introduction to some important early results on connections between differential privacy and learning theory see Dwork and Roth [ 14 , Chapter 11 ] . Randomized response ( RR ) . As a disclosure avoidance methodology , randomized response ( RR ) [ 36 ] precedes the advent of differential privacy . In RR ’ s basic form , with some probability the sensitive input is replaced by a sample drawn uniformly from its entire domain . RR can be implemented at the source , making it an essential building block in achieving Local DP [ 16 ] . More generally , RR is an instance of the input perturbation technique , which leaves the ML training algorithm intact and achieves differential privacy via input randomization . Input perturbations methods are particularly friendly to implementers as they require minimal changes to existing algorithms , treating them as black boxes . Additive noise mechanisms . Another important class of differentially private mechanisms are additive noise methods . Two canonical examples are additive Laplace [ 13 ] and additive Gaussian [ 12 ] that can be applied to vector-valued functions . Their noise distributions are calibrated to the function ’ s sensitivity—the maximal distance between the function ’ s outputs on adjacent inputs ( D and D′ in Definition 2.1 ) under the ` 1 or ` 2 metrics for the Laplace and Gaussian mechanisms respectively . PATE . Private Aggregation of Teacher Ensembles ( PATE ) [ 28 , 29 ] is a framework based on private knowledge aggregation of an ensemble model and knowledge transfer . PATE trains an ensemble of “ teachers ” on disjoint subsets of the private dataset . The ensemble ’ s knowledge is then transferred to a “ student ” model via differentially private aggregation of the teachers ’ votes on samples from an unlabeled public dataset . Only the student model is released as the output of the training , as it accesses sensitive data via a privacy-preserving interface . Since PATE consumes the privacy budget each time the student queries the teachers , the training algorithm must be optimized to minimize the number of queries . To this end , Papernot et al . explore the use of semi-supervised learning techniques , most notably GANs . Recent advances in semisupervised learning have been shown to boost the performance of PATE on standard benchmarks [ 5 ] . FixMatch . FixMatch [ 32 ] is a state-of-the-art semi-supervised learning algorithm , which achieves high accuracy on benchmark image datasets with very few labeled examples . It is based on the concepts of pseudo-labeling [ 23 ] and consistency regularization [ 3 ] : FixMatch uses a partially trained model to generate labels on weakly-augmented unlabeled data , and then , if the prediction is confident enough , uses the output as a label for the strongly-augmented version of the same data point . Label-only privacy . The setting of label-only DP was formally introduced by Chaudhuri and Hsu [ 8 ] , who proved lower bounds on sample complexity for private PAC-learners . Beimel et al . [ 4 ] demonstrated that the sample complexity of Label DP matches that of non-private PAC learning ( ignoring constants , dependencies on the privacy parameters , and computational efficiency ) . Wang and Xu [ 34 ] considered the linear regression problem in the local , label-only privacy setting . Most recently , Ghazi et al . [ 18 ] considered label-only privacy for deep learning . They propose a Label Privacy Multi-Stage Training ( LP-MST ) algorithm , which we revisit in Section 5 . The algorithm advances in stages ( Ghazi et al . evaluate it up to four ) , training the model on disjoint partitions of the dataset . In each stage , the private labels are protected using a randomized response scheme . The most innovative part of LP-MST happens at the point of transition between stages . Labels of the training examples of the next partition are perturbed using randomized response . The ( soft ) output of the previous stage ’ s model is interpreted as a prior , which is used to choose parameters for the randomized response algorithm separately for each example . For instance , if the current model assigns high probabilities to labels 1–10 and low probabilities to labels 11–100 , the randomized response will be constrained to outputting a label within the set 1–10 . We build on the insights of Ghazi et al . by making two observations . First , training algorithms used for deep learning ( such as SGD and its variants ) can naturally handle soft labels ( i.e. , probability distributions over the label space ) ; there is thus no need to force hard labels by sampling from the posterior . Second , by applying additive noise , we can repeatedly recompute the posterior distribution as better priors become available without consuming privacy budget . By following the “ once and done ” approach to label perturbation , we forgo the need to split training into disjoint stages . To summarize , Ghazi et al . fix the prior at the beginning of the stage , choose parameters of the randomized response algorithm on the basis of this prior , and update the model by feeding it the perturbed label . We flip the order of these operations by applying a DP mechanism ( additive Laplace ) first , and then repeatedly use Bayesian inference by combining the prior that changes with each model update and the observables that do not . To our knowledge , we are the first to explore connections between PATE and Label DP .

Summary: This paper focuses on deep learning with level differential privacy (DP), in which the privacy of only the label is preserved. The authors propose two new algorithms: a variation of PATE using a recent semi-supervised learning approach and the fact that features are public in label DP to train better teacher models and use fewer student model queries which consume DP budget ; and a new approach based on local DP noise for the labels, and a Bayesian post-processing that uses the current version of the model to refine guesses about the true underlying label. Finally, the authors run a label memorization attack to empirically assess the privacy.